[WIP] Code-split components when component testing

[pastelsky]

Code splitting ensures that the test JS / CSS is the minimal required for a test to run in isolation, which should ease up memory and CPU when executing tests.

Code splitting also introduces script boundaries, which means that a runtime error in the global code of another component would not lead to the entire test suite blowing up.

This comes at a price of a slight increase in bundling time if your module graph is large and you have a lot of components, but this should be offset by the fact that once compilation happens, running individual tests is a lot faster.

When running with an without code-splitting on a large real-world test suite at Atlassian with 900+ tests, I noticed that the average time per test dropped from ~5 seconds / test to < 2 seconds / test when code-splitting. Browsers also consumed less memory and CPU which meant we could push for better parallelization on the same hardware.

• Code split for React
• Code split for Vue
• Code split for Vue 2
• Code split for Solid
• Code split for Svelte
• Test all implementations manually / Add additional tests

[pavelfeldman]

Could you help me understand where exactly are the performance gains coming from? I.e. what used to happen and is no longer happening.

[pastelsky]

Currently, all of the examples being tested are compiled into a single bundle. In large apps, the JS bundle can become the bottleneck for running parallel tests. In my testing I could see bottlenecks around: 1. Web server serving large bundles to parallel browsers 2. Browsers downloading the bundle 3. Parse and eval time being large for such monolithic bundles. Ultimately for us, the speed of the test suite was directly proportional to the amount of work (memory, CPU) the browser was doing. In code split bundles, the work done is proportional to the component being tested. Without it, work done is proportional to the global number of components being used. I hope that makes some sense. If you’re aware of any medium- large scale public codebase that uses component testing, I can perhaps share more reproducible data.

On Mon, 23 Jan 2023 at 10:38 PM, Pavel Feldman ***@***.***> wrote: Could you help me understand where exactly are the performance gains coming from? I.e. what used to happen and is no longer happening. — Reply to this email directly, view it on GitHub <#20287 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACEIEH7TVUPA4LNXUEO5AE3WT23IVANCNFSM6AAAAAAUDUIMYQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Regards, Shubham Kanodia

[pavelfeldman]

I guess I'm trying to understand, which line gives bundler a hint on whether it should split the code and whether we can rely on that. You are still importing everything unconditionally as per my read of the code.

In terms of (1), (2) and (3), one would need to stress the system very hard in order to see the effect in (1) and (2). And even for (3), given that component testing is reusing the browser context, everything should settle in the disk and compile caches of the browser, so you should not see any amortized savings. Of course doing less work is always better, but I'm curious why the average test time has changed - maybe we are under-utilizing the browser's compilation cache.

[pastelsky]

I guess I'm trying to understand, which line gives bundler a hint on whether it should split the code and whether we can rely on that.

Dynamic imports establish code split points. See —
https://github.com/microsoft/playwright/pull/20287/files#diff-322969de049b8aa51c59cf00d1461d810e49833f396f39a6bd3e03d734a98919L306-L313

(In terms of (1), (2) and (3), one would need to stress the system very hard in order to see the effect in (1) and (2)

Yes, in our case, we are talking about a large app, with the monolithic bundle produced standing at ~8MB. With a concurrency of ~10 browsers on an M1 mac (say), the webserver needs to be streaming ~80MB at a time (worst case), in which case the performance of the webserver etc. does start to matter. Most apps using playwright might not be large enough for this, but should still see smaller improvements.

And even for (3), given that component testing is reusing the browser context, everything should settle in the disk and compile caches of the browser.

This is interesting because it seems in our case that the webserver essentially serves the bundle for each test case (we logged requests served). If there was disk caching/compile caching involved, we wouldn't be seeing this, nor would we pay the parse cost again. That is unexpected according to you?

Lastly, the isolation benefits (which aligns one of playwright's core philosophies) of code split entry points is a reason on its own — even if we discount the performance benefits.

[pavelfeldman]

Dynamic imports establish code split points.

I see, but it looks like all modules are loaded unconditionally, or why is it not happening? Every time index.js is loaded, you still load all the bundles, where is the performance win coming from?

[pastelsky]

@pavelfeldman Ah, I can see why that might have been confusing. Our internal implementation for code splitting was implemented in userland. I missed making the dynamic imports lazy while copying over changes in this PR to the vitePlugin.

They should be lazily loaded and evaluated on a need-basis now.

playwright/packages/playwright-test/src/plugins/vitePlugin.ts

Lines 311 to 316 in 2d6d374

	for (const [alias, value] of componentRegistry) {
	const importPath = value.isModuleOrAlias ? value.importPath : './' + path.relative(folder, value.importPath).replace(/\\/g, '/');
	if (value.importedName)
	lines.push(`const ${alias} = () => import('${importPath}').then((mod) => mod.${value.importedName});`);
	else
	lines.push(`const ${alias} = () => import('${importPath}');`);

[pastelsky]

I'll take a look at a few of the failing tests this week. But would be good to know if I'm right directionally here.

[pavelfeldman]

Yes, with the latest changes that turn the import into the arrow function, I understand how it becomes lazy / module split!

[pavelfeldman]

I think that's what is making the tests fail.