[Bug]: jest_test rule's performance is not upto the mark

[deepu-mungamuri94]

What happened?

We have around 200 modules/packages that have jest tests. All of these modules must run around 58K jest tests.

We enable the Bazel test target for each module using the jest_test rule, as seen below.
bazel test //module_1:jest_test (It has around 4000 jest tests.)
bazel test //module_2:jest_test
bazel test //module_200:jest_test

When I attempt to start jest_test for a certain module, it runs quite quickly.
Example:

• Command: 'bazel test //module_1:jest_test'
• Duration: 80 seconds

When I try to run jest_test across all modules, the bazel parallelization kicks in and the performance for module level targets suffers.
Example:

• Command: 'bazel test $(bazel query //... | grep "jest_test")'
• Rules are activated: all 200 modules are triggered and 15 are executed at a time. [parallelization]
• Duration : 250 seconds

Version

Development (host) and target OS/architectures:

Output of bazel --version: bazel 6.1.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: rules_jest-0.18.4

Language(s) and/or frameworks involved: Jest Framework

How to reproduce

Ideally, you should have roughly 100 packages/modules that have jest tests.

Check the total time it takes to complete any of the major modules exclusively.

Trigger all of the modules at once and compare the time it took for a big module to the time it took for an exclusive runtime.

Any other information?

No response

[jbedard]

What are your performance expectations in this case? Is it significantly faster outside bazel with other rulesets?

If 200 modules are tested in only 3x the time as 1 module that seems very efficient, or am I misunderstanding those numbers?

[deepu-mungamuri94]

Hi @jbedard, Sorry if I was unclear. This is what I'm saying :
I'm simply interested in understanding why there is no time parity b/w execution times of parallel vs exclusive runs.

• When I run a single bazel target (Command: "bazel test //module_1:jest_test"), that specific module took 80 seconds to complete. [ Only one Bazel job is active in this instance ]
• Running all bazel targets with the command "bazel test $(bazel query //... | grep "jest_test")" took 250 seconds for the specific module we are discussing about. My assumption is, because there are 15 jobs going at any given time, the computer may be busy in this case, thus the module we're discussing takes longer than the first.

My assumption is that regardless of how we run in Bazel (parallel/exclusive), the specific module will take around the same amount of time. But according to my observation, bazel takes more time when we use parallelism & performs best when we run exclusively with respect to the single module execution time.

When we run exclusively :

When we run using bazel parallelization : ( This is only with 4 modules at a time & this becomes too slow when I include even more modules)

[jbedard]

I assume it's just from general performance gains and losses that come with parallelization.

Bazel will estimate the best number of jobs to parallelize, but it's just a best-guess or estimate. You can give hints about targets such as size, timeout, flaky or better control parallelization (if the rule supports it) using shard_count, then bazel will use that information to try to do things in the optimal order and concurrency but it can never be perfect. You can also limit the number of concurrent jobs using --jobs, although normally the bazel default (based on number of CPUs) is best unless you know there are other processes on the machine not controlled by bazel.

I assume running all your tests in parallel is still faster then sequentially though?

[deepu-mungamuri94]

Thanks for the info @jbedard.

Yes, running all tests in parallel is still faster than sequential runs as a whole. The only concern is, sometimes parallel runs getting TIMEOUTs for few modules after 300 seconds as I am using size=medium. Increasing the size,timeout solves the issue temporarily(and its not optimal w.r.t machine configurations), but it'll raise again at later point of time as the test count raises.
Because of these TIMEOUTs, the test results for those modules are not available for our report generation purpose.

[jbedard]

If those tests exceed the timeout when run in parallel should the timeout be increased? Or maybe the tests split into multiple targets (or sharding might accomplish the same in this case?).

[deepu-mungamuri94]

Yes, it seems like a good idea to separate testing into various targets.
Using configuration/args would be great if the option was included to the "aspect rules_jest" Thus, the FR raised here is aligned.

[jbedard]

Can we close this issue and continue the discussion there then?