Parallelize unit tests #6483

andrewlock · 2024-12-24T12:46:49Z

Summary of changes

Parallelize the unit test runs in CI

Reason for change

As we test more frameworks, the runs take longer. We can easily parallize by TFM (e.g. we allow running a specific TFM locally), so this overall speeds up the unit test execution. As we (attempt) to improve the speed of other stages, this allows gaining confidence earlier. Additionally, in the case of flake, re-running a stage takes less time when each TFM is parallelized.

Implementation details

Generate the matricies, use them in CI

Test coverage

Did a test run, looks OK to me

Other details

Reduces the wall time from this:

to this:

datadog-ddstaging · 2024-12-24T13:26:22Z

Datadog Report

Branch report: andrew/ci/parallelize-unit-tests
Commit report: ebc81cd
Test service: dd-trace-dotnet

✅ 0 Failed, 229404 Passed, 2075 Skipped, 18h 28m 6.19s Total Time

andrewlock · 2024-12-24T13:30:08Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6483) - mean (69ms)  : 65, 72
     .   : milestone, 69,
    master - mean (69ms)  : 65, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6483) - mean (976ms)  : 951, 1001
     .   : milestone, 976,
    master - mean (977ms)  : 955, 1000
     .   : milestone, 977,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6483) - mean (107ms)  : 105, 109
     .   : milestone, 107,
    master - mean (107ms)  : 105, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6483) - mean (674ms)  : 659, 689
     .   : milestone, 674,
    master - mean (679ms)  : 662, 695
     .   : milestone, 679,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6483) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6483) - mean (632ms)  : 613, 651
     .   : milestone, 632,
    master - mean (634ms)  : 614, 654
     .   : milestone, 634,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6483) - mean (194ms)  : 190, 199
     .   : milestone, 194,
    master - mean (193ms)  : 189, 197
     .   : milestone, 193,

    section CallTarget+Inlining+NGEN
    This PR (6483) - mean (1,098ms)  : 1066, 1131
     .   : milestone, 1098,
    master - mean (1,099ms)  : 1070, 1129
     .   : milestone, 1099,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6483) - mean (278ms)  : 274, 282
     .   : milestone, 278,
    master - mean (278ms)  : 272, 283
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6483) - mean (870ms)  : 838, 902
     .   : milestone, 870,
    master - mean (871ms)  : 844, 897
     .   : milestone, 871,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6483) - mean (267ms)  : 263, 270
     .   : milestone, 267,
    master - mean (266ms)  : 262, 270
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (6483) - mean (850ms)  : 811, 889
     .   : milestone, 850,
    master - mean (855ms)  : 825, 885
     .   : milestone, 855,

andrewlock · 2024-12-24T14:16:31Z

Benchmarks Report for tracer 🐌

Benchmarks for #6483 compared to master:

1 benchmarks are slower, with geometric mean 1.117
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.15μs	45.4ns	305ns	0.0191	0.00765	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	56.7ns	345ns	0.026	0.0104	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	58.8ns	228ns	1.04	0.311	0.0901	6.21 KB
#6483	`StartStopWithChild`	net6.0	8.13μs	46.5ns	338ns	0.0124	0.00412	0	5.61 KB
#6483	`StartStopWithChild`	netcoreapp3.1	10μs	55.1ns	312ns	0.02	0.01	0	5.8 KB
#6483	`StartStopWithChild`	net472	16.3μs	58.8ns	228ns	1.05	0.324	0.0973	6.2 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	485μs	354ns	1.37μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	648μs	504ns	1.89μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	853μs	631ns	2.28μs	0.425	3.3 KB
#6483	`WriteAndFlushEnrichedTraces`	net6.0	474μs	305ns	1.18μs	0	2.7 KB
#6483	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	653μs	316ns	1.18μs	0	2.7 KB
#6483	`WriteAndFlushEnrichedTraces`	net472	858μs	671ns	2.6μs	0.422	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	153μs	1.2μs	11.9μs	0.14	14.47 KB
master	`SendRequest`	netcoreapp3.1	178μs	1.2μs	11.9μs	0.166	17.27 KB
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#6483	`SendRequest`	net6.0	159μs	986ns	9.86μs	0.16	14.47 KB
#6483	`SendRequest`	netcoreapp3.1	183μs	1.39μs	13.7μs	0.173	17.27 KB
#6483	`SendRequest`	net472	0.000957ns	0.000654ns	0.00245ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	557μs	2.53μs	9.78μs	0.551	0	0	41.52 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	700μs	3.9μs	25.6μs	0.329	0	0	41.86 KB
master	`WriteAndFlushEnrichedTraces`	net472	819μs	2μs	7.5μs	8.17	2.45	0.408	53.32 KB
#6483	`WriteAndFlushEnrichedTraces`	net6.0	544μs	2.5μs	11.5μs	0.541	0	0	41.49 KB
#6483	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	682μs	3.74μs	23.4μs	0.336	0	0	41.84 KB
#6483	`WriteAndFlushEnrichedTraces`	net472	839μs	4.44μs	22.6μs	8.45	2.53	0.422	53.28 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.28μs	1.4ns	5.43ns	0.0142	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.81μs	1.73ns	6.71ns	0.0135	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.1μs	3.89ns	14.6ns	0.157	0.00104	987 B
#6483	`ExecuteNonQuery`	net6.0	1.28μs	1.39ns	5.39ns	0.0146	0	1.02 KB
#6483	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	2.03ns	7.86ns	0.0135	0	1.02 KB
#6483	`ExecuteNonQuery`	net472	2.13μs	2.47ns	9.58ns	0.157	0.00106	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.23μs	1.08ns	4.03ns	0.0137	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.58μs	0.773ns	2.99ns	0.0127	976 B
master	`CallElasticsearch`	net472	2.58μs	0.927ns	3.47ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.34μs	0.45ns	1.68ns	0.0134	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	0.547ns	2.05ns	0.0137	1.02 KB
master	`CallElasticsearchAsync`	net472	2.66μs	1.14ns	4.27ns	0.167	1.05 KB
#6483	`CallElasticsearch`	net6.0	1.32μs	1.12ns	4.18ns	0.0132	976 B
#6483	`CallElasticsearch`	netcoreapp3.1	1.5μs	0.814ns	3.15ns	0.0127	976 B
#6483	`CallElasticsearch`	net472	2.57μs	2.05ns	7.68ns	0.157	995 B
#6483	`CallElasticsearchAsync`	net6.0	1.33μs	0.714ns	2.77ns	0.0134	952 B
#6483	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	1.07ns	4ns	0.0136	1.02 KB
#6483	`CallElasticsearchAsync`	net472	2.55μs	1.22ns	4.58ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.34μs	0.771ns	2.89ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.63μs	1.16ns	4.34ns	0.0122	952 B
master	`ExecuteAsync`	net472	1.77μs	0.599ns	2.32ns	0.145	915 B
#6483	`ExecuteAsync`	net6.0	1.27μs	0.898ns	3.48ns	0.0135	952 B
#6483	`ExecuteAsync`	netcoreapp3.1	1.63μs	4.72ns	18.3ns	0.0121	952 B
#6483	`ExecuteAsync`	net472	1.83μs	0.39ns	1.51ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.34μs	2.62ns	9.8ns	0.0324	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.35μs	2.18ns	8.15ns	0.0373	2.85 KB
master	`SendAsync`	net472	7.36μs	2.92ns	11.3ns	0.493	3.12 KB
#6483	`SendAsync`	net6.0	4.5μs	2ns	7.76ns	0.0337	2.31 KB
#6483	`SendAsync`	netcoreapp3.1	5.43μs	1.73ns	6.72ns	0.0356	2.85 KB
#6483	`SendAsync`	net472	7.39μs	1.47ns	5.48ns	0.494	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.53μs	0.58ns	2.17ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.22μs	3.9ns	14.6ns	0.0222	1.64 KB
master	`EnrichedLog`	net472	2.72μs	1.05ns	3.93ns	0.25	1.57 KB
#6483	`EnrichedLog`	net6.0	1.47μs	1.66ns	6.2ns	0.0229	1.64 KB
#6483	`EnrichedLog`	netcoreapp3.1	2.05μs	0.75ns	2.71ns	0.0226	1.64 KB
#6483	`EnrichedLog`	net472	2.72μs	1.08ns	4.02ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	115μs	170ns	657ns	0.0576	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	199ns	746ns	0	0	4.28 KB
master	`EnrichedLog`	net472	150μs	150ns	560ns	0.673	0.224	4.46 KB
#6483	`EnrichedLog`	net6.0	118μs	130ns	487ns	0.0588	0	4.28 KB
#6483	`EnrichedLog`	netcoreapp3.1	120μs	174ns	672ns	0.0606	0	4.28 KB
#6483	`EnrichedLog`	net472	151μs	152ns	587ns	0.682	0.227	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.04μs	0.838ns	3.25ns	0.0303	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.2μs	3.31ns	12.8ns	0.0294	2.2 KB
master	`EnrichedLog`	net472	4.84μs	1.46ns	5.65ns	0.319	2.02 KB
#6483	`EnrichedLog`	net6.0	3.11μs	1.19ns	4.6ns	0.031	2.2 KB
#6483	`EnrichedLog`	netcoreapp3.1	4μs	2.39ns	9.26ns	0.028	2.2 KB
#6483	`EnrichedLog`	net472	4.91μs	1.55ns	6.01ns	0.318	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.31μs	0.659ns	2.47ns	0.0164	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.71μs	1.84ns	7.12ns	0.0153	1.14 KB
master	`SendReceive`	net472	2.07μs	0.844ns	2.92ns	0.183	1.16 KB
#6483	`SendReceive`	net6.0	1.36μs	0.835ns	3.23ns	0.0156	1.14 KB
#6483	`SendReceive`	netcoreapp3.1	1.87μs	0.567ns	2.2ns	0.0159	1.14 KB
#6483	`SendReceive`	net472	2.08μs	1.08ns	4.18ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.71μs	0.7ns	2.71ns	0.0217	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.81μs	1.59ns	5.95ns	0.0208	1.65 KB
master	`EnrichedLog`	net472	4.4μs	0.843ns	3.04ns	0.322	2.04 KB
#6483	`EnrichedLog`	net6.0	2.78μs	1.14ns	4.26ns	0.0223	1.6 KB
#6483	`EnrichedLog`	netcoreapp3.1	3.95μs	2.12ns	8.2ns	0.0219	1.65 KB
#6483	`EnrichedLog`	net472	4.42μs	2.69ns	10.1ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6483

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.117	482.49	538.78

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	399ns	0.25ns	0.967ns	0.00805	576 B
master	`StartFinishSpan`	netcoreapp3.1	597ns	0.33ns	1.28ns	0.00752	576 B
master	`StartFinishSpan`	net472	656ns	0.281ns	1.05ns	0.0917	578 B
master	`StartFinishScope`	net6.0	483ns	0.227ns	0.849ns	0.00965	696 B
master	`StartFinishScope`	netcoreapp3.1	676ns	0.441ns	1.71ns	0.00949	696 B
master	`StartFinishScope`	net472	872ns	0.728ns	2.82ns	0.104	658 B
#6483	`StartFinishSpan`	net6.0	408ns	0.142ns	0.55ns	0.00805	576 B
#6483	`StartFinishSpan`	netcoreapp3.1	558ns	0.474ns	1.84ns	0.00757	576 B
#6483	`StartFinishSpan`	net472	683ns	0.19ns	0.686ns	0.0918	578 B
#6483	`StartFinishScope`	net6.0	539ns	0.155ns	0.557ns	0.00972	696 B
#6483	`StartFinishScope`	netcoreapp3.1	672ns	0.316ns	1.18ns	0.0096	696 B
#6483	`StartFinishScope`	net472	911ns	0.334ns	1.29ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	634ns	0.299ns	1.12ns	0.00988	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	906ns	0.326ns	1.26ns	0.00898	696 B
master	`RunOnMethodBegin`	net472	1.07μs	0.503ns	1.95ns	0.104	658 B
#6483	`RunOnMethodBegin`	net6.0	696ns	0.415ns	1.61ns	0.00977	696 B
#6483	`RunOnMethodBegin`	netcoreapp3.1	987ns	0.716ns	2.68ns	0.00943	696 B
#6483	`RunOnMethodBegin`	net472	1.08μs	0.601ns	2.33ns	0.105	658 B

vandonr

I'm interested

link04

Dam look at that time difference!

andrewlock added 3 commits December 24, 2024 11:30

Add strategy generation for unit tests

cf11317

Split by framework in the pipelines

ad6cf20

Exclude netfx from linux unit tests

ebc81cd

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Dec 24, 2024

andrewlock requested a review from a team as a code owner December 24, 2024 12:46

vandonr approved these changes Dec 24, 2024

View reviewed changes

link04 approved these changes Dec 24, 2024

View reviewed changes

andrewlock merged commit 31f9464 into master Dec 31, 2024
59 of 63 checks passed

andrewlock deleted the andrew/ci/parallelize-unit-tests branch December 31, 2024 09:47

github-actions bot added this to the vNext-v3 milestone Dec 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelize unit tests #6483

Parallelize unit tests #6483

andrewlock commented Dec 24, 2024

datadog-ddstaging bot commented Dec 24, 2024 •

edited

Loading

andrewlock commented Dec 24, 2024

andrewlock commented Dec 24, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6483

Raw results

Raw results

vandonr left a comment

link04 left a comment

Parallelize unit tests #6483

Parallelize unit tests #6483

Conversation

andrewlock commented Dec 24, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Dec 24, 2024 • edited Loading

Datadog Report

andrewlock commented Dec 24, 2024

Execution-Time Benchmarks Report ⏱️

andrewlock commented Dec 24, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6483

Raw results

Raw results

vandonr left a comment

Choose a reason for hiding this comment

link04 left a comment

Choose a reason for hiding this comment

datadog-ddstaging bot commented Dec 24, 2024 •

edited

Loading