Consolidate all integration test targets into a single target #6501

andrewlock · 2025-01-06T10:10:37Z

Summary of changes

Reason for change

We previously had RunWindowsIntegrationTests, RunLinuxIntegrationTests, RunOsxIntegrationTests, each with subtle differences. Overall that adds complexity which isn't really necessary

Implementation details

Combine everything into a single target, and just specialize the filter instead to reduce the code

Test coverage

Tested locally on Windows, and ran a CI test, but OSX side is untested currently.

Other details

Stacked on

Build the test sample projects in a separate CI stage #6498

https://datadoghq.atlassian.net/browse/LANGPLAT-66

andrewlock · 2025-01-06T10:44:58Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (69ms)  : 66, 71
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (979ms)  : 956, 1002
     .   : milestone, 979,
    master - mean (980ms)  : 955, 1005
     .   : milestone, 980,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (108ms)  : 106, 110
     .   : milestone, 108,
    master - mean (107ms)  : 106, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (680ms)  : 668, 691
     .   : milestone, 680,
    master - mean (678ms)  : 662, 693
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (92ms)  : 90, 94
     .   : milestone, 92,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (633ms)  : 612, 655
     .   : milestone, 633,
    master - mean (639ms)  : 624, 655
     .   : milestone, 639,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (195ms)  : 190, 199
     .   : milestone, 195,
    master - mean (195ms)  : 190, 200
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (1,100ms)  : 1066, 1134
     .   : milestone, 1100,
    master - mean (1,107ms)  : 1079, 1136
     .   : milestone, 1107,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (278ms)  : 275, 282
     .   : milestone, 278,
    master - mean (281ms)  : 276, 286
     .   : milestone, 281,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (871ms)  : 844, 898
     .   : milestone, 871,
    master - mean (878ms)  : 850, 906
     .   : milestone, 878,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6501) - mean (267ms)  : 262, 272
     .   : milestone, 267,
    master - mean (269ms)  : 265, 273
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (6501) - mean (847ms)  : 814, 881
     .   : milestone, 847,
    master - mean (855ms)  : 820, 891
     .   : milestone, 855,

datadog-ddstaging · 2025-01-06T10:55:58Z

Datadog Report

Branch report: andrew/ci/combine-integration-test-targets
Commit report: 8fd1a94
Test service: dd-trace-dotnet

❌ 1 Failed (0 Known Flaky), 249628 Passed, 2132 Skipped, 19h 42m 2.68s Total Time

❌ Failed Tests (1)

AzureFunctionsTests+IsolatedRuntimeV4.SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests - Details
Expand for error
```
 The sample did not exit in 600000ms. Memory dump taken: True. Killing process.
```

andrewlock · 2025-01-06T11:07:13Z

Benchmarks Report for appsec 🐌

Benchmarks for #6501 compared to master:

1 benchmarks are faster, with geometric mean 1.784
2 benchmarks have fewer allocations
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	203μs	118ns	457ns	2.64	0	188.76 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	298μs	181ns	677ns	2.54	0	196.09 KB
master	`AllCycleSimpleBody`	net472	275μs	836ns	3.24μs	35.8	2.02	225.78 KB
master	`AllCycleMoreComplexBody`	net6.0	207μs	103ns	385ns	2.7	0	192.26 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	321μs	198ns	768ns	2.72	0	199.51 KB
master	`AllCycleMoreComplexBody`	net472	275μs	236ns	881ns	36.4	2.06	229.3 KB
master	`ObjectExtractorSimpleBody`	net6.0	138ns	0.135ns	0.507ns	0.00396	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	199ns	0.186ns	0.695ns	0.00367	0	272 B
master	`ObjectExtractorSimpleBody`	net472	164ns	0.162ns	0.625ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	2.85μs	1.19ns	4.29ns	0.0532	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.71μs	2.97ns	10.3ns	0.0508	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.62μs	3.11ns	12ns	0.603	0.0054	3.8 KB
#6501	`AllCycleSimpleBody`	net6.0	207μs	224ns	836ns	2.64	0	188.76 KB
#6501	`AllCycleSimpleBody`	netcoreapp3.1	311μs	171ns	640ns	2.64	0	196.09 KB
#6501	`AllCycleSimpleBody`	net472	270μs	201ns	777ns	35.8	2.02	225.78 KB
#6501	`AllCycleMoreComplexBody`	net6.0	208μs	91.5ns	342ns	2.72	0	192.26 KB
#6501	`AllCycleMoreComplexBody`	netcoreapp3.1	320μs	138ns	518ns	2.72	0	199.5 KB
#6501	`AllCycleMoreComplexBody`	net472	272μs	127ns	460ns	36.4	2.05	229.3 KB
#6501	`ObjectExtractorSimpleBody`	net6.0	137ns	0.0784ns	0.293ns	0.00391	0	280 B
#6501	`ObjectExtractorSimpleBody`	netcoreapp3.1	192ns	0.174ns	0.653ns	0.00371	0	272 B
#6501	`ObjectExtractorSimpleBody`	net472	162ns	0.119ns	0.463ns	0.0446	0	281 B
#6501	`ObjectExtractorMoreComplexBody`	net6.0	2.89μs	1.31ns	4.54ns	0.0536	0	3.78 KB
#6501	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.78μs	3.56ns	13.8ns	0.0491	0	3.69 KB
#6501	`ObjectExtractorMoreComplexBody`	net472	3.64μs	3ns	11.6ns	0.602	0.00544	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EncodeArgs`	net6.0	37.3μs	24.3ns	87.6ns	0.448	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	54.4μs	37.4ns	135ns	0.433	0	32.4 KB
master	`EncodeArgs`	net472	67.3μs	87.6ns	339ns	5.16	0.067	32.5 KB
master	`EncodeLegacyArgs`	net6.0	71.4μs	29.6ns	107ns	0	0	2.14 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	107μs	426ns	1.65μs	0	0	2.14 KB
master	`EncodeLegacyArgs`	net472	156μs	106ns	410ns	0.313	0	2.15 KB
#6501	`EncodeArgs`	net6.0	37.4μs	19.7ns	73.7ns	0.445	0	32.4 KB
#6501	`EncodeArgs`	netcoreapp3.1	54.2μs	52.4ns	196ns	0.455	0	32.4 KB
#6501	`EncodeArgs`	net472	67μs	112ns	435ns	5.16	0.067	32.5 KB
#6501	`EncodeLegacyArgs`	net6.0	78.4μs	139ns	539ns	0	0	2.14 KB
#6501	`EncodeLegacyArgs`	netcoreapp3.1	108μs	107ns	416ns	0	0	2.14 KB
#6501	`EncodeLegacyArgs`	net472	158μs	146ns	546ns	0.319	0	2.15 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunWafRealisticBenchmark`	net6.0	176μs	170ns	657ns	0	2.44 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	188μs	248ns	962ns	0	2.39 KB
master	`RunWafRealisticBenchmark`	net472	203μs	141ns	547ns	0.304	2.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	114μs	50.9ns	190ns	0	1.47 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	124μs	180ns	696ns	0	1.46 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	133μs	207ns	801ns	0.201	1.48 KB
#6501	`RunWafRealisticBenchmark`	net6.0	174μs	120ns	465ns	0	2.44 KB
#6501	`RunWafRealisticBenchmark`	netcoreapp3.1	188μs	223ns	865ns	0	2.39 KB
#6501	`RunWafRealisticBenchmark`	net472	198μs	122ns	472ns	0.297	2.46 KB
#6501	`RunWafRealisticBenchmarkWithAttack`	net6.0	116μs	46.3ns	179ns	0	1.47 KB
#6501	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	123μs	181ns	701ns	0	1.46 KB
#6501	`RunWafRealisticBenchmarkWithAttack`	net472	132μs	37.2ns	144ns	0.198	1.49 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Faster 🎉 More allocations ⚠️

Faster 🎉 in #6501

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	1.784	617,500.00	346,050.00

More allocations ⚠️ in #6501

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	59.23 KB	61.77 KB	2.54 KB	4.28%

Fewer allocations 🎉 in #6501

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	267.8 KB	254.17 KB	-13.63 KB	-5.09%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	316.45 KB	263.61 KB	-52.84 KB	-16.70%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	60.6μs	763ns	7.63μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	60.5μs	666ns	6.49μs	42.64 KB
master	`StringConcatBenchmark`	net472	38.2μs	99ns	370ns	59.23 KB
master	`StringConcatAspectBenchmark`	net6.0	320μs	5.58μs	53μs	267.8 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	624μs	3.44μs	21.5μs	316.45 KB
master	`StringConcatAspectBenchmark`	net472	285μs	6.29μs	62.3μs	278.53 KB
#6501	`StringConcatBenchmark`	net6.0	60μs	749ns	7.49μs	43.44 KB
#6501	`StringConcatBenchmark`	netcoreapp3.1	55.1μs	310ns	2.01μs	42.64 KB
#6501	`StringConcatBenchmark`	net472	37μs	181ns	723ns	61.77 KB
#6501	`StringConcatAspectBenchmark`	net6.0	321μs	1.73μs	11.2μs	254.17 KB
#6501	`StringConcatAspectBenchmark`	netcoreapp3.1	349μs	1.91μs	10.8μs	263.61 KB
#6501	`StringConcatAspectBenchmark`	net472	284μs	5.73μs	55.6μs	278.53 KB

andrewlock · 2025-01-06T11:20:51Z

Benchmarks Report for tracer 🐌

Benchmarks for #6501 compared to master:

2 benchmarks are faster, with geometric mean 1.162
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.98μs	45.7ns	332ns	0.0162	0.00808	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.5μs	57.5ns	340ns	0.0256	0.0102	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	33.2ns	124ns	1.05	0.311	0.0981	6.22 KB
#6501	`StartStopWithChild`	net6.0	7.95μs	41.9ns	206ns	0.0148	0.00741	0	5.61 KB
#6501	`StartStopWithChild`	netcoreapp3.1	9.97μs	54.1ns	301ns	0.0197	0.00986	0	5.8 KB
#6501	`StartStopWithChild`	net472	16.4μs	45.3ns	175ns	1.04	0.31	0.098	6.22 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	506μs	353ns	1.37μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	632μs	250ns	967ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	869μs	1.05μs	4.05μs	0.431	3.3 KB
#6501	`WriteAndFlushEnrichedTraces`	net6.0	483μs	250ns	935ns	0	2.7 KB
#6501	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	659μs	398ns	1.49μs	0	2.7 KB
#6501	`WriteAndFlushEnrichedTraces`	net472	847μs	700ns	2.71μs	0.422	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	158μs	1.16μs	11.6μs	0.15	14.47 KB
master	`SendRequest`	netcoreapp3.1	172μs	991ns	8.35μs	0.179	17.27 KB
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#6501	`SendRequest`	net6.0	150μs	819ns	5.85μs	0.171	14.47 KB
#6501	`SendRequest`	netcoreapp3.1	170μs	990ns	9.34μs	0.157	17.27 KB
#6501	`SendRequest`	net472	0.00276ns	0.000879ns	0.00329ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	575μs	2.42μs	12.6μs	0.571	0	0	41.71 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	695μs	3.73μs	23μs	0.334	0	0	41.83 KB
master	`WriteAndFlushEnrichedTraces`	net472	836μs	2.94μs	11μs	8.36	2.64	0.44	53.27 KB
#6501	`WriteAndFlushEnrichedTraces`	net6.0	588μs	3.42μs	29.4μs	0.556	0	0	41.79 KB
#6501	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	708μs	3.47μs	15.1μs	0.336	0	0	41.68 KB
#6501	`WriteAndFlushEnrichedTraces`	net472	853μs	3.2μs	12.4μs	8.17	2.45	0.408	53.3 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.31μs	0.891ns	3.45ns	0.0144	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.7μs	1.9ns	7.37ns	0.0135	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.06μs	1.79ns	6.94ns	0.157	0.00104	987 B
#6501	`ExecuteNonQuery`	net6.0	1.23μs	1.21ns	4.69ns	0.014	0	1.02 KB
#6501	`ExecuteNonQuery`	netcoreapp3.1	1.82μs	1.11ns	4.16ns	0.0135	0	1.02 KB
#6501	`ExecuteNonQuery`	net472	2.09μs	1.16ns	4.17ns	0.156	0.00105	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.13μs	0.368ns	1.33ns	0.0136	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.6μs	0.939ns	3.64ns	0.0128	976 B
master	`CallElasticsearch`	net472	2.56μs	1.16ns	4.32ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.28μs	0.807ns	3.12ns	0.0135	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.6μs	0.485ns	1.75ns	0.0137	1.02 KB
master	`CallElasticsearchAsync`	net472	2.59μs	1.05ns	4.06ns	0.167	1.05 KB
#6501	`CallElasticsearch`	net6.0	1.14μs	0.583ns	2.18ns	0.0137	976 B
#6501	`CallElasticsearch`	netcoreapp3.1	1.59μs	5.41ns	20.9ns	0.0134	976 B
#6501	`CallElasticsearch`	net472	2.53μs	1.28ns	4.8ns	0.157	995 B
#6501	`CallElasticsearchAsync`	net6.0	1.36μs	0.944ns	3.53ns	0.0137	952 B
#6501	`CallElasticsearchAsync`	netcoreapp3.1	1.66μs	0.635ns	2.37ns	0.0142	1.02 KB
#6501	`CallElasticsearchAsync`	net472	2.64μs	1.85ns	7.17ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.35μs	1.7ns	6.37ns	0.013	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	1.1ns	4.27ns	0.013	952 B
master	`ExecuteAsync`	net472	1.78μs	0.755ns	2.82ns	0.145	915 B
#6501	`ExecuteAsync`	net6.0	1.39μs	2.16ns	8.1ns	0.0131	952 B
#6501	`ExecuteAsync`	netcoreapp3.1	1.61μs	0.385ns	1.44ns	0.0129	952 B
#6501	`ExecuteAsync`	net472	1.8μs	0.356ns	1.33ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.43μs	1.93ns	7.48ns	0.031	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.19μs	2.13ns	7.68ns	0.0392	2.85 KB
master	`SendAsync`	net472	7.29μs	1.62ns	6.29ns	0.495	3.12 KB
#6501	`SendAsync`	net6.0	4.38μs	1.39ns	5.39ns	0.0306	2.31 KB
#6501	`SendAsync`	netcoreapp3.1	5.17μs	1.84ns	6.9ns	0.0363	2.85 KB
#6501	`SendAsync`	net472	7.46μs	1.73ns	6.24ns	0.494	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.46μs	0.861ns	3.22ns	0.0233	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.16μs	1.17ns	4.55ns	0.0215	1.64 KB
master	`EnrichedLog`	net472	2.63μs	2.11ns	7.89ns	0.249	1.57 KB
#6501	`EnrichedLog`	net6.0	1.51μs	1.02ns	3.81ns	0.0231	1.64 KB
#6501	`EnrichedLog`	netcoreapp3.1	2.27μs	1.61ns	6.03ns	0.0216	1.64 KB
#6501	`EnrichedLog`	net472	2.81μs	1.42ns	5.31ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	116μs	97.3ns	377ns	0.0573	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	149ns	576ns	0	0	4.28 KB
master	`EnrichedLog`	net472	153μs	154ns	597ns	0.689	0.23	4.46 KB
#6501	`EnrichedLog`	net6.0	118μs	99.1ns	371ns	0.0585	0	4.28 KB
#6501	`EnrichedLog`	netcoreapp3.1	121μs	137ns	512ns	0	0	4.28 KB
#6501	`EnrichedLog`	net472	150μs	199ns	772ns	0.685	0.228	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.24μs	1.33ns	4.97ns	0.0307	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.19μs	1.29ns	5.01ns	0.0295	2.2 KB
master	`EnrichedLog`	net472	4.91μs	1.73ns	6.7ns	0.321	2.02 KB
#6501	`EnrichedLog`	net6.0	3.01μs	1.25ns	4.69ns	0.0314	2.2 KB
#6501	`EnrichedLog`	netcoreapp3.1	4.11μs	2.61ns	9.75ns	0.0287	2.2 KB
#6501	`EnrichedLog`	net472	4.92μs	2.05ns	7.95ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.46μs	0.882ns	3.18ns	0.0161	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.75μs	1.12ns	4.33ns	0.0157	1.14 KB
master	`SendReceive`	net472	2.08μs	1.18ns	4.57ns	0.183	1.16 KB
#6501	`SendReceive`	net6.0	1.38μs	0.617ns	2.39ns	0.0158	1.14 KB
#6501	`SendReceive`	netcoreapp3.1	1.77μs	0.835ns	3.01ns	0.0151	1.14 KB
#6501	`SendReceive`	net472	2.1μs	1.76ns	6.35ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.77μs	1.07ns	3.99ns	0.0221	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.98μs	3.38ns	13.1ns	0.0219	1.65 KB
master	`EnrichedLog`	net472	4.4μs	2.9ns	10.8ns	0.323	2.04 KB
#6501	`EnrichedLog`	net6.0	2.69μs	1.24ns	4.79ns	0.0225	1.6 KB
#6501	`EnrichedLog`	netcoreapp3.1	3.84μs	1.97ns	7.63ns	0.0212	1.65 KB
#6501	`EnrichedLog`	net472	4.28μs	1.92ns	7.45ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6501

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.120	923.65	824.38

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	439ns	0.349ns	1.35ns	0.00804	576 B
master	`StartFinishSpan`	netcoreapp3.1	578ns	0.623ns	2.41ns	0.00773	576 B
master	`StartFinishSpan`	net472	630ns	0.361ns	1.4ns	0.0915	578 B
master	`StartFinishScope`	net6.0	484ns	0.338ns	1.31ns	0.00972	696 B
master	`StartFinishScope`	netcoreapp3.1	714ns	0.311ns	1.16ns	0.00928	696 B
master	`StartFinishScope`	net472	924ns	1.24ns	4.81ns	0.105	658 B
#6501	`StartFinishSpan`	net6.0	420ns	0.345ns	1.34ns	0.00804	576 B
#6501	`StartFinishSpan`	netcoreapp3.1	551ns	0.454ns	1.7ns	0.00801	576 B
#6501	`StartFinishSpan`	net472	656ns	0.781ns	3.02ns	0.0915	578 B
#6501	`StartFinishScope`	net6.0	514ns	1.44ns	5.56ns	0.00973	696 B
#6501	`StartFinishScope`	netcoreapp3.1	734ns	0.444ns	1.72ns	0.00962	696 B
#6501	`StartFinishScope`	net472	824ns	0.899ns	3.48ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6501

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.205	725.24	602.03

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	725ns	0.549ns	2.13ns	0.00979	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	980ns	0.528ns	1.97ns	0.00921	696 B
master	`RunOnMethodBegin`	net472	1.16μs	0.367ns	1.37ns	0.104	658 B
#6501	`RunOnMethodBegin`	net6.0	602ns	0.58ns	2.25ns	0.00955	696 B
#6501	`RunOnMethodBegin`	netcoreapp3.1	914ns	0.923ns	3.58ns	0.00944	696 B
#6501	`RunOnMethodBegin`	net472	1.1μs	0.551ns	2.14ns	0.104	658 B

bouwkast

LGTM, worked for Windows

If you need/want someone to test the Mac I'll have mine setup later in the week hopefully

… target

andrewlock · 2025-01-10T23:49:59Z

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6501) (11.204M)   : 0, 11204319
    master (11.124M)   : 0, 11123640
    benchmarks/2.9.0 (11.045M)   : 0, 11045405

    section Automatic
    This PR (6501) (7.210M)   : 0, 7210469
    master (7.224M)   : 0, 7224362
    benchmarks/2.9.0 (7.885M)   : 0, 7885346

    section Trace stats
    master (7.529M)   : 0, 7528716

    section Manual
    master (11.114M)   : 0, 11113915

    section Manual + Automatic
    This PR (6501) (6.628M)   : 0, 6627941
    master (6.693M)   : 0, 6693005

    section DD_TRACE_ENABLED=0
    master (10.133M)   : 0, 10132801

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6501) (9.872M)   : 0, 9872358
    master (9.628M)   : 0, 9628045
    benchmarks/2.9.0 (9.586M)   : 0, 9586476

    section Automatic
    This PR (6501) (6.431M)   : 0, 6431307
    master (6.529M)   : 0, 6528563

    section Trace stats
    master (6.826M)   : 0, 6826395

    section Manual
    master (9.661M)   : 0, 9660963

    section Manual + Automatic
    This PR (6501) (5.843M)   : 0, 5843262
    master (5.992M)   : 0, 5991612

    section DD_TRACE_ENABLED=0
    master (8.972M)   : 0, 8971944

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6501) (10.073M)   : 0, 10073151
    master (9.804M)   : 0, 9803983

    section Automatic
    This PR (6501) (6.606M)   : 0, 6606020
    master (6.481M)   : 0, 6480741

    section Trace stats
    master (7.095M)   : 0, 7094744

    section Manual
    master (10.165M)   : 0, 10165163

    section Manual + Automatic
    This PR (6501) (6.218M)   : 0, 6217728
    master (6.197M)   : 0, 6196772

    section DD_TRACE_ENABLED=0
    master (9.340M)   : 0, 9339860

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 6, 2025

andrewlock requested a review from a team as a code owner January 6, 2025 10:10

andrewlock force-pushed the andrew/ci/combine-integration-test-targets branch from c9f6d1b to 100ea4c Compare January 7, 2025 13:52

bouwkast approved these changes Jan 7, 2025

View reviewed changes

andrewlock force-pushed the andrew/ci/build-samples-separately branch from 14b7e65 to 86ce608 Compare January 7, 2025 14:16

andrewlock requested review from a team as code owners January 7, 2025 14:16

andrewlock force-pushed the andrew/ci/combine-integration-test-targets branch from 100ea4c to 875b4c5 Compare January 7, 2025 14:16

andrewlock force-pushed the andrew/ci/build-samples-separately branch from 86ce608 to 1af72cc Compare January 8, 2025 16:18

andrewlock force-pushed the andrew/ci/combine-integration-test-targets branch from 875b4c5 to d813115 Compare January 8, 2025 16:18

andrewlock force-pushed the andrew/ci/build-samples-separately branch from 1af72cc to 0e3bc1f Compare January 9, 2025 09:49

andrewlock force-pushed the andrew/ci/combine-integration-test-targets branch from d813115 to df22af2 Compare January 9, 2025 09:49

andrewlock force-pushed the andrew/ci/build-samples-separately branch from 0e3bc1f to 0c3dab6 Compare January 9, 2025 13:17

Try to consolidate all the integration test definitions into a single…

8fd1a94

… target

andrewlock force-pushed the andrew/ci/build-samples-separately branch from 0c3dab6 to c89b2c6 Compare January 9, 2025 15:02

andrewlock force-pushed the andrew/ci/combine-integration-test-targets branch from df22af2 to 8fd1a94 Compare January 10, 2025 10:14

kevingosse approved these changes Jan 10, 2025

View reviewed changes

tonyredondo approved these changes Jan 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consolidate all integration test targets into a single target #6501

Consolidate all integration test targets into a single target #6501

andrewlock commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading

Raw results

Raw results

Raw results

Faster 🎉 in #6501

More allocations ⚠️ in #6501

Fewer allocations 🎉 in #6501

Raw results

andrewlock commented Jan 6, 2025 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6501

Raw results

Faster 🎉 in #6501

Raw results

bouwkast left a comment

andrewlock commented Jan 10, 2025

Consolidate all integration test targets into a single target #6501

Are you sure you want to change the base?

Consolidate all integration test targets into a single target #6501

Conversation

andrewlock commented Jan 6, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Jan 6, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Jan 6, 2025 • edited Loading

Datadog Report

❌ Failed Tests (1)

andrewlock commented Jan 6, 2025 • edited Loading

Benchmarks Report for appsec 🐌

Benchmark details

Raw results

Raw results

Raw results

Faster 🎉 in #6501

More allocations ⚠️ in #6501

Fewer allocations 🎉 in #6501

Raw results

andrewlock commented Jan 6, 2025 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6501

Raw results

Faster 🎉 in #6501

Raw results

bouwkast left a comment

Choose a reason for hiding this comment

andrewlock commented Jan 10, 2025

Throughput/Crank Report ⚡

andrewlock commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading

andrewlock commented Jan 6, 2025 •

edited

Loading