Skip the flaky XUnit named pipes tests #6528

andrewlock · 2025-01-10T15:47:30Z

Summary of changes

Skip the named pipes tests for now

Reason for change

These are very flaky, and aren't an explicitly supported scenario.

Implementation details

Just disable the test - we can look at re-enabling them again once/if we move to an aspnetcore-based internal agent, in the hope that solves the flake

Test coverage

Less now

Other details

There is a separate CI Vis flake issue which appears to be a deadlock on arm64, that's being looked at separately

andrewlock · 2025-01-10T16:19:46Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6528) - mean (69ms)  : 65, 72
     .   : milestone, 69,
    master - mean (69ms)  : 66, 71
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6528) - mean (978ms)  : 954, 1003
     .   : milestone, 978,
    master - mean (980ms)  : 949, 1011
     .   : milestone, 980,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6528) - mean (108ms)  : 106, 110
     .   : milestone, 108,
    master - mean (108ms)  : 106, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6528) - mean (676ms)  : 658, 694
     .   : milestone, 676,
    master - mean (681ms)  : 665, 697
     .   : milestone, 681,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6528) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6528) - mean (633ms)  : 617, 650
     .   : milestone, 633,
    master - mean (636ms)  : 618, 654
     .   : milestone, 636,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6528) - mean (194ms)  : 189, 200
     .   : milestone, 194,
    master - mean (195ms)  : 189, 200
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6528) - mean (1,101ms)  : 1065, 1137
     .   : milestone, 1101,
    master - mean (1,105ms)  : 1078, 1131
     .   : milestone, 1105,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6528) - mean (279ms)  : 274, 283
     .   : milestone, 279,
    master - mean (279ms)  : 275, 283
     .   : milestone, 279,

    section CallTarget+Inlining+NGEN
    This PR (6528) - mean (871ms)  : 845, 898
     .   : milestone, 871,
    master - mean (875ms)  : 850, 900
     .   : milestone, 875,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6528) - mean (267ms)  : 263, 271
     .   : milestone, 267,
    master - mean (268ms)  : 264, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6528) - mean (856ms)  : 834, 879
     .   : milestone, 856,
    master - mean (853ms)  : 824, 881
     .   : milestone, 853,

datadog-ddstaging · 2025-01-10T16:19:46Z

Datadog Report

Branch report: andrew/ci/skip-flaky-named-pipes-tests
Commit report: 0516efa
Test service: dd-trace-dotnet

✅ 0 Failed, 239075 Passed, 1975 Skipped, 18h 55m 50.77s Total Time

andrewlock · 2025-01-10T17:43:06Z

Benchmarks Report for tracer 🐌

Benchmarks for #6528 compared to master:

2 benchmarks are faster, with geometric mean 1.138
1 benchmarks are slower, with geometric mean 1.148
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.05μs	40.4ns	249ns	0.0116	0.00385	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	53.2ns	255ns	0.021	0.0105	0	5.8 KB
master	`StartStopWithChild`	net472	16.2μs	44.4ns	172ns	1.04	0.315	0.0968	6.2 KB
#6528	`StartStopWithChild`	net6.0	8.05μs	45ns	277ns	0.015	0.00752	0	5.61 KB
#6528	`StartStopWithChild`	netcoreapp3.1	10.2μs	54.3ns	302ns	0.0205	0.0102	0	5.8 KB
#6528	`StartStopWithChild`	net472	16.3μs	38.6ns	149ns	1.05	0.307	0.105	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	487μs	296ns	1.15μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	667μs	498ns	1.93μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	855μs	620ns	2.32μs	0.425	3.3 KB
#6528	`WriteAndFlushEnrichedTraces`	net6.0	490μs	406ns	1.46μs	0	2.7 KB
#6528	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	649μs	524ns	1.96μs	0	2.7 KB
#6528	`WriteAndFlushEnrichedTraces`	net472	850μs	751ns	2.81μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	157μs	1.07μs	10.4μs	0.149	14.47 KB
master	`SendRequest`	netcoreapp3.1	172μs	1.15μs	11.3μs	0.177	17.27 KB
master	`SendRequest`	net472	0.00244ns	0.000878ns	0.00329ns	0	0 b
#6528	`SendRequest`	net6.0	156μs	1.13μs	11.2μs	0.143	14.47 KB
#6528	`SendRequest`	netcoreapp3.1	188μs	1.51μs	15.1μs	0.17	17.27 KB
#6528	`SendRequest`	net472	0.00334ns	0.00121ns	0.00469ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	553μs	2.66μs	11.6μs	0.568	0	0	41.55 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	661μs	3.48μs	18.1μs	0.321	0	0	41.69 KB
master	`WriteAndFlushEnrichedTraces`	net472	853μs	4.01μs	15μs	8.13	2.57	0.428	53.3 KB
#6528	`WriteAndFlushEnrichedTraces`	net6.0	598μs	3.47μs	29.3μs	0.324	0	0	41.6 KB
#6528	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	706μs	4.05μs	31.4μs	0.347	0	0	41.62 KB
#6528	`WriteAndFlushEnrichedTraces`	net472	886μs	4.27μs	16.5μs	8.19	2.59	0.431	53.32 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.33μs	1.2ns	4.65ns	0.0147	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	2.39ns	9.26ns	0.0133	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.14μs	2.51ns	9.39ns	0.156	0.00107	987 B
#6528	`ExecuteNonQuery`	net6.0	1.28μs	0.797ns	2.87ns	0.014	0	1.02 KB
#6528	`ExecuteNonQuery`	netcoreapp3.1	1.86μs	1.67ns	6.48ns	0.013	0	1.02 KB
#6528	`ExecuteNonQuery`	net472	2.19μs	1.95ns	7.54ns	0.157	0.0011	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.16μs	0.576ns	2ns	0.0134	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.59μs	0.479ns	1.79ns	0.0128	976 B
master	`CallElasticsearch`	net472	2.53μs	2.14ns	7.99ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.27μs	0.633ns	2.45ns	0.0133	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.66μs	1.71ns	6.17ns	0.0139	1.02 KB
master	`CallElasticsearchAsync`	net472	2.61μs	1.8ns	6.95ns	0.166	1.05 KB
#6528	`CallElasticsearch`	net6.0	1.22μs	0.487ns	1.76ns	0.0135	976 B
#6528	`CallElasticsearch`	netcoreapp3.1	1.55μs	0.826ns	3.09ns	0.0132	976 B
#6528	`CallElasticsearch`	net472	2.55μs	1.47ns	5.71ns	0.157	995 B
#6528	`CallElasticsearchAsync`	net6.0	1.29μs	0.726ns	2.72ns	0.0135	952 B
#6528	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	4.65ns	18ns	0.0134	1.02 KB
#6528	`CallElasticsearchAsync`	net472	2.6μs	1.54ns	5.56ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.25μs	0.313ns	1.13ns	0.0132	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.68μs	0.882ns	3.42ns	0.0123	952 B
master	`ExecuteAsync`	net472	1.8μs	0.69ns	2.58ns	0.145	915 B
#6528	`ExecuteAsync`	net6.0	1.21μs	0.418ns	1.56ns	0.0134	952 B
#6528	`ExecuteAsync`	netcoreapp3.1	1.67μs	0.581ns	2.17ns	0.0126	952 B
#6528	`ExecuteAsync`	net472	1.82μs	0.409ns	1.58ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.57μs	2.38ns	9.22ns	0.0321	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.31μs	1.63ns	6.11ns	0.0372	2.85 KB
master	`SendAsync`	net472	7.3μs	2ns	7.49ns	0.495	3.12 KB
#6528	`SendAsync`	net6.0	4.42μs	1.59ns	5.95ns	0.031	2.31 KB
#6528	`SendAsync`	netcoreapp3.1	5.31μs	1.4ns	5.41ns	0.0371	2.85 KB
#6528	`SendAsync`	net472	7.42μs	2.47ns	9.57ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.53μs	0.809ns	3.03ns	0.0227	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.3μs	1.09ns	4.07ns	0.0218	1.64 KB
master	`EnrichedLog`	net472	2.61μs	1.19ns	4.6ns	0.249	1.57 KB
#6528	`EnrichedLog`	net6.0	1.55μs	0.563ns	2.03ns	0.0225	1.64 KB
#6528	`EnrichedLog`	netcoreapp3.1	2.19μs	2.05ns	7.65ns	0.0218	1.64 KB
#6528	`EnrichedLog`	net472	2.68μs	2.05ns	7.94ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	119μs	159ns	617ns	0.0592	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	117ns	454ns	0	0	4.28 KB
master	`EnrichedLog`	net472	154μs	172ns	645ns	0.688	0.229	4.46 KB
#6528	`EnrichedLog`	net6.0	118μs	133ns	514ns	0	0	4.28 KB
#6528	`EnrichedLog`	netcoreapp3.1	121μs	149ns	577ns	0	0	4.28 KB
#6528	`EnrichedLog`	net472	153μs	107ns	414ns	0.692	0.231	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.02μs	0.853ns	3.19ns	0.0303	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.26μs	1.42ns	5.32ns	0.0277	2.2 KB
master	`EnrichedLog`	net472	4.97μs	3.56ns	13.3ns	0.319	2.02 KB
#6528	`EnrichedLog`	net6.0	2.94μs	0.948ns	3.42ns	0.031	2.2 KB
#6528	`EnrichedLog`	netcoreapp3.1	4.28μs	1.13ns	4.4ns	0.03	2.2 KB
#6528	`EnrichedLog`	net472	4.75μs	1.28ns	4.97ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.43μs	0.963ns	3.73ns	0.0162	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.67μs	1.6ns	6.2ns	0.0151	1.14 KB
master	`SendReceive`	net472	2.09μs	1.54ns	5.96ns	0.183	1.16 KB
#6528	`SendReceive`	net6.0	1.45μs	1.06ns	4.11ns	0.0159	1.14 KB
#6528	`SendReceive`	netcoreapp3.1	1.73μs	1.88ns	7.27ns	0.0153	1.14 KB
#6528	`SendReceive`	net472	2.01μs	1.41ns	5.63ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.72μs	1.08ns	4.16ns	0.0218	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.86μs	1.15ns	4.45ns	0.0211	1.65 KB
master	`EnrichedLog`	net472	4.34μs	2.67ns	10.3ns	0.323	2.04 KB
#6528	`EnrichedLog`	net6.0	2.75μs	1.07ns	4.13ns	0.022	1.6 KB
#6528	`EnrichedLog`	netcoreapp3.1	3.73μs	1.38ns	5.36ns	0.0208	1.65 KB
#6528	`EnrichedLog`	net472	4.25μs	1.54ns	5.96ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6528

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.148	395.74	454.19

Faster 🎉 in #6528

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.149	567.54	494.11
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.127	730.93	648.54

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	395ns	0.241ns	0.932ns	0.008	576 B
master	`StartFinishSpan`	netcoreapp3.1	633ns	0.302ns	1.13ns	0.00795	576 B
master	`StartFinishSpan`	net472	730ns	0.534ns	2.07ns	0.0917	578 B
master	`StartFinishScope`	net6.0	568ns	0.351ns	1.36ns	0.00965	696 B
master	`StartFinishScope`	netcoreapp3.1	740ns	0.576ns	2.23ns	0.00931	696 B
master	`StartFinishScope`	net472	844ns	0.601ns	2.33ns	0.104	658 B
#6528	`StartFinishSpan`	net6.0	454ns	0.173ns	0.671ns	0.00799	576 B
#6528	`StartFinishSpan`	netcoreapp3.1	614ns	3.35ns	18ns	0.00776	576 B
#6528	`StartFinishSpan`	net472	649ns	0.198ns	0.74ns	0.0915	578 B
#6528	`StartFinishScope`	net6.0	494ns	0.181ns	0.7ns	0.00977	696 B
#6528	`StartFinishScope`	netcoreapp3.1	746ns	0.547ns	2.05ns	0.00922	696 B
#6528	`StartFinishScope`	net472	884ns	0.498ns	1.93ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	599ns	0.432ns	1.67ns	0.00996	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	979ns	0.836ns	3.24ns	0.00935	696 B
master	`RunOnMethodBegin`	net472	1.11μs	0.854ns	3.31ns	0.104	658 B
#6528	`RunOnMethodBegin`	net6.0	603ns	0.219ns	0.791ns	0.00999	696 B
#6528	`RunOnMethodBegin`	netcoreapp3.1	884ns	2.28ns	8.53ns	0.00943	696 B
#6528	`RunOnMethodBegin`	net472	1.13μs	0.284ns	1.1ns	0.104	658 B

Skip the flaky named pipes tests

0516efa

andrewlock added the area:tests unit tests, integration tests label Jan 10, 2025

andrewlock requested review from a team as code owners January 10, 2025 15:47

tonyredondo approved these changes Jan 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip the flaky XUnit named pipes tests #6528

Skip the flaky XUnit named pipes tests #6528

andrewlock commented Jan 10, 2025

andrewlock commented Jan 10, 2025

datadog-ddstaging bot commented Jan 10, 2025 •

edited

Loading

andrewlock commented Jan 10, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6528

Faster 🎉 in #6528

Raw results

Raw results

Skip the flaky XUnit named pipes tests #6528

Are you sure you want to change the base?

Skip the flaky XUnit named pipes tests #6528

Conversation

andrewlock commented Jan 10, 2025

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Jan 10, 2025

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Jan 10, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 10, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6528

Faster 🎉 in #6528

Raw results

Raw results

datadog-ddstaging bot commented Jan 10, 2025 •

edited

Loading