V4.21.0 proposal #3871

khanayan123 · 2023-12-13T15:15:18Z

Bug Fixes

profiling: add source code integration tags to profiles (Add source code integration tags to profiles #3821)
core: do not report HTTP requests over 5 seconds as errors on Node 20 ([core] Ensure Explicit Timeouts from Underlying Request Socket are Recorded as Errors When Using Node 20 #3853)
core: update protobuf for security reasons (chore: updating protobuf for security reasons #3851)
core: resolve the 0th argument of the restify controller promise (restify: resolve 0th promise arg, not arguments #3818)

Improvements

core: collapse Next.js static resources to reduce cardinality ([core] Collapse Next.js Static Resources to Reduce Cardinality #3809)
civisibility: waits for git to upload and re-request settings when require_git field is true ([ci-visibility] New ITR settings logic #3790)
civisibiity: do not report total code. coverage if itr is enabled ([ci-visibility] Do not report total code coverage unless user has explicitly added code coverage #3828)
profiling: Add DNS events to timeline (PROF-8520: Add DNS events to timeline #3822)
core: add a new http service configuration option enablePropagationWithAmazonHeaders (configurable tracing header w/ aws signed request #3836)
appsec: use existing response header instrumentation to detect Header Injection vulnerability when a unsafe string is written in a header (Detect header injection code vulnerability #3813)
profiling: Add Net events to timeline (PROF-8523: Add Net events to timeline #3832)
core: Partially upgrade instrumentation telemetry from v1 to v2 (Partially upgrade instrumentation telemetry from v1 to v2 #3827)
civisibility: Speed up git unshallow ([ci-visibility] Speed up git unshallow #3839)
core: add a new environment variable to enable span leak detection at either logging or logging + manual gc modes (configurable logging and manual gc of old spans #3849)
core: add a GitHub security policy via SECURITY.md (Create a GitHub security policy via SECURITY.md #3863)
appsec: Apply new rules for header injection detection to prevent false positives (Apply new rules for header injection detection to prevent false positives #3867)
dsm: Add Kafka offset lag to metrics sent by datastreams monitoring (DSM: add kafka offset lag #3761)
profiling: reduce overhead by removing lane logic from profiler library (PROF-8521: Remove lane logic from profiler library #3880)

Features

appsec: add support for schema extraction when calling the waf (API Security integration #3685)
core: add automatic instrumentation support for Aerospike v3.16.2 - v3.16.7, v4, v5 (Instrument Aerospike V3 #3830, instrument aerospike v4, v5 #3804)

collapse Next.js static resource path names

* Config variables. * Add support for schema reporting. * Add support for schema extraction on request addresses. * Pass response's status code as string. --------- Co-authored-by: Ugaitz Urien <[email protected]> Co-authored-by: simon-id <[email protected]>

This makes tests that depend on it having been started less brittle as they don't need to sleep for a guessed duration.

If `repositoryUrl` and `commitSHA` are set in profiler config, then set `git.repository_url` and `git.commit.sha` tags with these values respectively.

* instrument aerospike v4, v5

* Instrument Aerospike v3.16.2 - v3.16.7

…licitly added code coverage (#3828)

- adds a new `http` service configuration option `enablePropagationWithAmazonHeaders` - defaults to `false` which is current behavior - when set to `true` it will allow injecting tracing headers for requests signed via AWS IAM headers - allows customer applications which communicate across services with signed requests to be traced - making this a configurable since there may be use-cases which the change could break

#3841) [core] Do Not Report HTTP Requests Over 5 Seconds as Errors on Node 20

Bump pprof-nodejs version and add integration test for OOM in worker: Worker should exit but process should not abort.

…corded as Errors When Using Node 20 (#3853) * req.socket timeouts recorded as errors node 20 * check for process.send

* adding required headers for in tel * adding required headers for in tel * updating version in unit tests * updating version in unit tests * app_started- extended_heartbeat- header- * updating heartbeat and adding extendedheartbeat * updating telemetry to v2 * worked on addressing telemetry uniform heartbeat * added some tests for instel v2 * cleaned up instel tests * adding dependencies variable check & tests * change to capture initial load for dependencies * Initial retry logic for telemetry * fixing heartbeat tests * adding in extended heartbeat * Cleaned up Index and send data * fixed syntax on telemetry tests * Added extended heartbeat logic * updated app-heartbeat test * modified config file for app-started * updating extended heartbeat to setInterval * adding tests for extendedheartbeat * updating extended heartbeat to setInterval * adding tests for extendedheartbeat * fix app-dependencies-loaded * modified config file to support telemetry app-started * modified config file for telemetry * modified config file to report tracer config values to telemetry * finished updating telemetry event app-started from v1 to v2 * fixed config file * fixed lint errors * attempt to remove yarn.lock changes * fixed telemetry settimeout bug * removed config file changes for now * reverted changes in config.spec.js * fixed bugs * disabled telemetry app-heartbeat test for now * added in origin field for telemetry app-started config * only send integrations through telemetry when there are integrations to be sent * only send integrations through telemetry when there are integrations to be sent * corrected telemetry event name * added back missing telemetry config value * added commit to skip telemetry heartbeat test for now * fixed telemetry send-data * Update packages/dd-trace/src/telemetry/dependencies.js Co-authored-by: Thomas Hunter II <[email protected]> * Update packages/dd-trace/src/telemetry/dependencies.js Co-authored-by: Thomas Hunter II <[email protected]> * Update packages/dd-trace/src/telemetry/index.js Co-authored-by: Thomas Hunter II <[email protected]> * Update packages/dd-trace/src/telemetry/index.js Co-authored-by: Thomas Hunter II <[email protected]> * Update packages/dd-trace/src/telemetry/index.js Co-authored-by: Thomas Hunter II <[email protected]> * Update packages/dd-trace/src/telemetry/send-data.js Co-authored-by: Thomas Hunter II <[email protected]> * Update packages/dd-trace/src/telemetry/index.js Co-authored-by: Thomas Hunter II <[email protected]> * added agentless telemetry data center routing * modified agentless telemetry routing * updated telemetry logs schema * polished instrumentation telemetry PR * removed app-started error field for now --------- Co-authored-by: Crystal Magloire <[email protected]> Co-authored-by: Sam Brenner <[email protected]> Co-authored-by: Sam Brenner <[email protected]> Co-authored-by: bojbrook <[email protected]> Co-authored-by: Crystal Magloire <[email protected]> Co-authored-by: Thomas Hunter II <[email protected]>

… and avoids string concatenation (#3859)

github-actions · 2023-12-13T15:16:08Z

Overall package size

Self size: 5.72 MB
Deduped: 61.82 MB
No deduping: 62.57 MB

Dependency sizes

name	version	self size	total size
@datadog/native-iast-taint-tracking	1.6.4	16.43 MB	16.44 MB
@datadog/native-appsec	5.0.0	15.16 MB	15.17 MB
@datadog/pprof	4.1.0	9.36 MB	10.21 MB
protobufjs	7.2.5	2.77 MB	6.56 MB
@datadog/native-iast-rewriter	2.2.1	2.27 MB	2.36 MB
@opentelemetry/core	1.14.0	872.87 kB	1.47 MB
@datadog/native-metrics	2.0.0	898.77 kB	1.3 MB
@opentelemetry/api	1.4.1	780.32 kB	780.32 kB
import-in-the-middle	1.4.2	41.4 kB	704.79 kB
pprof-format	2.0.7	588.12 kB	588.12 kB
msgpack-lite	0.1.26	201.16 kB	281.59 kB
opentracing	0.14.7	194.81 kB	194.81 kB
semver	7.5.4	93.4 kB	123.8 kB
@datadog/sketches-js	2.1.0	109.9 kB	109.9 kB
lodash.sortby	4.7.0	75.76 kB	75.76 kB
lru-cache	7.14.0	74.95 kB	74.95 kB
ipaddr.js	2.1.0	60.23 kB	60.23 kB
ignore	5.2.4	51.22 kB	51.22 kB
int64-buffer	0.1.10	49.18 kB	49.18 kB
istanbul-lib-coverage	3.2.0	29.34 kB	29.34 kB
lodash.uniq	4.5.0	25.01 kB	25.01 kB
tlhunter-sorted-set	0.1.0	24.94 kB	24.94 kB
limiter	1.1.5	23.17 kB	23.17 kB
dc-polyfill	0.1.2	22.77 kB	22.77 kB
retry	0.13.1	18.85 kB	18.85 kB
lodash.kebabcase	4.1.1	17.75 kB	17.75 kB
node-abort-controller	3.1.1	16.89 kB	16.89 kB
lodash.pick	4.4.0	16.33 kB	16.33 kB
jest-docblock	29.7.0	8.99 kB	12.76 kB
crypto-randomuuid	1.0.0	11.18 kB	11.18 kB
path-to-regexp	0.1.7	6.78 kB	6.78 kB
koalas	1.0.2	6.47 kB	6.47 kB
methods	1.1.2	5.29 kB	5.29 kB
module-details-from-path	1.0.3	4.47 kB	4.47 kB

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2023-12-13T15:24:46Z

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (v4.x@8c9f5bb). Click here to learn what that means.

Additional details and impacted files

@@           Coverage Diff           @@
##             v4.x    #3871   +/-   ##
=======================================
  Coverage        ?   84.43%           
=======================================
  Files           ?      232           
  Lines           ?     9640           
  Branches        ?       33           
=======================================
  Hits            ?     8140           
  Misses          ?     1500           
  Partials        ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pr-commenter · 2023-12-13T15:27:12Z

Benchmarks

Benchmark execution time: 2023-12-14 17:19:03

Comparing candidate commit dba6ad5 in PR branch v4.21.0-proposal with baseline commit 8c9f5bb in branch v4.x.

Found 0 performance improvements and 9 performance regressions! Performance is the same for 769 metrics, 20 unstable metrics.

scenario:appsec-appsec-enabled-16

🟥 instructions [+54.9M instructions; +55.5M instructions] or [+5.162%; +5.219%]

scenario:appsec-appsec-enabled-18

🟥 instructions [+54.2M instructions; +55.1M instructions] or [+5.136%; +5.217%]

scenario:appsec-appsec-enabled-20

🟥 instructions [+53.1M instructions; +54.1M instructions] or [+5.585%; +5.694%]

scenario:appsec-appsec-enabled-with-attacks-20

🟥 instructions [+53.1M instructions; +54.0M instructions] or [+5.121%; +5.212%]

scenario:log-skip-log-16

🟥 execution_time [+16.961ms; +17.557ms] or [+5.215%; +5.398%]

scenario:log-with-debug-16

🟥 execution_time [+17.007ms; +17.576ms] or [+5.229%; +5.404%]

scenario:log-with-error-16

🟥 execution_time [+16.999ms; +17.594ms] or [+5.226%; +5.409%]

scenario:log-without-log-16

🟥 cpu_user_time [+15.737ms; +24.249ms] or [+5.531%; +8.522%]
🟥 execution_time [+16.970ms; +18.160ms] or [+5.383%; +5.761%]

szegedi · 2023-12-13T18:23:37Z

Hey, can you include #3870 too? It fixes a bug introduced by #3859

* fix aws-sdk plugin tests

- this is a convention used by GitHub

…ives (#3867)

* fix failing Aerospike tests on CI

…3879)

Add Kafka offset lag computation for DSM * trace consumer offset with upstream event emitter * trace producer offset with response data --------- Co-authored-by: Piotr Wolski <[email protected]>

- seems no longer compatible as of v4.1.2 - https://github.com/share/sharedb/commit/06cc3877fd9b53d1ddc3ecdd3df11f68fbdbb100\#diff-3fb720f70ea34fb2975fd6c38ac4b6b6131373be2abfa4760dfaae8a25daaee2R11 - Uncaught TypeError: util.hasOwn is not a function

sabrenner and others added 23 commits December 13, 2023 10:09

[core] Collapse Next.js Static Resources to Reduce Cardinality (#3809)

585ec42

collapse Next.js static resource path names

Expose a promise for profiler start (#3817)

dd240af

This makes tests that depend on it having been started less brittle as they don't need to sleep for a guessed duration.

Add source code integration tags to profiles (#3821)

b842082

If `repositoryUrl` and `commitSHA` are set in profiler config, then set `git.repository_url` and `git.commit.sha` tags with these values respectively.

[ci-visibility] Fix debug message for ITR settings (#3825)

dd42d7a

instrument aerospike v4, v5 (#3804)

ea6ccd6

* instrument aerospike v4, v5

Instrument Aerospike V3 (#3830)

54fe1df

* Instrument Aerospike v3.16.2 - v3.16.7

[ci-visibility] New ITR settings logic (#3790)

7c40c38

[ci-visibility] Do not report total code coverage unless user has exp…

0b12655

…licitly added code coverage (#3828)

PROF-8520: Add DNS events to timeline (#3822)

daf81b5

[core] Do Not Report HTTP Requests Over 5 Seconds as Errors on Node 20 (

20ca899

#3841) [core] Do Not Report HTTP Requests Over 5 Seconds as Errors on Node 20

Fix integ tests for DNS (#3844)

94ed847

Detect header injection code vulnerability (#3813)

b3ce51a

Add integration test for OOM in worker (#3843)

7c05480

Bump pprof-nodejs version and add integration test for OOM in worker: Worker should exit but process should not abort.

[core] Ensure Explicit Timeouts from Underlying Request Socket are Re…

8605394

…corded as Errors When Using Node 20 (#3853) * req.socket timeouts recorded as errors node 20 * check for process.send

chore: updating protobuf for security reasons (#3851)

bfbb7a7

PROF-8523: Add Net events to timeline (#3832)

65ba4f1

Emit address and port as separate labels; it's more compact like that…

2988bfe

… and avoids string concatenation (#3859)

[ci-visibility] Speed up git unshallow (#3839)

0a6b7e9

Update rules version to v1.10.0 (#3857)

70383b1

manual logging and garbage collection of old spans (#3849)

73cdb85

wconti27 and others added 3 commits December 14, 2023 09:45

chore: fix aws-sdk failing test suite (#3860)

9a01460

* fix aws-sdk plugin tests

create a security policy via SECURITY.md (#3863)

317c724

- this is a convention used by GitHub

Apply new rules for header injection detection to prevent false posit…

9eb9d18

…ives (#3867)

tlhunter and others added 3 commits December 14, 2023 09:45

restify: resolve 0th promise arg, not arguments (#3818)

84f9eaa

PROF-8523: Bugfix and integration test for Net timeline events (#3870)

5d9f715

Fix failing Aerospike tests (#3873)

8c7ff18

* fix failing Aerospike tests on CI

khanayan123 force-pushed the v4.21.0-proposal branch from d4f8c99 to 0fcfce0 Compare December 14, 2023 14:45

khanayan123 requested a review from tlhunter December 14, 2023 14:54

khanayan123 marked this pull request as ready for review December 14, 2023 14:54

khanayan123 requested a review from a team as a code owner December 14, 2023 14:54

rochdev previously approved these changes Dec 14, 2023

View reviewed changes

szegedi and others added 5 commits December 14, 2023 12:07

Only run DNS and Net timeline events integration tests on Node 16+ (#…

6f4dd76

…3879)

Fix tedious and elasticsearch plugin tests (#3877)

4e14d25

DSM: add kafka offset lag (#3761)

7c32247

Add Kafka offset lag computation for DSM * trace consumer offset with upstream event emitter * trace producer offset with response data --------- Co-authored-by: Piotr Wolski <[email protected]>

PROF-8521: Remove lane logic from profiler library (#3880)

6b39b4b

khanayan123 dismissed rochdev’s stale review via abc557b December 14, 2023 17:07

khanayan123 force-pushed the v4.21.0-proposal branch from 0fcfce0 to abc557b Compare December 14, 2023 17:07

ida613 and others added 2 commits December 14, 2023 12:09

quick telemetry test fix (#3882)

229ccfd

v4.21.0

dba6ad5

khanayan123 force-pushed the v4.21.0-proposal branch from abc557b to dba6ad5 Compare December 14, 2023 17:10

khanayan123 requested a review from rochdev December 14, 2023 17:15

tlhunter approved these changes Dec 14, 2023

View reviewed changes

khanayan123 merged commit bcd805b into v4.x Dec 14, 2023
104 of 107 checks passed

tlhunter deleted the v4.21.0-proposal branch January 19, 2024 22:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

V4.21.0 proposal #3871

V4.21.0 proposal #3871

khanayan123 commented Dec 13, 2023 •

edited

Loading

github-actions bot commented Dec 13, 2023 •

edited

Loading

codecov bot commented Dec 13, 2023 •

edited

Loading

pr-commenter bot commented Dec 13, 2023 •

edited

Loading

szegedi commented Dec 13, 2023

V4.21.0 proposal #3871

V4.21.0 proposal #3871

Conversation

khanayan123 commented Dec 13, 2023 • edited Loading

Bug Fixes

Improvements

Features

github-actions bot commented Dec 13, 2023 • edited Loading

Overall package size

Dependency sizes

codecov bot commented Dec 13, 2023 • edited Loading

Codecov Report

pr-commenter bot commented Dec 13, 2023 • edited Loading

Benchmarks

scenario:appsec-appsec-enabled-16

scenario:appsec-appsec-enabled-18

scenario:appsec-appsec-enabled-20

scenario:appsec-appsec-enabled-with-attacks-20

scenario:log-skip-log-16

scenario:log-with-debug-16

scenario:log-with-error-16

scenario:log-without-log-16

szegedi commented Dec 13, 2023

khanayan123 commented Dec 13, 2023 •

edited

Loading

github-actions bot commented Dec 13, 2023 •

edited

Loading

codecov bot commented Dec 13, 2023 •

edited

Loading

pr-commenter bot commented Dec 13, 2023 •

edited

Loading