Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add aws xray context propagation #3898

Open
wants to merge 59 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
2d39509
fix aws tests
wconti27 Dec 11, 2023
a3f31e7
Merge branch 'master' into conti/fix-aws-sdk-tests
wconti27 Dec 11, 2023
8f7f569
fix kinesis timeouts
wconti27 Dec 11, 2023
bbc5897
fix error
wconti27 Dec 11, 2023
51fe166
increase timeout
wconti27 Dec 11, 2023
107e02b
try again
wconti27 Dec 11, 2023
af16096
no timeout test
wconti27 Dec 11, 2023
aa84b7e
add dsm for sns sqs and kinesis
wconti27 Dec 11, 2023
1923d87
fix failing tests
wconti27 Dec 11, 2023
b3b35b6
add debugging statemtn
wconti27 Dec 12, 2023
c118fd3
fix tests
wconti27 Dec 12, 2023
afcefb1
fix sns tests
wconti27 Dec 12, 2023
f014b2c
fix again
wconti27 Dec 12, 2023
eb13676
try env variable to enable
wconti27 Dec 12, 2023
b0cb24b
fix lint
wconti27 Dec 12, 2023
6449e96
Merge branch 'master' into conti/add-dsm-for-aws-services
wconti27 Dec 12, 2023
62aad55
add message payload size tests for sns / kinesis
wconti27 Dec 12, 2023
43bb3f1
add more payloadSize tests
wconti27 Dec 13, 2023
3f38244
fix kinesis payload size
wconti27 Dec 13, 2023
d8e7bff
fix kinesis tests
wconti27 Dec 13, 2023
2b88ce5
Merge branch 'master' into conti/add-dsm-for-aws-services
wconti27 Dec 13, 2023
564bff3
add context propagation to kinesis
wconti27 Dec 13, 2023
3a692aa
more tests
wconti27 Dec 13, 2023
0e9010d
increase timeout
wconti27 Dec 13, 2023
c4a7d54
rewrite SQS to set DSM checkpoint for each message
wconti27 Dec 14, 2023
8fc0843
change payloadSize calculations
wconti27 Dec 15, 2023
8c5f653
fix payload size error
wconti27 Dec 15, 2023
73519f3
use stream name for checkpoint if arn unavailable
wconti27 Dec 15, 2023
5a11c04
Merge branch 'conti/add-dsm-for-aws-services' into conti/add-context-…
wconti27 Dec 15, 2023
6e0d695
add xray propagation
wconti27 Dec 20, 2023
8cb98ef
remove test that i don't remember adding
wconti27 Dec 20, 2023
b9c8bf3
fix failing tests
wconti27 Dec 20, 2023
bea6673
clean up code
wconti27 Apr 4, 2024
c210127
Merge branch 'master' into conti/add-aws-xray-header-context-extraction
wconti27 Apr 4, 2024
8528904
Revert all changes except propagation class
wconti27 Apr 4, 2024
2b58872
add baggage tags
wconti27 Apr 5, 2024
abf28ca
change encoding / decoding of hashes to match other tracers
wconti27 Apr 5, 2024
7d99016
Revert "change encoding / decoding of hashes to match other tracers"
wconti27 Apr 5, 2024
03008e6
Merge branch 'master' into conti/add-aws-xray-header-context-extraction
wconti27 Dec 13, 2024
f7a542f
add inject method
wconti27 Dec 13, 2024
33ee7de
clean up code
wconti27 Dec 13, 2024
991d69b
add testing
wconti27 Dec 16, 2024
4e306b2
add more test cases
wconti27 Dec 16, 2024
c4de314
fix lint
wconti27 Dec 17, 2024
f6ff0b5
small changes
wconti27 Dec 19, 2024
f427e9b
remove check for aws signed requests
wconti27 Dec 19, 2024
2f702c7
add test for baggage too large
wconti27 Dec 19, 2024
70cbfd8
add more tests to ensure new context propagation works with existing …
wconti27 Dec 26, 2024
4f7fe84
Merge branch 'master' into conti/add-aws-xray-header-context-extraction
wconti27 Jan 3, 2025
504b584
fix lint and other errors
wconti27 Jan 3, 2025
7b07cce
remove aws signature tests
wconti27 Jan 3, 2025
fc1e874
fix signature tests
wconti27 Jan 3, 2025
246994e
fix function call
wconti27 Jan 3, 2025
397984c
another fix
wconti27 Jan 3, 2025
ba57fba
another fixx
wconti27 Jan 3, 2025
f2be5c3
remove test
wconti27 Jan 3, 2025
6acb364
be more careful
wconti27 Jan 6, 2025
42ba094
fix logging
wconti27 Jan 6, 2025
57d68a7
fix error
wconti27 Jan 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion packages/datadog-plugin-http/src/client.js
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,11 @@ class HttpClientPlugin extends ClientPlugin {
}

shouldInjectTraceHeaders (options, uri) {
if (hasAmazonSignature(options) && !this.config.enablePropagationWithAmazonHeaders) {
if (hasAmazonSignature(options) && !this.tracer._config.tracePropagationStyle.inject.includes('xray')) {
log.debug(
'AWS Signature detected on HTTP request, skipping injecting headers. To enable header injection' +
' for signed AWS requests, please set DD_TRACE_PROPAGATION_STYLE=["xray", "datadog"]'
)
return false
}

Expand Down
44 changes: 0 additions & 44 deletions packages/datadog-plugin-http/test/client.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -1093,50 +1093,6 @@ describe('Plugin', () => {
})
})

describe('with config enablePropagationWithAmazonHeaders enabled', () => {
let config

beforeEach(() => {
config = {
enablePropagationWithAmazonHeaders: true
}

return agent.load('http', config)
.then(() => {
http = require(pluginToBeLoaded)
express = require('express')
})
})

it('should inject tracing header into AWS signed request', done => {
const app = express()

app.get('/', (req, res) => {
try {
expect(req.get('x-datadog-trace-id')).to.be.a('string')
expect(req.get('x-datadog-parent-id')).to.be.a('string')

res.status(200).send()

done()
} catch (e) {
done(e)
}
})

appListener = server(app, port => {
const req = http.request({
port,
headers: {
Authorization: 'AWS4-HMAC-SHA256 ...'
}
})

req.end()
})
})
})

describe('with validateStatus configuration', () => {
let config

Expand Down
2 changes: 1 addition & 1 deletion packages/dd-trace/src/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ const otelDdEnvMapping = {
OTEL_LOGS_EXPORTER: undefined
}

const VALID_PROPAGATION_STYLES = new Set(['datadog', 'tracecontext', 'b3', 'b3 single header', 'none'])
const VALID_PROPAGATION_STYLES = new Set(['datadog', 'tracecontext', 'b3', 'b3 single header', 'xray', 'none'])

const VALID_LOG_LEVELS = new Set(['debug', 'info', 'warn', 'error'])

Expand Down
173 changes: 172 additions & 1 deletion packages/dd-trace/src/opentracing/propagation/text_map.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,24 @@ const b3SampledKey = 'x-b3-sampled'
const b3FlagsKey = 'x-b3-flags'
const b3HeaderKey = 'b3'
const sqsdHeaderHey = 'x-aws-sqsd-attr-_datadog'
// AWS X-Ray specific constants
const xrayHeaderKey = 'x-amzn-trace-id'
const xrayRootKey = 'root'
const xrayRootPrefix = '1-'
const xrayParentKey = 'parent'
const xraySampledKey = 'sampled'
const xrayE2EStartTimeKey = 't0'
const xraySelfKey = 'self'
const xrayOriginKey = '_dd.origin'
const xrayDefaultE2EStartTime = '00000000'
const xrayMaxAdditionalBaggageBytes = 256
//
const b3HeaderExpr = /^(([0-9a-f]{16}){1,2}-[0-9a-f]{16}(-[01d](-[0-9a-f]{16})?)?|[01d])$/i
const baggageExpr = new RegExp(`^${baggagePrefix}(.+)$`)
const tagKeyExpr = /^_dd\.p\.[\x21-\x2b\x2d-\x7e]+$/ // ASCII minus spaces and commas
const tagValueExpr = /^[\x20-\x2b\x2d-\x7e]*$/ // ASCII minus commas
const ddKeys = [traceKey, spanKey, samplingKey, originKey]
const b3Keys = [b3TraceKey, b3SpanKey, b3ParentKey, b3SampledKey, b3FlagsKey, b3HeaderKey]
const b3Keys = [b3TraceKey, b3SpanKey, b3ParentKey, b3SampledKey, b3FlagsKey, b3HeaderKey, xrayHeaderKey]
const logKeys = ddKeys.concat(b3Keys)
const traceparentExpr = /^([a-f0-9]{2})-([a-f0-9]{32})-([a-f0-9]{16})-([a-f0-9]{2})(-.*)?$/i
const traceparentKey = 'traceparent'
Expand All @@ -60,6 +72,7 @@ class TextMapPropagator {
this._injectB3MultipleHeaders(spanContext, carrier)
this._injectB3SingleHeader(spanContext, carrier)
this._injectTraceparent(spanContext, carrier)
this._injectAwsXrayContext(spanContext, carrier)
wconti27 marked this conversation as resolved.
Show resolved Hide resolved

if (injectCh.hasSubscribers) {
injectCh.publish({ spanContext, carrier })
Expand Down Expand Up @@ -252,6 +265,81 @@ class TextMapPropagator {
carrier.tracestate = ts.toString()
}

_injectAwsXrayContext (spanContext, carrier) {
// injects AWS Trace Header (X-Amzn-Trace-Id) to carrier
//
// ex: 'Root=1-00000000-00000000fffffffffffffffe;Parent=ffffffffffffffff;...
// ...Sampled=1;_dd.origin=fakeOrigin;baggage_k=baggage_v...;
//
// Header Format:
// 'Root=1-' (always uses '1-')
// 8 hexadecimal characters representing start time of the operation
// '-'
// 24 hexadecimal characters representing trace id
// ';'
// 'Parent='
// 16 hexadecimal characters representing parent span id
// 'Sampled='
// int of [0, 1] representing sampling decision
// Additional baggage in k=v pairs separated by ';' delimiter/
//
// based off: https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader
if (!this._hasPropagationStyle('inject', 'xray')) return

// TODO: Do we have access to a start time? Java does
const e2eStart = this._getEndToEndStartTime(spanContext.start_ms)

let str = (
xrayRootKey + '=' +
xrayRootPrefix +
e2eStart + '-' +
spanContext._traceId.toString().padStart(24, '0') +
';' + xrayParentKey + '=' +
spanContext._spanId.toString().padStart(16, '0')
)

if (spanContext?._sampling?.priority) {
str += ';' + xraySampledKey + '=' + (spanContext._sampling.priority > 0 ? '1' : '0')
}

// limit trace header size to the size of the necessary bytes plus 256
const maxAdditionalCapacity = str.length + xrayMaxAdditionalBaggageBytes

const origin = spanContext._trace.origin
if (origin) {
this._addXrayBaggage(str, xrayOriginKey, origin, maxAdditionalCapacity)
}
if (e2eStart !== xrayDefaultE2EStartTime) {
this._addXrayBaggage(str, xrayE2EStartTimeKey, e2eStart.toString(), maxAdditionalCapacity)
}

for (const [key, value] of Object.entries(spanContext._baggageItems)) {
if (!this._isReservedXrayKey(key)) {
str = this._addXrayBaggage(str, key, value, maxAdditionalCapacity)
}
}

carrier[xrayHeaderKey] = str
}

_getEndToEndStartTime (start) {
if (!start) return xrayDefaultE2EStartTime

const e2eStart = start > 0 ? start : Date.now() / 1000
return e2eStart.toString().padStart(8, '0')
}

_isReservedXrayKey (key) {
return [xrayRootKey, xrayParentKey, xrayOriginKey, xrayE2EStartTimeKey, xraySampledKey].includes(key.toLowerCase())
}

_addXrayBaggage (str, key, value, maxCapacity) {
if (str.length + key.length + value.toString().length + 2 <= maxCapacity) {
str += `;${key}=${value}`
}
return str
}

_hasPropagationStyle (mode, name) {
return this._config.tracePropagationStyle[mode].includes(name)
}
Expand Down Expand Up @@ -303,6 +391,9 @@ class TextMapPropagator {
case 'b3 single header': // TODO: delete in major after singular "b3"
extractedContext = this._extractB3SingleContext(carrier)
break
case 'xray':
extractedContext = this._extractAwsXrayContext(carrier)
break
case 'b3':
if (this._config.tracePropagationStyle.otelPropagators) {
// TODO: should match "b3 single header" in next major
Expand Down Expand Up @@ -692,6 +783,86 @@ class TextMapPropagator {
return spanContext._traceId.toString(16)
}

_extractAwsXrayContext (carrier) {
if (!this._hasPropagationStyle('extract', 'xray')) {
return null
}

if (carrier[xrayHeaderKey]) {
const parsedHeader = this._parseAWSTraceHeader(carrier[xrayHeaderKey])

let traceId
let spanId
let samplingPriority
let ddOrigin
const baggage = {}

if (!(xrayRootKey in parsedHeader)) {
return null
}

for (const key in parsedHeader) {
const value = parsedHeader[key]
if (key === xrayRootKey) {
const awsTraceIdSegments = value.split('-')
for (let i = 0; i < awsTraceIdSegments.length; i++) {
if (awsTraceIdSegments[i] === '1') {
continue
} else if (i === 1 && awsTraceIdSegments.length === 3) {
continue
} else if (i === 2 && awsTraceIdSegments.length === 3) {
// trace id padded by: "00000000"
traceId = awsTraceIdSegments[i].substring(8)
}
}
} else if (key === xrayParentKey) {
spanId = value
} else if (key === xraySampledKey) {
samplingPriority = parseInt(value)
} else if (key === xrayOriginKey) {
ddOrigin = String(value)
} else if (key === xraySelfKey) {
// self is added by load balancers and should be ignored
continue
} else if (key === xrayE2EStartTimeKey) {
baggage.startMs = parseInt(value)
} else {
baggage[key] = value
}
}

if (traceId && spanId) {
const spanContext = new DatadogSpanContext({
traceId: id(traceId, 16),
spanId: id(spanId, 16),
sampling: { samplingPriority },
baggageItems: baggage
})
if (ddOrigin) {
spanContext._trace.origin = ddOrigin
}
return spanContext
}
return null
} else {
return null
}
}

_parseAWSTraceHeader (header) {
// parses AWSTraceHeader string to object
// ex: 'Root=1-00000000-00000000fffffffffffffffe;Parent=ffffffffffffffff;Sampled=1;_dd.origin=fakeOrigin;
const obj = {}
const keyValuePairs = header.split(';')
keyValuePairs.forEach(pair => {
const [key, value] = pair.split('=')
if (key && value) {
obj[key.toLowerCase()] = value.toLowerCase()
}
})
return obj
}

static _convertOtelContextToDatadog (traceId, spanId, traceFlag, ts, meta = {}) {
const origin = null
let samplingPriority = traceFlag
Expand Down
Loading
Loading