[EXPORTER] Support handling retry-able errors for OTLP/HTTP

[chusitoo]

Contributes to #2049

Changes

This change introduces a retry mechanism for OTLP/HTTP for select failures, mimicking the same exponential backoff approach used in OTLP/gRPC.

• Add support to set retry values via environment variables.
• Enabled by default, using the same configuration values as in OTel dotnet, java and js.
• Users can opt-out of the retry capabilities by zeroing out any (or all) of the retry settings.
• Similar to OTLP/gRPC, retries are transparent to the user so only the last attempt is "bubbled up" as the actual response.

The changes to support retries for OTLP/gRPC exporter are addressed in #3219

For significant contributions please make sure you have completed the following items:

• CHANGELOG.md updated for non-trivial changes
• Unit tests have been added
• Changes in public API reviewed

[netlify]

✅ Deploy Preview for opentelemetry-cpp-api-docs canceled.

Name	Link
🔨 Latest commit	082597a
🔍 Latest deploy log	https://app.netlify.com/sites/opentelemetry-cpp-api-docs/deploys/6772c02f4c1c9800081accd4

[netlify]

✅ Deploy Preview for opentelemetry-cpp-api-docs canceled.

Name	Link
🔨 Latest commit	905e7ae
🔍 Latest deploy log	https://app.netlify.com/sites/opentelemetry-cpp-api-docs/deploys/678a5a452eb906000855d49b

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.78%. Comparing base (02cda51) to head (905e7ae).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3223      +/-   ##
==========================================
+ Coverage   87.71%   87.78%   +0.07%     
==========================================
  Files         198      198              
  Lines        6273     6308      +35     
==========================================
+ Hits         5502     5537      +35     
  Misses        771      771              

Files with missing lines	Coverage Δ
sdk/src/common/env_variables.cc	99.03% <100.00%> (+0.50%)	⬆️

[marcalff]

Excellent work.

The only part remaining at this point is to add a feature flag:

In CMakeList.txt, define a new option WITH_OTLP_RETRY_PREVIEW, default OFF

The same option name should be fine for both HTTP and gRPC.

In api/CMakeList.txt, add a target_compile_definitions (aka, a define), for ENABLE_OTLP_RETRY_PREVIEW

Use ifdef ENABLE_OTLP_RETRY_PREVIEW to protect the code path for this feature. A good place could be inside the implementation of IsRetryable().

Adjust the unit tests using the same ifdef, to make sure they pass when compiling with and without WITH_OTLP_RETRY_PREVIEW.

In CI, add WITH_OTLP_RETRY_PREVIEW=ON in all maintainer builds.

Having a feature flag is needed because:

• the environment variables are not in the spec yet, so this code is considered experimental and subject to change
• the code itself is new, so this mitigates risks for people in production with the OTLP exporter. Adoption will be voluntary since the default is OFF.

Much later, the feature flag will be ON by default, and then removed, assuming we get this in the spec at some point, to declare the whole thing stable.

[marcalff]

LGTM

Excellent work, thanks for the feature.

[marcalff]

Ok to ignore CI failurews for test BasicCurlHttpTests.FinishInAsyncCallback, due to #3172

[marcalff]

@chusitoo

Sorry for the merge conflicts.

This should be trivial, as we both added some members in the same place, let me know if you need help with resolving.

Please note that there is a new constructor in ext, for HttpClient, so your changes will be needed there as well.

I will review again after conflict resolution anyway.

[marcalff]

@chusitoo

Please resolve the merge conflicts, or indicate if you need help with this.

Thanks (and sorry for the conflicts)

[chusitoo]

@chusitoo

Please resolve the merge conflicts, or indicate if you need help with this.

Thanks (and sorry for the conflicts)

Apologies for the delay, having a hard time finding time for this during the week.

Conflicts resolved but I have to look into addressing a concern raised in this discussion

[marcalff]

nit, optional:

Please use ifdef ENABLE_OTLP_RETRY_PREVIEW as well in HttpClient::doRetrySessions(), to protect this code path.

[marcalff]

To summarize remaining outstanding issues for this review:

I have a nit comment on HttpClient::doRetrySessions(bool)
@chusitoo please take a look.

HttpClient::doRetrySessions(bool) now takes a bool report_all parameter.

@owent
This should resolve the concern you raised, please take a look.

[owent]

LGTM now, thanks.