Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[serverless] upload r2r artifacts to azure #6543

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open

[serverless] upload r2r artifacts to azure #6543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
10220f1
make sure `upload_to_azure` includes `r2r` artifacts
duncanista Jan 13, 2025
d28c801
typo missing r2r
duncanista Jan 13, 2025
f952263
maybe this fixes the files not being found for R2R package?
duncanista Jan 13, 2025
8df2c7c
another typo not using a variable
duncanista Jan 13, 2025
73d99be
get artifact from same place for r2r
duncanista Jan 13, 2025
2595d15
see if adding the profiler to r2r avoids issues
duncanista Jan 13, 2025
662be38
typo in matrix name
duncanista Jan 13, 2025
e837894
add another artifact env var to see if that makes the ZipMonitoringHo…
duncanista Jan 14, 2025
6340013
add universal artifact suffix for arm
duncanista Jan 14, 2025
0b3bd4c
Merge branch 'master' into jordan.gonzalez/r2r/add-r2r-artifacts-to-a…
duncanista Jan 14, 2025
fe7fbd4
try to do it programatically instead
duncanista Jan 14, 2025
0d99ca5
Merge branch 'jordan.gonzalez/r2r/add-r2r-artifacts-to-azure-pipeline…
duncanista Jan 14, 2025
b48994f
typo in build steps
duncanista Jan 14, 2025
140c27f
somehow doing contains didnt work, trying eq
duncanista Jan 14, 2025
7da60c0
i cannot find a way to do this in one line :(
duncanista Jan 14, 2025
3f3e6fb
again
duncanista Jan 14, 2025
e895108
try this other way of doing it
duncanista Jan 14, 2025
c090510
remove condition from zipmonitoringhome
duncanista Jan 14, 2025
aa8b919
again try with quotes now
duncanista Jan 14, 2025
177ed65
arch typo
duncanista Jan 14, 2025
610ada1
some testing
duncanista Jan 15, 2025
f923184
it needs quotes
duncanista Jan 15, 2025
3ce0707
another try
duncanista Jan 15, 2025
4454b58
typo
duncanista Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 48 additions & 9 deletions .azure-pipelines/ultimate-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,7 @@ stages:
parameters:
target: builder
baseImage: debian
command: "BuildManagedTracerHomeR2R ExtractDebugInfoLinux"
command: "BuildManagedTracerHomeR2R ExtractDebugInfoLinux ValidateNativeTracerGlibcCompatibility"
retryCountForRunCommand: 1

- publish: $(monitoringHome)
Expand Down Expand Up @@ -537,6 +537,11 @@ stages:
managedBaseImage: debian
artifactSuffix: linux-x64
useNativeSdkVersion: true
x64-r2r:
baseImage: centos7
managedBaseImage: debian
artifactSuffix: linux-x64-r2r
useNativeSdkVersion: true
Comment on lines +540 to +544
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty certain you don't need to do this. The tracer is a native binary, so r2r doesn't make any sense 🤔 Plus you don't actually want to use the profiler in the lambda layer anyway AFAIK?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm just trying to figure out where I should update so the pipeline package-x goes through, but it might be a good idea to package it just in case – still trying to figure it out

alpine:
baseImage: alpine
managedBaseImage: alpine
Expand Down Expand Up @@ -601,6 +606,9 @@ stages:
x64:
baseImage: debian
artifactSuffix: linux-x64
x64-r2r:
baseImage: debian
artifactSuffix: linux-x64-r2r
alpine:
baseImage: alpine
artifactSuffix: linux-musl-x64
Expand All @@ -617,14 +625,14 @@ stages:
# but only the musl package if we're building alpine package
- task: DownloadPipelineArtifact@2
displayName: Download tracer linux native binary (linux-x64)
condition: eq(variables['artifactSuffix'], 'linux-x64')
condition: or(eq(variables['artifactSuffix'], 'linux-x64'), eq(variables['artifactSuffix'], 'linux-x64-r2r'))
inputs:
artifact: linux-tracer-home-linux-x64
artifact: linux-tracer-home-$(artifactSuffix)
path: $(monitoringHome)

- task: DownloadPipelineArtifact@2
displayName: Download profiler linux native binary (linux-x64)
condition: eq(variables['artifactSuffix'], 'linux-x64')
condition: or(eq(variables['artifactSuffix'], 'linux-x64'), eq(variables['artifactSuffix'], 'linux-x64-r2r'))
inputs:
artifact: linux-profiler-home-linux-x64
path: $(monitoringHome)
Expand Down Expand Up @@ -671,7 +679,7 @@ stages:
build: true
target: builder
baseImage: $(baseImage)
command: "ZipMonitoringHome"
command: "${{format('ZipMonitoringHome --ReadyToRun {0}', eq($(variables['artifactSuffix']), 'linux-x64-r2r'))}}"
retryCountForRunCommand: 1

- publish: $(artifacts)/linux-x64
Expand Down Expand Up @@ -883,6 +891,9 @@ stages:
arm64:
baseImage: debian
artifactSuffix: linux-arm64
arm64-r2r:
baseImage: debian
artifactSuffix: linux-arm64-r2r
alpine:
baseImage: alpine
artifactSuffix: linux-musl-arm64
Expand All @@ -897,16 +908,18 @@ stages:

# Download _both_ musl and glibc if we're building the universal package
# but only the musl package if we're building alpine package
#
# R2R only requires the tracer, as the profiler is not used there yet.
- task: DownloadPipelineArtifact@2
displayName: Download tracer arm64 native binary
condition: eq(variables['artifactSuffix'], 'linux-arm64')
condition: or(eq(variables['artifactSuffix'], 'linux-arm64'), eq(variables['artifactSuffix'], 'linux-arm64-r2r'))
inputs:
artifact: linux-tracer-home-linux-arm64
artifact: linux-tracer-home-$(artifactSuffix)
path: $(monitoringHome)

- task: DownloadPipelineArtifact@2
displayName: Download profiler arm64 native binary
condition: eq(variables['artifactSuffix'], 'linux-arm64')
condition: or(eq(variables['artifactSuffix'], 'linux-arm64'), eq(variables['artifactSuffix'], 'linux-arm64-r2r'))
inputs:
artifact: linux-profiler-home-linux-arm64
path: $(monitoringHome)
Expand Down Expand Up @@ -953,7 +966,7 @@ stages:
build: true
target: builder
baseImage: $(baseImage)
command: "ZipMonitoringHome"
command: "${{format('ZipMonitoringHome --ReadyToRun {0}', eq($(variables['artifactSuffix']), 'linux-arm64-r2r'))}}"
retryCountForRunCommand: 1

- publish: $(artifacts)/linux-arm64
Expand Down Expand Up @@ -4380,6 +4393,7 @@ stages:
artifact: linux-universal-symbols-linux-arm64
path: $(Build.ArtifactStagingDirectory)/symbols/linux-musl-arm64

# dd-dotnet
- task: DownloadPipelineArtifact@2
displayName: Download dd-dotnet linux-x64 symbols
inputs:
Expand Down Expand Up @@ -4484,6 +4498,31 @@ stages:
inputs:
artifact: windows-msi-x64
path: $(Build.ArtifactStagingDirectory)

# We also don't include R2R packages in the release artifacts
- task: DownloadPipelineArtifact@2
displayName: Download linux x64 r2r packages
inputs:
artifact: linux-packages-linux-x64-r2r
path: $(Build.ArtifactStagingDirectory)
Comment on lines +4503 to +4507
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part lgtm 🙂


- task: DownloadPipelineArtifact@2
displayName: Download linux Arm64 r2r packages
inputs:
artifact: linux-packages-linux-arm64-r2r
path: $(Build.ArtifactStagingDirectory)

- task: DownloadPipelineArtifact@2
displayName: Download linux tracer x64 r2r symbols
inputs:
artifact: linux-tracer-symbols-linux-x64-r2r
path: $(Build.ArtifactStagingDirectory)/symbols

- task: DownloadPipelineArtifact@2
displayName: Download linux tracer Arm64 r2r symbols
inputs:
artifact: linux-tracer-symbols-linux-arm64-r2r
path: $(Build.ArtifactStagingDirectory)/symbols

- bash: |
# Write tracer version number to version.txt
Expand Down
7 changes: 6 additions & 1 deletion .gitlab/download-serverless-artifacts.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,18 @@ target_dir=artifacts
mkdir -p $target_dir

branchName="refs/heads/$CI_COMMIT_BRANCH"
branchArg=$CI_COMMIT_BRANCH
if [ -n "$CI_COMMIT_TAG" ]; then
branchName="refs/tags/$CI_COMMIT_TAG"
branchArg=$CI_COMMIT_TAG
fi

echo "Looking for azure devops PR builds for branch '$branchName' for commit '$CI_COMMIT_SHA' to start"

# We should _definitely_ have the build by now, so if not, there probably won't be one
# Check for PR builds first (as more likely to be "full" builds)
allBuildsForPrUrl="https://dev.azure.com/datadoghq/dd-trace-dotnet/_apis/build/builds?api-version=7.1&definitions=54&\$top=100&queryOrder=queueTimeDescending&reasonFilter=pullRequest"
buildId=$(curl -sS $allBuildsForPrUrl | jq --arg version $CI_COMMIT_SHA --arg branch $CI_COMMIT_BRANCH '.value[] | select(.triggerInfo["pr.sourceBranch"] == $branch and .triggerInfo["pr.sourceSha"] == $version) | .id' | head -n 1)
buildId=$(curl -sS $allBuildsForPrUrl | jq --arg version $CI_COMMIT_SHA --arg branch $branchArg '.value[] | select(.triggerInfo["pr.sourceBranch"] == $branch and .triggerInfo["pr.sourceSha"] == $version) | .id' | head -n 1)

if [ -z "${buildId}" ]; then
echo "No PR builds found for commit '$CI_COMMIT_SHA' on branch '$branchName'. Checking for standalone builds..."
Expand Down
3 changes: 2 additions & 1 deletion tracer/build/_build/Build.Steps.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -891,7 +891,8 @@ async Task DownloadWafVersion(string libddwafVersion = null, string uncompressFo
var nfpm = Nfpm.Value;

var (arch, ext) = GetUnixArchitectureAndExtension();
var workingDirectory = ArtifactsDirectory / $"linux-{UnixArchitectureIdentifier}";
var workingDirectory = ArtifactsDirectory / $"linux-{UnixArchitectureIdentifier}{
(ReadyToRun ? "-r2r" : "")}";
EnsureCleanDirectory(workingDirectory);

const string packageName = "datadog-dotnet-apm";
Expand Down
3 changes: 3 additions & 0 deletions tracer/build/_build/Build.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,9 @@ partial class Build : NukeBuild
[Parameter("RuntimeIdentifier sets the target platform for ReadyToRun assemblies in 'PublishManagedTracerR2R'." +
"See https://learn.microsoft.com/en-us/dotnet/core/rid-catalog")]
string RuntimeIdentifier { get; }

[Parameter("Enable ReadyToRun configuration")]
bool ReadyToRun = false;

public Build()
{
Expand Down
Loading