Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: get bpf clang from pinned nixpkgs #987

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

JakeHillion
Copy link
Contributor

@JakeHillion JakeHillion commented Nov 26, 2024

This PR represents a move towards reproducible builds starting with the clang
version, which causes possible complications due to subtle differences in BPF
assembly generation. This specifically moves us from 18.1.3 (ubuntu), bundled
with the CI system, to 18.1.8 (Nix's version). Moreover, it pins the version
in perpetuity. If the Nix caches dry up for some reason, we will build clang
and its dependencies from source, making this specific part of our CI fully
reproducible.

This PR does not go any further in making the build reproducible, only the BPF
ASM generation part of the meson build. We can see this version reported
correctly in the CI logs. The BPF generation is very specific to clang version
and this change pulls a more recently patched version & ensures it never
changes on old commits.

While this change doesn't force any user of the repository to use Nix in the
build, it begins to offer the option. Other open source repositories[1] are
taking advantage of Nix's ability to offer devshells in a more complete way,
which enable reproducible builds regardless of the end user's operating system
and package setup, that don't interfere with whatever they have setup already.
This is future work - one of the most frustrating things as a developer is when
the CI fails and you can't reproduce it locally. nix develop fixes this, and
should come down the line to make our builds fully reproducible.

Test plan:

  • CI

[1] https://github.com/bpftrace/bpftrace/blob/master/flake.nix

@JakeHillion JakeHillion force-pushed the pr987 branch 2 times, most recently from f6f3163 to 7113936 Compare November 26, 2024 17:12
@likewhatevs
Copy link
Contributor

I generally don't like mixing package managers (particularly with complex things, like clang), but all for this if it solves a problem tbh.

@likewhatevs
Copy link
Contributor

Would prefer a ppa or tar installer if possible but kinda get if this is the only way around having to compile something super new (although we do that with bpftool via meson and that kinda works so idk).

@JakeHillion
Copy link
Contributor Author

I generally don't like mixing package managers (particularly with complex things, like clang), but all for this if it solves a problem tbh.

The problem this solves is reproducibility, which our CI/dev environments will really benefit from. bpftrace does this in a fantastic way with a set of builds/development shells all controlled by a flake.nix with perfectly pinned versions and a good matrix (https://github.com/bpftrace/bpftrace/blob/e65b5f280ddfdd4442700c52e8f79163c81ef6cc/flake.nix). The developers are super happy with the environment provided and it.

Nix does a good job in this change too. We're not changing anything about the system except for putting new content in /nix. The clang version there only links against things in /nix, and doesn't affect anything else on the system - it's not in PATH, and not available to be accidentally (mis)used. The only reference is in the single environment variable.

Would prefer a ppa or tar installer if possible but kinda get if this is the only way around having to compile something super new (although we do that with bpftool via meson and that kinda works so idk).

Installing complex things like clang from a PPA can cause a mess, this is why Nix works so nicely here. No system change is made, only a package added to a store and all of its dependencies, which only it can see. tar has similar problems, unless we're pulling something statically linked. I think Nix is the best choice here :) - building from source should be a last resort, and nix will do that if this revision falls out of the cache!

Will add the proper body and stuff to this PR now it works and we can discuss further if necessary.

@likewhatevs
Copy link
Contributor

idk, rly hesitant to add another package manager into the mix. this is another moving part/step in the "why are things broke" process.

https://apt.llvm.org has ppa's with relevant clang versions and it isn't clear how that + github matrices are insufficient for preventing the problems we face today.

we have the bpf-next job for tracking issues with upstream bpf and could add another w/ clang's equivalent if that is the concern.

@likewhatevs
Copy link
Contributor

I wanna say at least hide nix behind meson if we're gonna do this, but also maybe stick to ppa's etc.

@JakeHillion JakeHillion changed the title WIP: ci: get stress-ng and bpf clang from pinned nixpkgs ci: get bpf clang from pinned nixpkgs Dec 2, 2024
likewhatevs and others added 4 commits December 2, 2024 10:41
This PR represents a move towards reproducible builds starting with the clang
version, which causes possible complications due to subtle differences in BPF
assembly generation. This specifically moves us from `18.1.3 (ubuntu)`, bundled
with the CI system, to `18.1.8` (Nix's version). Moreover, it pins the version
in perpetuity. If the Nix caches dry up for some reason, we will build clang
and its dependencies from source, making this specific part of our CI fully
reproducible.

This PR does not go any further in making the build reproducible, only the BPF
ASM generation part of the meson build. We can see this version reported
correctly in the CI logs. The BPF generation is very specific to clang version
and this change pulls a more recently patched version & ensures it never
changes on old commits.

While this change doesn't force any user of the repository to use Nix in the
build, it begins to offer the option. Other open source repositories[1] are
taking advantage of Nix's ability to offer devshells in a more complete way,
which enable reproducible builds regardless of the end user's operating system
and package setup, that don't interfere with whatever they have setup already.
This is future work - one of the most frustrating things as a developer is when
the CI fails and you can't reproduce it locally. `nix develop` fixes this, and
should come down the line to make our builds fully reproducible.

Test plan:
- CI

[1] https://github.com/bpftrace/bpftrace/blob/master/flake.nix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants