rpm-ostree: use prefetched RPMs, add hermetic option #928
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chmeliik
force-pushed
the
rpm-ostree-plus-cachi2-rpms
branch
from
42bd22c to
323fbfc
Compare
chmeliik
added a commit
to chmeliik/centos-bootc-tmp
that referenced
this pull request
Apr 9, 2024
ref konflux-ci/build-definitions#928 Signed-off-by: Adam Cmiel <[email protected]>
chmeliik
added a commit
to chmeliik/centos-bootc-tmp
that referenced
this pull request
Apr 9, 2024
ref konflux-ci/build-definitions#928 Signed-off-by: Adam Cmiel <[email protected]>
chmeliik
force-pushed
the
rpm-ostree-plus-cachi2-rpms
branch
3 times, most recently
from
322c237 to
5d7b80c
Compare
chmeliik
changed the title
|
Tested in chmeliik/centos-bootc-tmp#7, chmeliik/centos-bootc-tmp#8 and chmeliik/centos-bootc-tmp#9 Prefetch + hermetic works, hermetic without prefetch fails as expected, non-hermetic without prefetch still works. Some notes.
|
|
Also tested how this works with the source build task (see final pipeline in https://github.com/chmeliik/centos-bootc-tmp/blob/hermetically/.tekton/ostree-build.yaml). There is a bug in the source build task which causes the source container to not include any SRPMs. Will fix. |
chmeliik
requested review from
tkdchen,
scoheb,
brunoapimentel,
mmorhun and
arewm
eskultety
reviewed
Apr 9, 2024
chmeliik
force-pushed
the
rpm-ostree-plus-cachi2-rpms
branch
from
5d7b80c to
ccb00e3
Compare
Also, if I'm not wrong, 2.4 Gi of that are duplicates $ fdupes -r cachi2-output > dupes.txt
$ cat dupes.txt | xargs du -b | awk '{ s += $1 } END { print s }' | numfmt --to=iec-i
4.8GiOf those, roughly 400Mi are noarch duplicates (these are needed - each noarch RPM has to be in every arch-specific subdirectory, otherwise only one arch will have access to them). The remaining 2 Gi are duplicate SRPMs (these are not needed). $ grep 'noarch.rpm$' dupes.txt | xargs du -b | awk '{ s += $1 } END { print s }' | numfmt --to=iec-i
792Mi
$ grep 'src.rpm$' dupes.txt | xargs du -b | awk '{ s += $1 } END { print s }' | numfmt --to=iec-i
4.0GiTLDR, the lockfile generation needs a lot of work. |
|
|
/retest |
|
Just screenshotting the passing CI for future reference 😅
|
chmeliik
force-pushed
the
rpm-ostree-plus-cachi2-rpms
branch
from
ccb00e3 to
375ca7d
Compare
chmeliik
force-pushed
the
rpm-ostree-plus-cachi2-rpms
branch
from
375ca7d to
62e9b36
Compare
|
/retest e2e-tests failed with first time seeing that |
|
/retest |
|
Another EC pipeline timeout |
STONEBLD-2280 STONEBLD-2281 The experimental RPM support that's being added to cachi2 will stay gated behind the --dev-package-managers flag. Give users a way to enable dev package managers. Document the option as discouraged. Signed-off-by: Adam Cmiel <[email protected]>
STONEBLD-2281 When the prefetch-dependencies task prefetches RPMs, make the rpm-ostree task use them. Do this by removing all the original repo files from the source directory and replacing them with the generated cachi2.repo. --- Note that the rpm-ostree task determines whether there are prefetched RPMs by checking the existence of the platform-specific RPMs directory. This means that if RPMs are prefetched for platform A but not for platform B, the build for platform B will still work (and download the RPMs at build time). One may argue that this is undesirable, as it can lead to a situation where one platform prefetches RPMs and another one does not. But this is best addressed by enabling the 'hermetic' option to cut off network access altogether (to be added in a follow-up commit). Signed-off-by: Adam Cmiel <[email protected]>
Signed-off-by: Adam Cmiel <[email protected]>
STONEBLD-2281 With HERMETIC=true, the rpm-ostree build will have its network access disabled. This is done by setting --network=none for the podman container executed on the remote VM (which runs the rpm-ostree command). Note that rpm-ostree has an --offline option, but it doesn't do anything useful for hermetic builds. Just prints a warning when RPMs have to be downloaded. Signed-off-by: Adam Cmiel <[email protected]>
chmeliik
force-pushed
the
rpm-ostree-plus-cachi2-rpms
branch
from
62e9b36 to
eab672a
Compare
|
|
interesting /retest |
While pushing bundles. Thanks Quay. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See individual commits for more details
STONEBLD-2281
If the prefetch-dependencies task has prefetched RPMs, make the
rpm-ostree build use them. Do this by removing all the original repo
files from the source directory and replacing them with the generated
cachi2.repo.
Also expose the
--dev-package-managersflag for the prefetch task (therefore this also relates to STONEBLD-2280)And add a hermetic option to the rpm-ostree task