You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The basic idea here is to have a provider-level setting (off by default) that directs apko_config to prefetch the APKs to warm the go-apk cache. Something like:
provider"apko" {
# Have apko_config prefetch the APKs for the relevant architectures.prefetch_apks=true
}
This would shift some of the latency from apko_build to apko_config, but we think that practically this will improve throughput because the dependency graph for a common image roughly looks like:
Today the apko_config rules in the above diagram will be very quick, and the apko_build rules will be scheduled in parallel. Both of these builds overlap in the packages for the latest variant, but if the cache is empty they will likely both try to fill it at the same time and have a very low cache hit rate.
However, if we shift to prefetching the APKs during the config step, since latest-dev depends on latest, we will see latest prefetch all of the shared packages, and only then latest-dev will fetch the things that it additionally needs (likely all hits from other parallel dev builds in a mega-module execution). Then once each builds packages have been prefetched then their builds can be scheduled in parallel without one artificially waiting on the other.
The basic idea here is to have a provider-level setting (off by default) that directs
apko_config
to prefetch the APKs to warm thego-apk
cache. Something like:This would shift some of the latency from
apko_build
toapko_config
, but we think that practically this will improve throughput because the dependency graph for a common image roughly looks like:Today the
apko_config
rules in the above diagram will be very quick, and theapko_build
rules will be scheduled in parallel. Both of these builds overlap in the packages for thelatest
variant, but if the cache is empty they will likely both try to fill it at the same time and have a very low cache hit rate.However, if we shift to prefetching the APKs during the config step, since
latest-dev
depends onlatest
, we will seelatest
prefetch all of the shared packages, and only thenlatest-dev
will fetch the things that it additionally needs (likely all hits from other paralleldev
builds in a mega-module execution). Then once each builds packages have been prefetched then their builds can be scheduled in parallel without one artificially waiting on the other.cc @jonjohnsonjr
The text was updated successfully, but these errors were encountered: