Container installation support

[rjschwei]

As systems become leaner and functionality moves from system installed packages (RPM, deb or other) to containers it is important that kiwi can pre-install conatiners from specified registries to build system images.

Changes proposed in this pull request:

• Schema support for container installation with podman

[rjschwei]

@schaefi can you give this a once over to see if I am going in the right direction? The idea is to eventually have kiwi execute podman pull in the chroot image environment after package install is finished. This sets up XML support. Also if you'd rather have separate PRs as the feature evolves then we can remove the WIP label and I can send another PR for implementation bits.

[schaefi]

@rjschwei thanks for starting this effort. I took a look to the schema changes you are about to add and those for sure makes sense when the task is to pull a container into the local registry of the image. However, I've been tasked with this several times in the past and had to deal with some hurdles which held me back to do what you started now. Let me share some thoughts:

1. OBS. It's not possible to run outgoing connections in OBS, this includes fetching an arbitrary container from a container registry, even if that registry is the suse registry. Thus your new container elements would imply a magic handling inside of OBS like it is done with packages and that doesn't exist
2. The container registry is connected to a storage driver. That storage driver is configurable and by default set to overlayfs. When using overlayfs you can pre-load containers to the local registry in kiwi, however if you use e.g btrfs then loading of containers requires the btrfs layer to be present at the time of the pulling. In kiwi this means a generic solution to pull containers has to run with the device layer active. This limits the feature to disk images because they expose the real devices (through loops) and should allow the pull for any storage driver. Either this limitation or forcing into overlay as storage driver is a decision that needs to be taken
3. Containers. Many people build images in CI/CD systems that are based on containers. Running kiwi in a container and building an image that itself pulls containers has caused very weird error conditions in the past for which I could not come up with a solution other than "build in a VM" ... so yet another limitation

There is more but I don't want to scare you ;)

With the above in mind I did not add an extra container tag to the kiwi schema so far. The concept we use so far uses a different approach based on the following design:

1. Package the container. We can do this in a very simple way using OBS, any other way to put a container into a package would also work. If you stick with OBS you only need to add support: containment-rpm-docker to your prjconf. Any container build in that project will also be provided as a package
2. Preload the container as a package. Instead of a new tag for handling containers you just add <package name="the-container-package"/> This now also works inside of OBS, outside of OBS and with any other package that contains the container
3. Image description. In the image description you need to add code that does podman load -i the-container-from-the-package and if the raw container should not be kept you also need to put the package into the <packages type="uninstall"/> section. For that podman call I thought a helper method in kiwi/config/functions.sh might be useful but as it's only one line I did not add something like that so far. We could think of an extra attribute that marks the package as a container package and automatically includes it into the registry. Something like <package name="foo" oci_import="true"/>
4. The problem with the storage driver, container policy, etc etc. If you have a container package you can build the image and store the container locally. You can then load the container at a later point in time e.g on first boot which will then eliminate all storage driver issues and more. It's like a pre-provision and set active later approach

I setup all images that needs containers that way. Yes it requires putting the container into a package which might be considered cumbersome to people

If we add a new container tag I strongly recommend that we don't aim for a direct pull into the local registry but load the container archive as file first and then allow via an attribute what should happen with it: stay as archive, try-loading-but-might-fail---document-exceptions. I could also imagine if we add a new mime type to reference the container name e.g obsregistry://... that the obs team will add some magic such that it can be fetched from a worker.

Thus from my perspective a step that we can go is something like this

<packages type="image">
    <container name="URI" save_as="/usr/share/my/containers/" preload="true|false"/>
</packages>

The URI references from where to fetch it and supports the format that you can use with podman pull plus additional mime types that we might need for integration with OBS ... or others

Sorry long story, but I had some experience and trouble with all this already 😃

[schaefi]

I like the idea to reference a container and operate on it, but I think for consistency and simplicity the way to describe it should be changed. Also see my initial comment. Thoughts ?

[schaefi]

We only change the schema version on incompatible schema changes which requires an XSLT transformation. You are only adding stuff which is not an incompatible change. For tracking features according to the version of kiwi we use the main kiwi version not the schema version. Please keep the version as it is

This also obsoletes your XSLT transformation which does basically nothing than changing the version. So please drop the XSLT

Also please drop the adaptions to the various XML snippets that only changes the schema version.
It will make the pull request a lot smaller :)

Thanks much

[schaefi]

As I wrote in my long first comment I think we can make our live a bit easier if we just add a new container element inside of the <packages> section. A packages sub-element in kiwi speak can be a package, an archive and now also a container. The registry information should be imho placed in an URI reference which is the way we support source references in kiwi. I don't think we need to invent a different way to do this unless I forgot something ?

[rjschwei]

@schaefi thanks for sharing your thoughts. On the OBS side, yes while we want this feature in OBS, OBS is not the only place where to build images. Therefore, if kiwi would have a feature that doesn't work in OBS then that would just be it. I am not certain we should let potential limitations of 1 tool dictate features in kiwi. That said, the problem can certainly be solved in a similar way to repositories, as you already stated obsregistry://.

At first I also thought about adding the location of the container to the container xml-element but then I decided that this could lead to a lot of repetition, if for every container I want to get installed I need to add some-registry then this gets really old. Also a reason I started with podman as we can configure the registries to search in the config file.

[registries.search]
registries = ["registry.opensuse.org", "registry.suse.com", "docker.io"]

and that then works with

# podman pull opensuse/distrobox

While the default install would render a choice problem. Meaning kiwi would have to overwrite the default distribution setup. Which in and off itself is not a problem since the user has specified the registries to search with the setting of the registry element. That of course becomes a problem when the value for registry turns into obsregistry, meaning the config file in the resulting image would be messed up and additional podman pull operations would fail.

This can be handled by the user via config.sh with the emission of the /etc/containers/registries.conf file. Meaning a user that wants to build in OBS would need to have <registry url="obsregistry://"/> and then have

cat >> "/etc/containers/registries.conf" EOF
[registries.search]
registries = ["registry.opensuse.org", "registry.suse.com", "docker.io"]
EOF

in config.sh if containers from those registries are getting installed and OBS has some magic to make them available. Or if the image builder simply wants those registries to be available for search in the image being built. While this is not super convenient it is also not a total horror show.

Still would leave kiwi with dealing with the default config files and for that my initial thought was that kiwi would move them out of the way for the image builder to restore as the builder sees fit, i.e. for SUSE that would look like

mv /etc/containers/registries.conf /etc/containers/registries.conf.default
mv /etc/containers/registries.conf.d/000-shortnames.conf /etc/containers/registries.conf.d/000-shortnames.conf.default

of course this would have to be implemented in a generic name, but that's pretty straight forward as the locations of the config files are known.

The packaging of containers in RPMs brings it's own problems and there have been a number of discussions about that.

Well at least we have this documented now for referencing. I am going to close this PR and we'll see where the conversation in other places around containers wrapped in RPMs leads. I will prepare a branch with just the XML changes containerinstall we can use it to continue the conversation about the pros and cons of a element and then the <container> element simply containing the relative location of the container within the already specified registries.

[schaefi]

Therefore, if kiwi would have a feature that doesn't work in OBS then that would just be it

I agree and have no problem with that. The feature should just at least in theory allow to be expanded for the OBS use case. I think the idea would have covered that part

While the default install would render a choice problem. Meaning kiwi would have to overwrite the default distribution
setup.

yes and as you explained this can be done in kiwi code for the time the image is being built. I don't think it would be problematic to implement that. We had to do it for all package managers we support too

The packaging of containers in RPMs brings it's own problems and there have been a number of discussions about that.

right and it has one advantage in a way that it does not require extra implementations in kiwi. The handling of the installed data via package or by a pull from a registry would be the same code

I wonder why you closed your PR. I think adding a feature that allows to fetch a container from a registry (without looking at OBS for the moment) would be a nice and great addition. We only need to agree on the format to describe it in XML speak and you could go ahead.

I hope I did not demotivate with you with my feedback, it was not my intention

[rjschwei]

@schaefi rather than changing this PR I created a new branch 4ff11a5 in light of what I learned today this is getting a bit more pressing as well. So no you did not break my motivation. Just thought it would be nicer to start with the smaller scope schema changes rather than undoing some of the schema work I did here.