[Bug]: Runfiles resolution is broken for oci_python_image example

[uhlajs]

What happened?

Runfiles resolution described in rules_python/python/runfiles does not work with bazel-examples/oci_python_image example.

This is caused by the fact that _repo_mapping file is NOT in runfiles object and so it is invisible to the custom Starlark rule 17941. The bugfix implemented in 4e60992432a49dda255eb005544ae8c03b30e243 will not help us to resolve this problem, since it only fixes zip file generating with --build_python_zip.

Version

Development (host) and target OS/architectures:

NAME="Fedora Linux"
VERSION="38 (Container Image)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f38/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="Container Image"
VARIANT_ID=container

Output of bazel --version:

bazel 6.3.1

Docker version:

Docker version 24.0.6, build ed223bc

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

bazel_dep(name = "aspect_bazel_lib", version = "1.32.0")
bazel_dep(name = "rules_python", version = "0.26.0")
bazel_dep(name = "rules_oci", version = "1.4.0")

Language(s) and/or frameworks involved:

• Python
• Starlark
• OCI

How to reproduce

Check out branch bug/oci_python_image/runfiles from my fork uhlajs/bazel-examples.

Run:

cd oci_python_image
bazel run hello_world
bazel run hello_world:tarball && docker run --rm gcr.io/oci_python_hello_world:latest

Any other information?

In logs bellow the output of these two command SHOULD be identical. However, the runfiles resolution does not work correctly when running in OCI Container. The problem is with missing _repo_mapping file, so the runfiles._Runfiles._repo_mapping is NOT filled with correct data. This results in fact that runfiles resolution ends up with path /opt/hello_world/hello_world.runfiles/my_workspace/hello_world/data/config.json (note the my_workspace directory in path), where is the real location of the file is in /opt/hello_world/hello_world.runfiles/_main/hello_world/data/config.json (note _main directory in path). See also this comment.

$ bazel run hello_world

INFO: Analyzed target //hello_world:hello_world (77 packages loaded, 2621 targets configured).
INFO: Found 1 target...
Target //hello_world:hello_world up-to-date:
  bazel-bin/hello_world/hello_world
INFO: Elapsed time: 5.033s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/hello_world/hello_world
  _____________________
| hello py_image_layer! |
  =====================
                     \
                      \
                        ^__^
                        (oo)\_______
                        (__)\       )\/\
                            ||----w |
                            ||     ||
Runtime Environments
{'RUNFILES_DIR': '/var/home/honza/.cache/bazel/_bazel_honza/b78b5595904abfab5c46be08ce597b7c/execroot/_main/bazel-out/k8-fastbuild/bin/hello_world/hello_world.runfiles',
 'RUNFILES_MANIFEST_FILE': '/var/home/honza/.cache/bazel/_bazel_honza/b78b5595904abfab5c46be08ce597b7c/execroot/_main/bazel-out/k8-fastbuild/bin/hello_world/hello_world.runfiles_manifest',
 'RUNFILES_REPO_MAPPING': '/var/home/honza/.cache/bazel/_bazel_honza/b78b5595904abfab5c46be08ce597b7c/execroot/_main/bazel-out/k8-fastbuild/bin/hello_world/hello_world.repo_mapping'}

Runfiles resolution
/var/home/honza/git/github/bazel-examples/oci_python_image/hello_world/data/config.json
True
File content
['{\n', '    "version": 5\n', '}\n']

Runfiles resolution with addtional mapping
/var/home/honza/git/github/bazel-examples/oci_python_image/hello_world/data/config.json
True
File content
['{\n', '    "version": 5\n', '}\n']

$ bazel run hello_world:tarball && docker run --rm gcr.io/oci_python_hello_world:latest
INFO: Analyzed target //hello_world:tarball (25 packages loaded, 746 targets configured).
INFO: Found 1 target...
Target //hello_world:tarball up-to-date:
  bazel-bin/hello_world/tarball/tarball.tar
INFO: Elapsed time: 7.945s, Critical Path: 7.05s
INFO: 4 processes: 1 internal, 3 linux-sandbox.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/hello_world/tarball.sh
02e6b283ae9d: Loading layer  16.31kB/16.31kB
The image gcr.io/oci_python_hello_world:latest already exists, renaming the old one with ID sha256:59dbc2de9c53128f114c823717931629a52e6796c1f5621da095e665a8db8032 to empty string
Loaded image: gcr.io/oci_python_hello_world:latest
  _____________________
| hello py_image_layer! |
  =====================
                     \
                      \
                        ^__^
                        (oo)\_______
                        (__)\       )\/\
                            ||----w |
                            ||     ||
Runtime Environments
{'RUNFILES_DIR': '/opt/hello_world/hello_world.runfiles',
 'RUNFILES_MANIFEST_FILE': '',
 'RUNFILES_REPO_MAPPING': ''}

Runfiles resolution
/opt/hello_world/hello_world.runfiles/my_workspace/hello_world/data/config.json
False

Runfiles resolution with addtional mapping
Applying repo mapping
/opt/hello_world/hello_world.runfiles/_main/hello_world/data/config.json
True
File content
['{\n', '    "version": 5\n', '}\n']

[alexeagle]

@fmeum suggested a clever way to build a custom mtree program with repo-mapping awareness that can make this work.

[genert]

Any ways to fix?