Support 'explorer' and 'dev' images (#30)

To aid with development, I am introducing different image types,
controlled by the following two build arguments:

- `build_type`: This build argument has three different possible values:
  - `standard`: This is the standard build type. it is what customer
    uses.
  - `explorer`: The 'explorer' build type is almost identical to the
    'standard' build type but it doesn't include the entrypoint. This is
    useful for debugging purposes to run the image and look around its
    content without starting airflow, which might require further setup.
  - `explorer-root`: This is similar to the 'explorer' build type, but
    additionally uses the root user, giving the user of this Docker
    image elevated permissions.  The user can, thus, install packages,
    remove packages, or anything else.
- `dev`: When this build argument is set to True, it will result in
  additional packages being installed to aid with development.

For each combination of these two build arguments, a different Docker
image is generated. Thus, we are currently generating these images:

- `amazon-mwaa/airflow:2.8.0`
- `amazon-mwaa/airflow:2.8.0-dev`
- `amazon-mwaa/airflow:2.8.0-explorer`
- `amazon-mwaa/airflow:2.8.0-explorer-dev`
- `amazon-mwaa/airflow:2.8.0-explorer-privileged`
- `amazon-mwaa/airflow:2.8.0-explorer-privileged-dev`

images/airflow/2.8.0/.gitignore

@@ -0,0 +1 @@
+run.sh

images/airflow/2.8.0/Dockerfile.j2

@@ -95,8 +95,32 @@ WORKDIR ${AIRFLOW_USER_HOME}
 # Copy python files.
 COPY ./python /python
 
+{% if build_type == 'standard' %}
+{# This is the standard build type. it is what customer uses.#}
 USER airflow
 
 ENTRYPOINT ["python3", "-m", "mwaa.entrypoint"]
 
-CMD shell
+CMD shell
+{% elif build_type == 'explorer' %}
+{#
+The 'explorer' build type is almost identical to the 'standard' build type but
+it doesn't include the entrypoint. This is useful for debugging purposes to run
+the image and look around its content without starting airflow, which might
+require further setup.
+#}
+USER airflow
+
+ENTRYPOINT ["/bin/bash"]
+{% elif build_type == 'explorer-privileged' %}
+{#
+This is similar to the 'explorer' build type, but additionally uses the root
+user, giving the user of this Docker image elevated permissions.  The user can,
+thus, install packages, remove packages, or anything else.
+#}
+USER root
+
+ENTRYPOINT ["/bin/bash"]
+{% else %}
+{{ raise("Invalid build type.") }}
+{% endif %}

images/airflow/2.8.0/Dockerfile → images/airflow/2.8.0/Dockerfiles/Dockerfile

@@ -3,7 +3,7 @@
 # the Jinja2-templated Dockerfile.j2 file, so you need to change that file
 # instead.
 #
-# This file was generated on 2024-01-25 22:38:38.940576
+# This file was generated on 2024-01-30 16:34:46.229453
 #
 
 FROM public.ecr.aws/amazonlinux/amazonlinux:2023
@@ -92,8 +92,6 @@ RUN /bootstrap/03-root-secondpass/001-create-mwaa-dir.sh
 
 RUN /bootstrap/03-root-secondpass/100-chown-airflow-folder.sh
 
-RUN /bootstrap/03-root-secondpass/200-install-debugging-tools.sh
-
 
 #<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
 # END marker for root user, second pass bootstrapping steps.
@@ -117,8 +115,10 @@ WORKDIR ${AIRFLOW_USER_HOME}
 # Copy python files.
 COPY ./python /python
 
+
+
 USER airflow
 
 ENTRYPOINT ["python3", "-m", "mwaa.entrypoint"]
 
-CMD shell
+CMD shell

images/airflow/2.8.0/Dockerfiles/Dockerfile-dev

@@ -0,0 +1,126 @@
+#
+# WARNING: Don't change this file manually. This file is auto-generated from
+# the Jinja2-templated Dockerfile.j2 file, so you need to change that file
+# instead.
+#
+# This file was generated on 2024-01-30 16:34:46.218794
+#
+
+FROM public.ecr.aws/amazonlinux/amazonlinux:2023
+
+# Environment variables
+ENV AIRFLOW_AMAZON_PROVIDERS_VERSION=8.13.0
+ENV AIRFLOW_USER_HOME=/usr/local/airflow
+ENV AIRFLOW_HOME=${AIRFLOW_USER_HOME}
+ENV AIRFLOW_VERSION=2.8.0
+ENV MWAA_HOME=/usr/local/mwaa
+ENV PYTHON_VERSION=3.11.7
+
+# We don't want those variables to stay in the final image, so we use ARG instead of ENV.
+ARG PATH_DEFAULT=${PATH}
+ARG PATH_AIRFLOW_USER=${AIRFLOW_USER_HOME}/.local/bin:${PATH_DEFAULT}
+ARG PYTHON_MD5_CHECKSUM=d96c7e134c35a8c46236f8a0e566b69c
+ARG MARIADB_DOWNLOAD_BASE_URL=https://mirror.mariadb.org/yum/11.1/fedora38-amd64/rpms
+ARG MARIADB_RPM_COMMON=MariaDB-common-11.1.2-1.fc38.x86_64.rpm
+ARG MARIADB_RPM_COMMON_CHECKSUM=e87371d558efa97724f3728fb214cf19
+ARG MARIADB_RPM_DEVEL=MariaDB-devel-11.1.2-1.fc38.x86_64.rpm
+ARG MARIADB_RPM_DEVEL_CHECKSUM=cfce6e9b53f4e4fb1cb14f1ed720c92c
+ARG MARIADB_RPM_SHARED=MariaDB-shared-11.1.2-1.fc38.x86_64.rpm
+ARG MARIADB_RPM_SHARED_CHECKSUM=ed82ad5bc5b35cb2719a9471a71c6cdb
+
+# Copy bootstrapping files.
+COPY ./bootstrap /bootstrap
+RUN chmod -R +x /bootstrap
+
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+# BEGINNING marker for root user, first pass bootstrapping steps.
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+
+
+RUN /bootstrap/01-root-firstpass/001-init.sh
+
+RUN /bootstrap/01-root-firstpass/002-install-python.sh
+
+RUN /bootstrap/01-root-firstpass/003-install-mariadb.sh
+
+RUN /bootstrap/01-root-firstpass/004-create-airflow-user.sh
+
+RUN /bootstrap/01-root-firstpass/005-install-aws-cli.sh
+
+RUN /bootstrap/01-root-firstpass/100-install-needed-dnf-packages.sh
+
+
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+# END marker for root user, first pass bootstrapping steps.
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+
+
+
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+# > BEGINNING marker for airflow user bootstrapping steps.
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+
+# Switch to 'airflow' user and update the PATH environment variable.
+USER airflow
+ENV PATH=${PATH_AIRFLOW_USER} 
+
+
+RUN /bootstrap/02-airflow/001-install-airflow.sh
+
+
+# Revert the PATH and user.
+ENV PATH=${PATH_DEFAULT}
+USER root
+
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+# > END marker for airflow user bootstrapping steps.
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+
+
+
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+# BEGINNING marker for root user, second pass bootstrapping steps.
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+
+# Those steps are also executed as the root user. However, they rely on the
+# successfull execution of the airflow user bootstrapping steps. For example,
+# giving ownership of the Airflow home user to the 'airflow' user requires the
+# the folder to be fully setup first.
+
+
+RUN /bootstrap/03-root-secondpass/001-create-mwaa-dir.sh
+
+RUN /bootstrap/03-root-secondpass/100-chown-airflow-folder.sh
+
+RUN /bootstrap/03-root-secondpass/200-devonly-install-dev-tools.sh
+
+
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+# END marker for root user, second pass bootstrapping steps.
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+
+# remove bootstrapping files.
+RUN rm -rf /bootstrap
+
+# Create a volume for syncing files with the sidecar. The actual folder
+# is created by the `001-create-mwaa-dir.sh` script.
+VOLUME ["${MWAA_HOME}"]
+
+# TODO We should only expose this port if the command is 'webserver'.
+EXPOSE 8080
+
+ENV PATH=${PATH_AIRFLOW_USER}
+ENV PYTHONPATH="/python"
+
+WORKDIR ${AIRFLOW_USER_HOME}
+
+# Copy python files.
+COPY ./python /python
+
+
+
+USER airflow
+
+ENTRYPOINT ["python3", "-m", "mwaa.entrypoint"]
+
+CMD shell

images/airflow/2.8.0/Dockerfiles/Dockerfile-explorer

@@ -0,0 +1,122 @@
+#
+# WARNING: Don't change this file manually. This file is auto-generated from
+# the Jinja2-templated Dockerfile.j2 file, so you need to change that file
+# instead.
+#
+# This file was generated on 2024-01-30 16:34:46.232966
+#
+
+FROM public.ecr.aws/amazonlinux/amazonlinux:2023
+
+# Environment variables
+ENV AIRFLOW_AMAZON_PROVIDERS_VERSION=8.13.0
+ENV AIRFLOW_USER_HOME=/usr/local/airflow
+ENV AIRFLOW_HOME=${AIRFLOW_USER_HOME}
+ENV AIRFLOW_VERSION=2.8.0
+ENV MWAA_HOME=/usr/local/mwaa
+ENV PYTHON_VERSION=3.11.7
+
+# We don't want those variables to stay in the final image, so we use ARG instead of ENV.
+ARG PATH_DEFAULT=${PATH}
+ARG PATH_AIRFLOW_USER=${AIRFLOW_USER_HOME}/.local/bin:${PATH_DEFAULT}
+ARG PYTHON_MD5_CHECKSUM=d96c7e134c35a8c46236f8a0e566b69c
+ARG MARIADB_DOWNLOAD_BASE_URL=https://mirror.mariadb.org/yum/11.1/fedora38-amd64/rpms
+ARG MARIADB_RPM_COMMON=MariaDB-common-11.1.2-1.fc38.x86_64.rpm
+ARG MARIADB_RPM_COMMON_CHECKSUM=e87371d558efa97724f3728fb214cf19
+ARG MARIADB_RPM_DEVEL=MariaDB-devel-11.1.2-1.fc38.x86_64.rpm
+ARG MARIADB_RPM_DEVEL_CHECKSUM=cfce6e9b53f4e4fb1cb14f1ed720c92c
+ARG MARIADB_RPM_SHARED=MariaDB-shared-11.1.2-1.fc38.x86_64.rpm
+ARG MARIADB_RPM_SHARED_CHECKSUM=ed82ad5bc5b35cb2719a9471a71c6cdb
+
+# Copy bootstrapping files.
+COPY ./bootstrap /bootstrap
+RUN chmod -R +x /bootstrap
+
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+# BEGINNING marker for root user, first pass bootstrapping steps.
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+
+
+RUN /bootstrap/01-root-firstpass/001-init.sh
+
+RUN /bootstrap/01-root-firstpass/002-install-python.sh
+
+RUN /bootstrap/01-root-firstpass/003-install-mariadb.sh
+
+RUN /bootstrap/01-root-firstpass/004-create-airflow-user.sh
+
+RUN /bootstrap/01-root-firstpass/005-install-aws-cli.sh
+
+RUN /bootstrap/01-root-firstpass/100-install-needed-dnf-packages.sh
+
+
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+# END marker for root user, first pass bootstrapping steps.
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+
+
+
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+# > BEGINNING marker for airflow user bootstrapping steps.
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+
+# Switch to 'airflow' user and update the PATH environment variable.
+USER airflow
+ENV PATH=${PATH_AIRFLOW_USER} 
+
+
+RUN /bootstrap/02-airflow/001-install-airflow.sh
+
+
+# Revert the PATH and user.
+ENV PATH=${PATH_DEFAULT}
+USER root
+
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+# > END marker for airflow user bootstrapping steps.
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+
+
+
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+# BEGINNING marker for root user, second pass bootstrapping steps.
+#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+
+# Those steps are also executed as the root user. However, they rely on the
+# successfull execution of the airflow user bootstrapping steps. For example,
+# giving ownership of the Airflow home user to the 'airflow' user requires the
+# the folder to be fully setup first.
+
+
+RUN /bootstrap/03-root-secondpass/001-create-mwaa-dir.sh
+
+RUN /bootstrap/03-root-secondpass/100-chown-airflow-folder.sh
+
+
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+# END marker for root user, second pass bootstrapping steps.
+#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+
+# remove bootstrapping files.
+RUN rm -rf /bootstrap
+
+# Create a volume for syncing files with the sidecar. The actual folder
+# is created by the `001-create-mwaa-dir.sh` script.
+VOLUME ["${MWAA_HOME}"]
+
+# TODO We should only expose this port if the command is 'webserver'.
+EXPOSE 8080
+
+ENV PATH=${PATH_AIRFLOW_USER}
+ENV PYTHONPATH="/python"
+
+WORKDIR ${AIRFLOW_USER_HOME}
+
+# Copy python files.
+COPY ./python /python
+
+
+
+USER airflow
+
+ENTRYPOINT ["/bin/bash"]