diff --git a/source/guides/handling-missing-extras-at-runtime.rst b/source/guides/handling-missing-extras-at-runtime.rst new file mode 100644 index 000000000..2d2b5a2af --- /dev/null +++ b/source/guides/handling-missing-extras-at-runtime.rst @@ -0,0 +1,171 @@ +.. _handling-missing-extras-at-runtime: + +============================================================ +Handling missing optional dependencies ("extras") at runtime +============================================================ + +If your package has :ref:`optional dependencies ("extras") +` which the package consumer hasn't installed, the +default outcome is an ordinary :exc:`ModuleNotFoundError` exception being raised +at the first attempted import of a missing module. + +This can make for a bad user experience, because there is no guidance about why +the module is missing - users might think they've found a bug. If you're not +careful, it can even make your package unusable without the extras installed, +e.g. if your package is a library that imports the affected modules from the +top-level module or if it's an application that imports them unconditionally. + +As of the time of writing, there is no *great* way to handle this issue in +the Python packaging ecosystem, but there are a few options that might be +better than nothing: + +Detecting missing extras +======================== + +We first consider how to *detect* if an extra is missing, leaving what to do +about it for the next section. + +Trying to import and handling failure +------------------------------------- + +The perhaps simplest option, which is also in line with the :term:`EAFP` +principle, is to just import your optional dependency modules as normal and +handle the relevant exceptions if the import fails: + +.. code-block:: python + + try: + import your_optional_dependency + except ModuleNotFoundError: + ... # handle missing dependency + +However, this can lead to difficult-to-debug errors when +``your_optional_dependency`` *is* installed, but at the wrong version (e.g. +because another installed package depends on it with a wider version +requirement than specified by your extra). + + +Using ``importlib.metadata`` and ``packaging`` +---------------------------------------------- + +As a safer alternative that does check whether the optional dependencies are +installed at the correct versions, :py:mod:`importlib.metadata` and +:ref:`packaging` can be used to iterate through the extra's requirements +recursively and check whether all are installed in the current environment +(based on `code `_ from the `hbutils`_ library): + +.. code-block:: python + + # Adapted from (see there for copyright & license): + # https://github.com/HansBug/hbutils/blob/927b0757449a781ce8e30132f26b06089a24cd71/LICENSE + # SPDX-License-Identifier: Apache-2.0 + + from collections.abc import Iterable + from importlib.metadata import PackageNotFoundError, distribution, metadata + + from packaging.metadata import Metadata, RawMetadata + from packaging.requirements import Requirement + + + def check_reqs(req_strs: Iterable[str]) -> bool: + return all( + _check_req_recursive(req) + for req_str in req_strs + if not (req := Requirement(req_str)).marker or req.marker.evaluate() + ) + + + def _check_req_recursive(req: Requirement) -> bool: + try: + version = distribution(req.name).version + except PackageNotFoundError: + return False # req not installed + + if not req.specifier.contains(version): + return False # req version does not match + + req_metadata = Metadata.from_raw(metadata(req.name).json, validate=False) + for child_req in req_metadata.requires_dist or []: + # A dependency is only required to be present if ... + if ( + not child_req.marker # ... it doesn't have a marker + or child_req.marker.evaluate() # ... its marker matches our env + or any( # ... its marker matches our env given one of our extras + child_req.marker.evaluate({"extra": extra}) for extra in req.extras + ) + ): + if not _check_req_recursive(child_req): + return False + + return True + + + # Perform check, e.g.: + extra_installed = check_reqs(["your-package[your-extra]"]) + +The possibility of offering a helper function similar to ``check_reqs`` in +``importlib.metadata`` or ``packaging`` themselves is still being discussed +(`packaging-problems #317 `_). + +In contrast to the method above, this check is typically done in :term:`LBYL` +style prior to importing the modules in question. +In principle, it could also be done after the imports succeeded just to check +the version, in which case the imports themselves would have to be wrapped in a +``try``-``except`` block to handle the possibility of not being installed at +all. + + +Using ``pkg_resources`` (deprecated) +------------------------------------ + +.. attention:: + + ``pkg_resources`` is **deprecated** and the PyPA **strongly discourages** + its use. + This method is included in this guide for completeness's sake and only until + functionality with a similar level of convenience exists in + ``importlib.metadata`` or ``packaging``. + +The now-deprecated `pkg_resources `_ package (part of the +``setuptools`` distribution) provides a ``require`` function, which was the +inspiration for ``check_reqs`` from the previous section. Its usage is quite +similar to ``check_reqs`` but not identical: + +.. code-block:: python + + from pkg_resources import require, DistributionNotFound, VersionConflict + + try: + require(["your-package-name[your-extra]"]) + except DistributionNotFound: + ... # handle package(s) not being installed at all + except VersionConflict: + ... # handle version mismatches + + +Handling missing extras +======================= + +Where and how to embed the detection of missing extras in a package and what +actions to take upon learning the outcome depends on the specifics of both the +package and feature requiring the extra. +Some common options are: + +- Raise a custom exception that includes the name of the missing extra. +- In applications, show an error message when an attempt is made to use the + feature that requires the extra. +- In libraries, provide a function that lets library consumers query which + features are available. + +... and probably more. + + +------------------ + +.. _hbutils-snippet: https://github.com/HansBug/hbutils/blob/927b0757449a781ce8e30132f26b06089a24cd71/hbutils/system/python/package.py#L171-L242 + +.. _hbutils: https://pypi.org/project/hbutils/ + +.. _pkg_resources: https://setuptools.pypa.io/en/latest/pkg_resources.html + +.. _packaging-problems-317: https://github.com/pypa/packaging-problems/issues/317 diff --git a/source/guides/section-build-and-publish.rst b/source/guides/section-build-and-publish.rst index eb10c389f..29af0483e 100644 --- a/source/guides/section-build-and-publish.rst +++ b/source/guides/section-build-and-publish.rst @@ -16,3 +16,4 @@ Building and Publishing making-a-pypi-friendly-readme publishing-package-distribution-releases-using-github-actions-ci-cd-workflows modernize-setup-py-project + handling-missing-extras-at-runtime