Merge branch 'develop'

CS-SI · Jun 24, 2024 · 7955e0b · 7955e0b
2 parents ef764e9 + 1e0e577
commit 7955e0b
Show file tree

Hide file tree

Showing 161 changed files with 23,104 additions and 11,146 deletions.
diff --git a/.github/workflows/fetch.yml b/.github/workflows/fetch.yml
@@ -39,7 +39,7 @@ jobs:
         key: ${{ runner.os }}-pip-${{ steps.get-week.outputs.WEEK }}-${{ hashFiles('setup.cfg') }}
     - name: Install eodag
       run: |
-        python -m pip install .
+        python -m pip install ".[all-providers]"
     - name: Fetch and update external product types reference
       run: |
         export JSON_OUTPUT_FILE="eodag/resources/ext_product_types.json"
@@ -59,15 +59,15 @@ jobs:
         echo "commit [${COMMIT_SHA}](${COMMIT_URL})" >> $GITHUB_STEP_SUMMARY
         echo '```diff' >> $GITHUB_STEP_SUMMARY
         git show --name-only --format=tformat: >> $GITHUB_STEP_SUMMARY
-        (diff <(curl ${JSON_REF_FILE} | jq ) <(cat ${JSON_OUTPUT_FILE} | jq) || true) >> $GITHUB_STEP_SUMMARY
+        (diff <(curl ${JSON_REF_FILE} | jq --sort-keys) <(cat ${JSON_OUTPUT_FILE} | jq --sort-keys) || true) >> $GITHUB_STEP_SUMMARY
         echo '```' >> $GITHUB_STEP_SUMMARY
         # truncated PR body (too long causes error)
         full_update_summary=$(cat $GITHUB_STEP_SUMMARY)
         echo 'UPDATE_SUMMARY<<EOF' >> $GITHUB_ENV
         echo "${full_update_summary:0:65536}" >> $GITHUB_ENV
         echo 'EOF' >> $GITHUB_ENV
     - name: Create Pull Request
-      uses: peter-evans/create-pull-request@v4
+      uses: peter-evans/create-pull-request@v6
       with:
         branch: external-product-types-ref-update
         delete-branch: true

diff --git a/CHANGES.rst b/CHANGES.rst
@@ -1,10 +1,114 @@
 Release history
 ---------------
 
+3.0.0b1 (2024-06-24)
+++++++++++++++++++++
+
+Breaking changes
+^^^^^^^^^^^^^^^^
+
+* `search() <https://eodag.readthedocs.io/en/latest/notebooks/api_user_guide/4_search.html#search()>`_ method now
+  returns only a :class:`~eodag.api.search_result.SearchResult` instead of a 2 values tuple (:pull:`1200`). It can
+  optionally store the estimated total number of products in ``SearchResult.number_matched`` if the method is called
+  with ``count=True`` (``False`` by  default).
+* Packaging refactoring and new `optional dependencies
+  <https://eodag.readthedocs.io/en/latest/getting_started_guide/install.html#optional-dependencies>`_ (:pull:`1108`)
+  (:pull:`1219`). EODAG default installs with a minimal set of dependencies.
+  New sets of extra requirements are: ``eodag[all]``, ``eodag[all-providers]``, ``eodag[ecmwf]``, ``eodag[usgs]``,
+  ``eodag[csw]``, ``eodag[server]``. Previous existing sets of extra requirements are also kept:
+  ``eodag[notebook]``, ``eodag[tutorials]``, ``eodag[dev]``, ``eodag[docs]``.
+
+Core features and fixes
+^^^^^^^^^^^^^^^^^^^^^^^
+
+* Providers groups (:pull:`1071`)
+* Configurable download timeout (:pull:`1124`)
+* `Search by id <https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/4_search.html#id-and-provider>`_ now
+  uses :meth:`~eodag.api.core.EODataAccessGateway.search_all` and
+  `crunch <https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/6_crunch.html#Filter-by-property>`_
+  (:pull:`1099`).
+* Free text search available for all fields when `guessing a produc type
+  <https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/6_crunch.html#Filter-by-property>`_ (:pull:`1070`),
+  mission dates filtering support (:pull:`1222`)
+* Configurable requests ``ssl_verify`` (:pull:`1045`)
+* Fixed and refactored `queryables` (:pull:`1050`)(:pull:`1097`)(:pull:`1102`)(:pull:`1157`), authentication fix
+  (:pull:`1194`), support for local constraints files (:pull:`1105`)
+* Fixed `metadata mapping` in templates detection (:pull:`1139`), ``format_query_params()`` fixes (:pull:`1145`) and
+  refactor (:pull:`1142`)
+
+Providers and product types updates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* `dedl <https://hda.data.destination-earth.eu/ui>`_ as new provider (:pull:`750`)
+* `dedt_lumi <https://polytope.lumi.apps.dte.destination-earth.eu/openapi>`_ as new provider (:pull:`1119`)
+  (:pull:`1126`), with authentication using destine credentials (:pull:`1127`)
+* `cop_marine <https://marine.copernicus.eu/>`_ as new provider (:pull:`1131`)(:pull:`1224`)
+* `eumetsat_ds <https://data.eumetsat.int/>`_ as new provider (:pull:`1060`), including `METOP` product types
+  (:pull:`1143`)(:pull:`1189`)
+* `OData` API usage for ``creodias`` & ``cop_dataspace`` (:pull:`1149`), fixes for empty geometries (:pull:`1186`),
+  search datetime intervals (:pull:`1158`), and removed `discover_product_types` (:pull:`1112`)
+* ``cop_ads`` and ``cop_cds`` now use :class:`~eodag.plugins.search.build_search_result.BuildSearchResult` and
+  :class:`~eodag.plugins.download.http.HTTPDownload` instead of move ``CdsApi`` (:pull:`1029`), `EFAS` dates formatting
+  (:pull:`1178`), ``area`` metadata mapping fix (:pull:`1225`)
+* ``wekeo`` now uses `hda-broker 2.0` API (:pull:`1034`), lists queryables (:pull:`1104`), has fixed pagination
+  (:pull:`1098`) and CLMS search by id (:pull:`1100`)
+* Adjusted timeouts (:pull:`1163`)
+* Opened time intervals supported for STAC providers (:pull:`1144`)
+* New product types (:pull:`1164`)(:pull:`1227`), providers and product types configuration update (:pull:`1212`)
+
+Plugins new features and fixes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* Refactored search plugins methods to use ``PreparedSearch`` and ``RawSearchResult`` new classes (:pull:`1191`)
+* Refresh token for :class:`~eodag.plugins.authentication.openid_connect.OIDCAuthorizationCodeFlowAuth` plugin
+  (:pull:`1138`), tests (:pull:`1135`), and fix (:pull:`1232`)
+* :class:`~eodag.plugins.authentication.header.HTTPHeaderAuth` accepts headers definition in credentials (:pull:`1215`)
+* ``flatten_top_dirs`` download plugins option set to true by default (#1220)
+* ``base_uri`` download plugins setting is not systematically mandatory any more (:pull:`1230`)
+* Re-login in :class:`~eodag.plugins.apis.usgs.UsgsApi` plugin on api file error (:pull:`1046`)
+* Allow no auth for :class:`~eodag.plugins.download.http.HTTPDownload` download requests (:pull:`1196`)
+* Refactorization of ``Api`` base plugin that now inherits from ``Search`` and ``Download`` (:pull:`1051`)
+* ``orderLink`` support in `build_search_result.*` plugins (:pull:`1082`), and parsing fix (:pull:`1091`)
+
+Server mode
+^^^^^^^^^^^
+
+* Offline products order handling (:pull:`918`)
+* External enhanced product types metadata (:pull:`1008`)(:pull:`1171`)(:pull:`1176`)(:pull:`1180`)(:pull:`1197`)
+* Collections search using updated :meth:`~eodag.api.core.EODataAccessGateway.guess_product_type` (:pull:`909`)
+* Providers groups (:pull:`1192`), and fixes for listing (:pull:`1187`) and items self links (:pull:`1090`)
+* ``HEAD`` requests enabled (:pull:`1120`)
+* LRU caching (:pull:`1073`)
+* Additional item properties (:pull:`1170`)
+* ``order`` and ``storage`` extensions usage (:pull:`1117`)
+* ``bbox`` in queryables (:pull:`1185`), fixed some types missing (:pull:`1083`)
+* Blacklist configution for assets alternate URLs (:pull:`1213`)
+* ``id`` vs ``title`` in item metadata fix (:pull:`1193`)
+* Error handling fixes (:pull:`1078`)(:pull:`1103`)(:pull:`1182`)
+* Other server-mode fixes  (:pull:`1065`)(:pull:`1087`)(:pull:`1094`)(:pull:`1095`)(:pull:`1096`)(:pull:`1106`)
+  (:pull:`1113`)(:pull:`1115`)(:pull:`1156`)(:pull:`1174`)(:pull:`1210`)(:pull:`1221`)(:pull:`1223`)
+
+Miscellaneous
+^^^^^^^^^^^^^
+
+* **[build]** Updated requirements for ``uvicorn`` (:pull:`1152`), ``shapely`` (:pull:`1155`), ``orjson`` (:pull:`1150`)
+  (:pull:`1079`)
+* **[build]** Remove ``requests-ftp`` (:pull:`1085`)
+* **[style]** type hints related fixes and refactoring (:pull:`1052`)
+* **[docs]** sphinx theme updated and removed jquery (:pull:`1054`), newlines between badges fixes (:pull:`1109`), and other
+  documentation fixes and updates (:pull:`1057`)(:pull:`1059`)(:pull:`1062`)(:pull:`1063`)(:pull:`1081`)(:pull:`1121`)
+  (:pull:`1122`)
+* **[ci]** Fetch product types Github action updates (:pull:`1202`)(:pull:`1205`)
+* Various minor fixes and improvements (:pull:`1072`)(:pull:`1077`)(:pull:`1101`)(:pull:`1111`)(:pull:`1118`)
+  (:pull:`1132`)(:pull:`1141`)(:pull:`1190`)
+* External product types reference updates (:pull:`1086`)(:pull:`1093`)(:pull:`1107`)(:pull:`1110`)(:pull:`1114`)
+  (:pull:`1136`)(:pull:`1137`)(:pull:`1140`)(:pull:`1146`)(:pull:`1151`)(:pull:`1153`)(:pull:`1160`)(:pull:`1165`)
+  (:pull:`1203`)(:pull:`1204`)(:pull:`1206`)(:pull:`1207`)(:pull:`1208`)(:pull:`1229`)
+
 2.12.1 (2024-03-05)
 +++++++++++++++++++
 
-* :class:`~eodag.plugins.apis.cds.CdsApi` queryables fix (:pull:`1048`)
+* `CdsApi` queryables fix (:pull:`1048`)
 
 2.12.0 (2024-02-19)
 +++++++++++++++++++
@@ -22,10 +126,10 @@ Release history
 * New product types added for `cop_ads` and `cop_cds` (:pull:`898`)
 * Adds missing `tileIdentifier` and `quicklook` for `creodias`, `creodias_s3` and `cop_dataspace` (:pull:`957`)
   (:pull:`1014`)
-* HTTP download with :class:`~eodag.plugins.apis.cds.CdsApi` (:pull:`946`)
+* HTTP download with `CdsApi` (:pull:`946`)
 * Download streaming available for :class:`~eodag.plugins.download.aws.AwsDownload` plugin (:pull:`997`)
 * Lists STAC alternate assets in server mode (:pull:`961`)
-* `_dc_qs` used in server-mode to store :class:`~eodag.plugins.apis.cds.CdsApi` search criteria (:pull:`958`)(:pull:`1000`)
+* `_dc_qs` used in server-mode to store `CdsApi` search criteria (:pull:`958`)(:pull:`1000`)
 * New eodag exception :class:`~eodag.utils.exceptions.TimeOutError` (:pull:`982`)
 * Cast loaded environment variables type using config type-hints (:pull:`987`)
 * Type hints fixes (:pull:`880`)(:pull:`983`)
@@ -106,7 +210,7 @@ Release history
   (:pull:`659`)
 * Fetch product types optimization (:pull:`683`)
 * Fixes external product types update for unknown provider (:pull:`682`)
-* Default dates and refactor for `:class:`~eodag.plugins.apis.cds.CdsApi` and `:class:`~eodag.plugins.apis.cds.EcmwfApi` (:pull:`672`)(:pull:`678`)(:pull:`679`)
+* Default dates and refactor for `CdsApi` and :class:`~eodag.plugins.apis.ecmwf.EcmwfApi` (:pull:`672`)(:pull:`678`)(:pull:`679`)
 * `peps` `storageStatus` update (:pull:`677`)
 * Customized and faster `deepcopy` (:pull:`664`)
 * Various minor fixes and improvements (:pull:`665`)(:pull:`666`)(:pull:`667`)(:pull:`668`)(:pull:`669`)(:pull:`670`)
@@ -198,7 +302,7 @@ Release history
   (:pull:`480`)(:pull:`467`)(:pull:`470`)(:pull:`471`)(:pull:`472`)(:pull:`473`)(:pull:`481`)(:pull:`486`)(:pull:`493`)
   (:pull:`491`)(:pull:`500`)
 * New providers `cop_ads <https://ads.atmosphere.copernicus.eu>`_ and `cop_cds <https://cds.climate.copernicus.eu>`_
-  for Copernicus Atmosphere and Climate Data Stores using :class:`~eodag.plugins.apis.cds.CdsApi` plugin, developed in
+  for Copernicus Atmosphere and Climate Data Stores using `CdsApi` plugin, developed in
   the context of DOMINO-X (:pull:`504`)(:pull:`513`)
 * :class:`~eodag.plugins.apis.usgs.UsgsApi` plugin fixed and updated (:pull:`489`)(:pull:`508`)
 * Cache usage for ``jsonpath.parse()`` (:pull:`502`)

diff --git a/NOTICE b/NOTICE
@@ -22,7 +22,6 @@ https://github.com/boto/botocore
 https://github.com/h2non/jsonpath-ng
 https://github.com/CS-SI/eodag-cube
 https://github.com/ecmwf/ecmwf-api-client
-https://github.com/ecmwf/cdsapi
 https://github.com/ijl/orjson
 https://github.com/stac-utils/pystac
 
@@ -46,7 +45,11 @@ https://github.com/tiangolo/fastapi
 https://github.com/urllib3/urllib3
 https://github.com/annotated-types/annotated-types
 https://github.com/pypa/setuptools
-
+https://github.com/pydantic/pydantic
+https://github.com/pydantic/pydantic-core
+https://github.com/pydantic/pydantic-settings
+https://github.com/geopython/pygeofilter
+https://github.com/tkem/cachetools
 
 ================================================================
 The BSD-2-Clause Licence

diff --git a/README.rst b/README.rst
@@ -3,32 +3,36 @@
 
 |
 
-.. image:: https://badge.fury.io/py/eodag.svg
+.. |pypi-badge| image:: https://badge.fury.io/py/eodag.svg
     :target: https://badge.fury.io/py/eodag
 
-.. image:: https://img.shields.io/conda/vn/conda-forge/eodag
+.. |conda-badge| image:: https://img.shields.io/conda/vn/conda-forge/eodag
     :target: https://anaconda.org/conda-forge/eodag
 
-.. image:: https://readthedocs.org/projects/eodag/badge/?version=latest&style=flat
+.. |rtd-badge| image:: https://readthedocs.org/projects/eodag/badge/?version=latest&style=flat
     :target: https://eodag.readthedocs.io/en/latest/
 
-.. image:: https://github.com/CS-SI/eodag/actions/workflows/test.yml/badge.svg
+.. |gha-badge| image:: https://github.com/CS-SI/eodag/actions/workflows/test.yml/badge.svg
     :target: https://github.com/CS-SI/eodag/actions
 
-.. image:: https://img.shields.io/github/issues/CS-SI/eodag.svg
+.. |ghi-badge| image:: https://img.shields.io/github/issues/CS-SI/eodag.svg
     :target: https://github.com/CS-SI/eodag/issues
 
-.. image:: https://mybinder.org/badge_logo.svg
+.. |binder-badge| image:: https://mybinder.org/badge_logo.svg
     :target: https://mybinder.org/v2/git/https%3A%2F%2Fgithub.com%2FCS-SI%2Feodag.git/master?filepath=docs%2Fnotebooks%2Fintro_notebooks.ipynb
 
+|pypi-badge| |conda-badge| |rtd-badge| |gha-badge| |ghi-badge| |binder-badge|
+
 |
 
-.. image:: https://img.shields.io/pypi/l/eodag.svg
+.. |license-badge| image:: https://img.shields.io/pypi/l/eodag.svg
     :target: https://pypi.org/project/eodag/
 
-.. image:: https://img.shields.io/pypi/pyversions/eodag.svg
+.. |versions-badge| image:: https://img.shields.io/pypi/pyversions/eodag.svg
     :target: https://pypi.org/project/eodag/
 
+|license-badge| |versions-badge|
+
 |
 
 ..
@@ -80,6 +84,9 @@ And with ``conda`` from the `conda-forge channel <https://anaconda.org/conda-for
 
    conda install -c conda-forge eodag
 
+**[New in v3.0.0]** Please note that EODAG comes with a minimal set of dependencies. If you want more features, please install using one of
+the `available extras <https://eodag.readthedocs.io/en/latest/getting_started_guide/install.html#optional-dependencies>`_.
+
 Usage
 =====
 
@@ -98,7 +105,7 @@ Example usage for interacting with the api in your Python code:
 
     dag = EODataAccessGateway()
 
-    search_results, total_count = dag.search(
+    search_results = dag.search(
         productType='S2_MSI_L1C',
         geom={'lonmin': 1, 'latmin': 43.5, 'lonmax': 2, 'latmax': 44}, # accepts WKT polygons, shapely.geometry, ...
         start='2021-01-01',
@@ -112,10 +119,13 @@ This will search for Sentinel 2 level-1C products on the default provider and re
 an estimated total number of products matching the search criteria. And then it will download these products. Please
 check the `Python API User Guide <https://eodag.readthedocs.io/en/latest/api_user_guide.html>`_ for more details.
 
+**[New in v3.0.0]** `search() <https://eodag.readthedocs.io/en/latest/notebooks/api_user_guide/4_search.html#search()>`_
+method now returns only a single ``SearchResult`` instead of a 2 values tuple.
+
 STAC REST API
 -------------
 
-An eodag instance can be exposed through a STAC compliant REST api from the command line:
+An eodag instance can be exposed through a STAC compliant REST api from the command line (``eodag[server]`` needed):
 
 .. code-block:: bash
 
@@ -173,7 +183,7 @@ An eodag instance can be exposed through a STAC compliant REST api from the comm
 
 .. code-block:: bash
 
-    docker run -p 5000:5000 --rm csspace/eodag-server:2.12.1
+    docker run -p 5000:5000 --rm csspace/eodag-server:3.0.0b1
 
 You can also browse over your STAC API server using `STAC Browser <https://github.com/radiantearth/stac-browser>`_.
 Simply run:

diff --git a/charts/eodag-server/Chart.lock b/charts/eodag-server/Chart.lock
@@ -1,6 +1,6 @@
 dependencies:
 - name: common
   repository: oci://registry-1.docker.io/bitnamicharts
-  version: 2.4.0
-digest: sha256:b371e6f7f1449fa3abdcb97a04b0bbb2b5d36a4facb8e79041ac36a455b02bb0
-generated: "2023-06-19T12:39:44.271254606+02:00"
+  version: 2.19.3
+digest: sha256:de997835d9ce9a9deefc2d70d8c62b11aa1d1a76ece9e86a83736ab9f930bf4d
+generated: "2024-06-03T18:43:30.71045378+02:00"
diff --git a/charts/eodag-server/Chart.yaml b/charts/eodag-server/Chart.yaml
@@ -1,11 +1,11 @@
 apiVersion: v2
-appVersion: 2.12.1
+appVersion: 3.0.0b1
 dependencies:
 - name: common
   repository:  oci://registry-1.docker.io/bitnamicharts
   tags:
   - bitnami-common
-  version: 2.x
+  version: 2.x.x
 description: EODAG (Earth Observation Data Access Gateway) is a tool for searching,
   aggregating results and downloading remote sensed images offering a unified API
   for data access regardless of the data provider.
@@ -15,4 +15,4 @@ name: eodag-server
 sources:
 - https://github.com/CS-SI/eodag
 type: application
-version: 2.12.1
+version: 3.0.0b1
diff --git a/charts/eodag-server/README.md b/charts/eodag-server/README.md
@@ -46,7 +46,6 @@ The command removes all the Kubernetes components associated with the chart and
 | `global.imagePullSecrets` | Global Docker registry secret names as an array | `[]`  |
 | `global.storageClass`     | Global StorageClass for Persistent Volume(s)    | `""`  |
 
-
 ### Common parameters
 
 | Name                | Description                                                          | Value           |
@@ -60,7 +59,6 @@ The command removes all the Kubernetes components associated with the chart and
 | `clusterDomain`     | Kubernetes cluster domain name                                       | `cluster.local` |
 | `extraDeploy`       | Array of extra objects to deploy with the release                    | `[]`            |
 
-
 ### EODAG Server parameters
 
 | Name                                                | Description                                                                                                               | Value                    |
@@ -179,6 +177,11 @@ The command removes all the Kubernetes components associated with the chart and
 | `serviceAccount.name`                               | The name of the ServiceAccount to use.                                                                                    | `""`                     |
 | `serviceAccount.annotations`                        | Additional custom annotations for the ServiceAccount                                                                      | `{}`                     |
 | `serviceAccount.automountServiceAccountToken`       | Automount service account token for the server service account                                                            | `true`                   |
+| `autoscaling.enabled`                               | Enable autoscaling                                                                                                        | `false`                  |
+| `autoscaling.minReplicas`                           | Minimum number of replicas                                                                                                | `1`                      |
+| `autoscaling.maxReplicas`                           | Maximum number of replicas                                                                                                | `10`                     |
+| `autoscaling.targetCPU`                             | Target CPU utilization percentage                                                                                         | `""`                     |
+| `autoscaling.targetMemory`                          | Target Memory utilization percentage                                                                                      | `""`                     |
 
 
 ```console