Use PyPI JSON and "Simple" API instead of deprecated XML-RPC API #201

nuclearsandwich · 2024-07-09T00:04:25Z

The XML-RPC API is deprecated. Tests using it were failing due to what
seems like a malformed url. I have no idea if this API has already been
dropped or if there's some usage issue inside stdeb but I was not
motivated to investigate since the entire API is deprecated.

Instead I replaced the functionality with a new Requests HTTP session
client that uses the JSON representation of the Simple API and
the JSON API where best appropriate. It could all be done with the
JSON API but parts of it are also deprecated...

There is one regression in behavior, which is that packages with a
release on PyPI but no download url on PyPI will not be checked for an
external download url as this also deprecated, so I find this acceptable.

The XML-RPC API is deprecated. Tests using it were failing due to what seems like a malformed url. I have no idea if this API has already been dropped or if there's some usage issue inside stdeb but I was not motivated to investigate since the entire API is deprecated. Instead I replaced the functionality with a new Requests HTTP session client that uses the [JSON representation][1] of the [Simple API][2] and the [JSON API][3] where best appropriate. It could all be done with the JSON API but parts of it are also deprecated... There is one regression in behavior, which is that packages with a release on PyPI but no download url on PyPI will not be checked for an external download url as this also deprecated. [1]: https://peps.python.org/pep-0691/ [2]: https://warehouse.pypa.io/api-reference/legacy.html [3]: https://warehouse.pypa.io/api-reference/json.html

nuclearsandwich · 2024-07-09T00:06:50Z

stdeb/downloader.py

-    if download_url is None:
-        # PyPI doesn't have package. Is download URL provided?
-        result = _call(pypi.release_data, package_name, version)
-        if result['download_url'] != 'UNKNOWN':
-            download_url = result['download_url']
-            # no download URL provided, see if PyPI itself has download
-            urls = _call(pypi.release_urls, result['name'], result['version'])


This is the behavior for which I could find no sensible replacement in the current PyPI APIs.

Yeah, I don't think the concept carries over to the new API.

nuclearsandwich · 2024-07-09T00:07:24Z

stdeb/downloader.py

        if verbose >= 2:
            myprint(
-                'found default release: %s' % (', '.join(default_releases),))
+                'found default release: %s' % (', '.join(default_release),))


I think this is actually a long-lived typo and the intent was always to log the single default release found.

nuclearsandwich · 2024-07-09T00:12:54Z

It's not complete because I'm juggling a number of other changes to fix the test scripts but with this and a few other changes (mostly related to python vs python3 and environmental/dependency updates I was able to get as far as running py2dsc on psycopg 2.7 before it fails to find a pg_config executable which is much deeper into the test script than I was previously able to get.

cottsay · 2024-07-09T14:00:24Z

stdeb/pypi_simple.py

+USER_AGENT = "pypi-install/%s ( https://github.com/astraw/stdeb )" % \
+    stdeb.__version__


Since the only current use case passes in an explicit USER_AGENT anyway, it might be good to remove the duplicated string here and make the default behavior not specify a user agent at all.

I forgot to make this change but I agree with it.

In 0787a84 I ended up importing the USER_AGENT constant from the downloader. I think adding a download method to the pypi client would also be another way to push this down. But this cuts the duplication although it keeps the default user agent.

cottsay · 2024-07-09T14:02:23Z

stdeb/pypi_simple.py

+
+    def release_versions(self, package_name, current_only=False):
+        package_name = normalize_package_name(package_name)
+        if current_only:


The two code paths determined by current_only share only the package name normalization which is already in a separate function. I'd recommend introducing a new function for the True path here, especially since the return type seems to change based on that, which is pretty confusing.

Great point! I was culting forward a pretty straight port of the interface from the old XML-RPC transport, but even that was a carry-over from an interface that was not actually being used for what it is for. I've split the functions in 2d0a089 and the result is much better. Thanks for spotting it.

cottsay · 2024-07-09T14:03:18Z

stdeb/pypi_simple.py

+        if current_only:
+            response = self._http_session.get("%s/pypi/%s/json" % (self.pypi_url, package_name))
+            data = response.json()
+            return data['info']['version']


I believe that all of the other string quotes in this file are double quotes. It would be nice to be consistent.

That said, the other file you're changing here seems to use single quotes. Your call.

Suggested change

return data['info']['version']

return data["info"]["version"]

Thanks. I think the ruff style which I'm exploring for this project uses double quotes ubiquitously and this project currently uses a mix of quote characters.

Fixed in 2d0a089

cottsay · 2024-07-09T14:06:14Z

stdeb/pypi_simple.py

+            if url["packagetype"] == "sdist" and \
+                    url["python_version"] == "source" and \
+                    url["url"].endswith((".tar.gz", ".zip")):
+                download_url = url["url"]
+                md5_digest = url["digests"]["md5"]


The original algorithm has a break here, meaning that the first encountered match is used. This continues to loop, meaning that the last match is used. Is that an intentional deviation?

Good catch! No it isn't. I don't know that there would be a situation with multiple sdist payloads for a given version so I think the break should just be a minor optimization, but it is worth preserving the behavior. In the event that there are multiple sdist urls I'm not sure that the sorting from the server is reliable so we just have to trust that the first one we get is good enough.

Restored the break in 86d8edc

cottsay · 2024-07-09T14:13:31Z

stdeb/downloader.py

-    if download_url is None:
-        # PyPI doesn't have package. Is download URL provided?
-        result = _call(pypi.release_data, package_name, version)
-        if result['download_url'] != 'UNKNOWN':
-            download_url = result['download_url']
-            # no download URL provided, see if PyPI itself has download
-            urls = _call(pypi.release_urls, result['name'], result['version'])


Yeah, I don't think the concept carries over to the new API.

I was tracking quite closely the XML-RPC API interface that was used for this information previously which repurposed a "show_hidden" parameter to return either the current release or all releases in the response. Moving these to separate client methods is a much clearer internal and external organization.

nuclearsandwich requested a review from astraw July 9, 2024 00:04

nuclearsandwich force-pushed the json-transport branch from 083e630 to be6c994 Compare July 9, 2024 00:05

nuclearsandwich commented Jul 9, 2024

View reviewed changes

nuclearsandwich marked this pull request as ready for review July 9, 2024 00:13

nuclearsandwich mentioned this pull request Jul 9, 2024

Migrate to use PyPI's JSON API #195

Closed

cottsay reviewed Jul 9, 2024

View reviewed changes

nuclearsandwich added 2 commits July 9, 2024 08:37

Break after finding an sdist urls.

86d8edc

nuclearsandwich requested a review from cottsay July 9, 2024 16:17

nuclearsandwich added this to the 0.10.1 milestone Jul 9, 2024

cottsay approved these changes Jul 9, 2024

View reviewed changes

Use USER_AGENT from stdeb.downloader instead of duplicating it.

0787a84

nuclearsandwich merged commit c68317e into master Jul 9, 2024

nuclearsandwich deleted the json-transport branch July 9, 2024 18:30

This was referenced Jul 9, 2024

Fix cyclic import dependency by moving USER_AGENT to pypi_simple client. #202

Merged

PyPI XML-RPC API that pypi-download uses has been removed #194

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use PyPI JSON and "Simple" API instead of deprecated XML-RPC API #201

Use PyPI JSON and "Simple" API instead of deprecated XML-RPC API #201

nuclearsandwich commented Jul 9, 2024

nuclearsandwich Jul 9, 2024

cottsay Jul 9, 2024

nuclearsandwich Jul 9, 2024

nuclearsandwich commented Jul 9, 2024

cottsay Jul 9, 2024

nuclearsandwich Jul 9, 2024

nuclearsandwich Jul 9, 2024

cottsay Jul 9, 2024

nuclearsandwich Jul 9, 2024

cottsay Jul 9, 2024

nuclearsandwich Jul 9, 2024

cottsay Jul 9, 2024

nuclearsandwich Jul 9, 2024

nuclearsandwich Jul 9, 2024

cottsay Jul 9, 2024

		USER_AGENT = "pypi-install/%s ( https://github.com/astraw/stdeb )" % \
		stdeb.__version__

	return data['info']['version']
	return data["info"]["version"]

Use PyPI JSON and "Simple" API instead of deprecated XML-RPC API #201

Use PyPI JSON and "Simple" API instead of deprecated XML-RPC API #201

Conversation

nuclearsandwich commented Jul 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuclearsandwich commented Jul 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment