Universal search

[AMDmi3]

There are requirements for search which do not play nicely with our projects filtering:

• search by exact name (Bump tokio from 1.39.2 to 1.39.3 #15)
• search by name prefix
• search by original package name (Add tooltips to item counts on metapackage information page #76)
• search by historic project name

The idea is to implement universal search page which returns results from all these categories with a single query (may include maintainers and repos as well), which would allow having a single search box on a title page and in the top navbar.

[simon04]

Maybe also allow searching for www=*musicpd.org* in order to find https://repology.org/api/v1/project/mpd

Maybe also allow searching for name=nodejs-webpack on repo=aur in order to find https://repology.org/api/v1/project/node:webpack

[ppmathis]

I am very interested in having a possibility to search for a specific original package name through the API, ideally even restricted to a specific repository. Many distribution repositories do not offer any machine-readable HTTP endpoint for querying information about the current state of a package, so having a way to get that information by e.g. calling /api/v1/package/<original package name> or even /api/v1/package/<repo name>/<original package name> would be wonderful.

[AMDmi3]

I am very interested in having a possibility to search for a specific original package name through the API, ideally even restricted to a specific repository

@ppmathis You probably want repology/repology-webapp#66.

[ppmathis]

@AMDmi3 Thank you very much for your reply and please excuse the delay on my side, but due to other circumstances coming up I was not able to continue working on the project which would require this specific API functionality.

I've managed to find https://repology.org/tools/project-by which seems to translate a source/binary package name into the correct Repology project. By choosing API v1 project information as the desired target page (which weirdly enough mentions .svg in the select field, but returns a JSON as expected), I get a template URL which allows me to do what I've described in my previous comment.

My specific goal is however still not easy to solve using Repology due to these two circumstances:

• Some repositories, including Debian, are only parsing source packages while others also feature binary packages - is there any specific reasoning behind this?
• Some repositories are referred to with a name which changes over time, e.g. Debian Stable will switch from buster to bullseye in the future as soon as the Debian team makes the switch, whereas Alpine [X], Fedora [X] and others always refer to the same distribution release and remain stable over time.
• Some repositories, e.g. Alpine, are still missing from that page - is it planned to implement all of them in the near future or should support for them be raised in Implement endpoints based on original package name repology-webapp#66 (as written on the website) despite being a closed issue?

To elaborate a bit further on what I'm trying to achieve, I basically need a solution which retrieves the current package version of a specific distribution repository, based on the name which gets originally passed to the respective package manager, e.g. I'd like to lookup musl-tools for Debian Buster and get the exact same version as in the OS repositories which currently is 1.1.21-2, just as if I'd visit https://packages.debian.org/buster/musl-tools.

As of today, I can find the original version in the origversion attribute, however musl-tools can't be found (no binary packages available for Debian) and I'm unable to specifically pin the repository to buster, as only stable is available on Repology.

[AMDmi3]

which weirdly enough mentions .svg in the select field, but returns a JSON as expected

Fixed, thanks!

Some repositories, including Debian, are only parsing source packages while others also feature binary packages - is there any specific reasoning behind this?

No specific reason - Repology uses whatever is the most suitable source to extract usable package information from. There are plans to support binary packages in addition to source where it is possible (e.g. Debian), but right now it would introduce a lot of extra data to process for little to no benefit - in fact, it would only provide binary package names (may be usable, however may also be extracted from source package information) and archs the binary package is available for (not currently used).

Some repositories are referred to with a name which changes over time, e.g. Debian Stable will switch from buster to bullseye in the future as soon as the Debian team makes the switch, whereas Alpine [X], Fedora [X] and others always refer to the same distribution release and remain stable over time.

It was unimportant till now and it was convenient to not have to track new releases for another distro. I'm all for switching it to fixed releases if there's demand.

Some repositories, e.g. Alpine, are still missing from that page - is it planned to implement all of them in the near future or should support for them be raised in repology/repology-webapp#66 (as written on the website) despite being a closed issue?

The plan was to do it on-demand. An issue here is that I don't have clear understanding of what can be considered source and binary package names for all distros, and I don't want to provide just any value which can change later and break links. See https://repology.org/repositories/fields, Names column for an overview S and B mean that there's notion of source and binary names for the repo, N means this was not yet considered. Alpine looks good, will enable it now. Please list other repos you need.

As of today, I can find the original version in the origversion attribute, however musl-tools can't be found (no binary packages available for Debian) and I'm unable to specifically pin the repository to buster, as only stable is available on Repology.

So as I understand what you need is:

• more repos supported in project-by tool (please list)
  should be as easy as in adding repository to a whitelist, or still quite easy as in deciding which are source and binary package names for a distro, in which I may need a suggestion
• fixed Debian releases
  would be easy to just add new repository entries for fixed releases, but to gracefully remove old ones some extra manual work is needed (preserve history, add redirects, avoid unwanted events in feeds etc.)
• add binary package names for Debian
  quick&dirty implementation can be implemented soon

[ppmathis]

Thank you very much for your detailed reply! Let me first start by mentioning my specific use case for Repology, as I just noticed that I did not do so before. I've contributed to Renovate in the past, which is an open-source solution for automatically managing dependencies in Git-based projects from various sources. It automatically detects common dependencies (e.g. image references in Dockerfile, Python packages in a requirements.txt, package.json from Node.js modules, ...), looks them up using various data sources and then acts in a configurable way if a newer version was found.

I have a larger set of Docker images which I'd like to have automatically maintained as well, so that whenever a new package in a distribution (e.g. Alpine Linux) gets released, the Dockerfile would automatically be adjusted, triggering an image rebuild. While Renovate can already parse these files really well, a data source for OS packages simply does not exist. Some distributions offer HTML-based pages with current version information, but it's not pretty at all to fetch. It was also a consideration on my side to just implement all possible repository formats by myself, but this seemed really tedious.

So I was made aware of another contributor (which is looking for the same feature) that Repology exists and getting that added to the project would be superb, as it would open up a completely new way of keeping things maintained while only having to implement one API endpoint. As you can imagine though, I'd need an API endpoint which is specifically able to answer Get the current version for [package name as specified with apt-get install] in [Debian Buster], which is why I've specifically asked for the API endpoints mentioned in my previous comments.

I hope this makes it somewhat easier for you to judge (a) if Repology will be a good fit for this, (b) what I am trying to achieve and (c) how things could be implemented on your side, if interested. What are your thoughts on this? Let me quickly answer your summarized task list as well:

more repos supported in project-by tool (please list)

As of today, I'd be fine with just Debian and Alpine being supported, however other users might chime in here as well in the future and request some other popular distributions.

fixed Debian releases

Exactly, I'd like to be able to clearly pin a specific distribution release version. As you mentioned though, this will clearly require some maintenance every here and then to avoid bloating the database, but I can't really judge how hard that would be to implement as I'm not familiar with the codebase.

add binary package names for Debian

Correct, being able to look up binary packages to retrieve their current version would be what is required for implementing the use case mentioned above.

[AMDmi3]

Ok, alpine added and I'll start working on other two changes after I finish some ongoing work related to vulnerabilities support.

[AMDmi3]

Binary names support is ready.
https://repology.org/tools/project-by?repo=debian_stable&name_type=binname&target_page=project_versions&name=0ad-data
Was: 404, now: 302 → https://repology.org/project/0ad/versions

[ppmathis]

Binary names support is ready.

This is looking good and seems to work fine for Debian Stable. I've spotted that tools/project-by returns 500 Internal Server Error when the given repository name does not exist, e.g.: https://repology.org/tools/project-by?repo=this-is-invalid&name_type=binname&target_page=api_v1_project&name=gcc

Additionally, repositories which are not yet supported return 403 Forbidden, e.g.:
https://repology.org/tools/project-by?repo=alpine_3_11&name_type=binname&target_page=api_v1_project&name=gcc

Those are just minor details in error handling though where a more elaborate API response might be wise. Aside from that, everything is working fine - thank you very much for getting this implemented! The only features I'm now still missing as of today are:

• Adding tools/project-by support for Alpine 3.11 + 3.12 (not yet available on Repology)
• Precise versioning of Debian repositories (Debian 10, Debian 9, ...)

Shall I create separate issues for those two points or are you fine tracking this in the current issue? While the Debian versioning is fairly low priority on my side, I'm mostly looking forward to Alpine 3.11 + 3.12 support as this would allow me to cover 95% of my use cases.

[AMDmi3]

I've spotted that tools/project-by returns 500 Internal Server Error when the given repository name does not exist

Yes, I've seen these in the logs. Supposed to be 404, will fix.

Additionally, repositories which are not yet supported return 403 Forbidden

This was intentional, but I didn't realize that Alpine was still disallowed.

Anyway, better handling for these (as pages with proper explanation) is in the works.

Adding tools/project-by support for Alpine 3.11 + 3.12 (not yet available on Repology)

Alpine 3.12 repo added, alpine family allowed for project-by, so this should work.

Precise versioning of Debian repositories (Debian 10, Debian 9, ...)

This is still pending.

Shall I create separate issues for those two points or are you fine tracking this in the current issue?

That's kind of too late, as this did not belong to this very issue from the start, but the whole discussion is already there and there's no way to split the issue. Let's leave it like this. Btw, Debian repository renames are related to repology/repology-updater#998.

[gzmarstone]

Is it possible to add Amazon Linux 2 to the supported repositories?