New GitHub rate limits

[jpmckinney]

https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/

https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28

You can make unauthenticated requests if you are only fetching public data. Unauthenticated requests are associated with the originating IP address, not with the user or application that made the request. The primary rate limit for unauthenticated requests is 60 requests per hour.

I think this will mainly be a problem for OCDS extensions, where we would need to add authentication (or some other solution).

Grep'ing for:

• githubusercontent (-*.json,-*.po)
• Base URL|Download URL
• ["']dependencies["']
• testDependencies

Action needed for:

• extension_registry.py: This package abstracts all interactions with OCDS extensions, so it is the ideal place to add authentication to (or rewrite) all raw.githubusercontent.com requests.
• pelican-backend: See Add fallback in case remote codelists are inaccessible pelican-backend#48, which was an issue already. Otherwise, no requests, only documentation and test fixtures.
• software-development-handbook: It loads two images remotely. Move them into the repository.

Action potentially needed for:

• deploy: We configured Salt to make some requests. If dependencies also make requests, it could potentially exceed the rate limit. I think (hope) Salt and dependencies check the HTTP status, in which case we'll know if there's an error. Hard to know without deploying a new server (which would generate the most requests).
• cove-oc4ids and lib-cove-oc4ids: OC4IDS has no extension mechanism. A workflow (run on GitHub Actions) makes one request for example data. It otherwise calls load_core_codelists from lib-cove. Depends on whether lib-cove is updated. @jarofgreen cc @duncandewhurst

No action needed for:

• .github: A workflow (run on GitHub Actions) makes one request for deploy-docs.sh.
• bi.open-contracting.org: Makefile (run locally) makes fewer than 60 requests to do one-time project setup.
• cardinal-rs: The documentation describes a request to install Transifex CLI locally (which we might move away from).
• cove-ocds: The tests make requests (near CUSTOM_SERVER_URL) (unsure how many). When run locally, I encounter no related error.
• data-support: The Netherlands transformer (run locally) makes fewer than 60 requests.
• data-support-private: The field-list command (run locally) makes a single request to initialize ExtensionRegistry, follows extension dependencies and testDependencies, and downloads schema patches. It encounters no errors.
• european-union-support: A management command makes a single request to initialize ProfileBuilder. A bash script makes a couple requests to update files. Both are run locally.
• extension-explorer: A workflow (run on GitHub Actions) and build step (run locally) uses extension_registry.py to build a data file. Otherwise, no requests, only documentation.
• extension_registry: The tests make requests to check that "Base URL" and "Download URL" resolve. When run locally, I encounter no error. Otherwise, no requests, only text manipulation.
• field-level-mapping-template: The management command (run locally) makes requests via extension_registry.py.
• jscc: extend_schema() recurses through an extension's dependencies and testDependencies. This is only called by standard-maintenance-scripts, so we'll know if there is an error there.
• kingfisher-collect: The DatabaseStore extension makes requests via ocdskit to merge releases. Otherwise, no requests, only documentation.
• kingfisher-process: Workers (in production) make requests via extension_registry.py or lib-cove-ocds.
• lib-cove-ocds (rewrite): Makes requests via extension_registry.py.
• lib-cove-ocds (pre-rewrite): The tests make requests. When run locally, I encounter no error. I haven't checked, but I assume lib-cove makes requests, as it doesn't use extension_registry.py like the lib-cove-ocds rewrite.
• notebooks-oc4ids: The coverage notebook (run in Google Colab) makes a few requests.
• notebooks-ocds: No requests, only documentation.
• ocds-merge: No requests, only documentation.
• ocdskit: Commands (and a management command) make requests via extension_registry.py.
• pelican-frontend: A workflow (run on GitHub Actions) makes two requests for SQL migrations.
• sample-data: The documentation describes a few requests for maintenance tasks.
• sphinxcontrib-opencontracting: A directive (run locally or by deploy-docs.sh on GitHub Actions) initializes ExtensionRegistry.
• standard: Visitors request the extension_registry's JSON file when loading specific pages (script.js). The browser caches the response, so we don't expect any issue.
• standard-development-handbook: No requests, only documentation.
• standard-maintenance-scripts: The tests make requests (unsure how many). I pushed an extension, and there was no error on GitHub Actions. Some Rake tasks make requests within the rate limit, or via extension_registry.py.
• standard_profile_template (and OCDS profiles): The update and diff scripts (run locally) make fewer than 60 requests.

[jpmckinney]

The limits apparently do not apply to GitHub Pages: artifacthub/hub#4397 (comment)

GitHub is also adjusting the throttling logic to allow more legitimate traffic: https://github.com/orgs/community/discussions/159123#discussioncomment-13148279

The above cited page describes the rate limit for unauthenticated requests to the GitHub API. Apparently the githubusercontent (CDN) rate limit is undocumented: docker/actions-toolkit#696

@jarofgreen Have you actually hit the limit? How? How many requests?

[jarofgreen]

I did see that "we'll adjust" comment (but then lost it again, thanks for ref!).

I haven't done concrete experiments to work out the exact limit; it may always change and may hinge on factors I didn't think to test (earlier there are some reports that Language header may change it).

But yes, we have seen it. For instance, when getting 24 requests very quickly on the CDN. https://github.com/codeforIATI/iatikit/blob/dev/iatikit/utils/download.py#L237

[jpmckinney]

Aha. For OCDS, instead of requesting multiples files directly from GitHub like at the iatikit link, for OCDS extensions, we instead recognize GitHub, GitLab and Bitbucket links, download and cache ZIP files, and read files from that cache.

We do something similar for OCDS itself.

The requests cache is configured with environment variables.

Since we essentially make one request per repository, and cache carefully, we haven't encountered this issue (and I anticipate we will not encounter it, since GitHub's goal is to respond to valid requests).

I think IATI Kit does something a bit strange by requesting 24 files instead of one ZIP file, so, to me, it is not unusual that GitHub punishes that kind of code.

In any case, our approach already limits requests, so I don't expect we need to take any additional precautions, so I'll close this issue.