From 4ace5121e6a97533d718756681e613a5febc3cc2 Mon Sep 17 00:00:00 2001 From: David Fischer Date: Wed, 20 May 2026 17:48:57 -0700 Subject: [PATCH 1/4] Document policies for automated access - Rate limits - Agent usage - Cross link to Markdown support - Caching - (Almost) no automated account creation --- docs/user/about/index.rst | 4 +++ docs/user/automated-access.rst | 53 ++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) create mode 100644 docs/user/automated-access.rst diff --git a/docs/user/about/index.rst b/docs/user/about/index.rst index e0c966995dd..0179ae8bd69 100644 --- a/docs/user/about/index.rst +++ b/docs/user/about/index.rst @@ -53,6 +53,9 @@ we've brought most of the most important ones below. ⏩ :doc:`/support` Read this before asking for help: How to get support and where. +⏩ :doc:`/automated-access` + Policies for automated access, scraping, and agents. + ⏩ :doc:`/glossary` A useful index of terms used in our docs @@ -72,4 +75,5 @@ we've brought most of the most important ones below. /open-source-philosophy /team /support + /automated-access /glossary diff --git a/docs/user/automated-access.rst b/docs/user/automated-access.rst new file mode 100644 index 00000000000..d6d6115d3d2 --- /dev/null +++ b/docs/user/automated-access.rst @@ -0,0 +1,53 @@ +Automated access to Read the Docs +================================= + +Read the Docs generally welcomes most respectful crawlers, spiders, and bots on documentation sites. +Agent use of Read the Docs is also acceptable. + +Please respect the following policies when automating access to Read the Docs: + +* **Rate limits**: + Keep to under **4 requests per second**. + Small bursts above this will not result in rate limiting or bans. + Many projects build their documentation as a downloadable zip file or other formats. + Please keep these downloads to 1 request per second. + API requests have their own rate limits (see :ref:`API rate limiting `). +* **Product dashboards**: + Scraping pages from our product dashboards on |org_brand| or |com_brand| is frowned upon in bulk. +* **Copyright**: + While automated access to documentation is acceptable, + content is the copyright of the respective project owners and contributors. +* **Account creation**: + Automated account creation is not acceptable beyond a single personal account. +* **Bot identification**: + Identify yourself. Put a domain or email in your user agent. + This is very helpful and reduces your chances of being blocked. +* **Markdown format**: + Bots can request documentation pages as Markdown directly. + See :doc:`/reference/markdown-for-agents`. +* **Traffic analysis**: + We analyze our traffic beyond simple IP rate limiting. + We frown upon large amounts of traffic impersonating real browsers or traffic distributed to get around rate limits. + Distributed traffic is far more likely to result in a ban. +* **Cloudflare verification**: + We use Cloudflare heavily (thanks Cloudflare for sponsoring |org_brand|). + Cloudflare `verified bots `_ + are less likely to be limited or blocked. +* **Caching**: + Read the Docs implements most caching strategies. + Well-behaved crawlers can take advantage of cache headers, etags, date headers, and more. + Cached requests generally don't count against rate limits. + +Please help keep the web open by respecting these policies. +Read our blog post about `AI crawler abuse `_ to understand the impact of poor behavior. + + +Special considerations +---------------------- + +We are open to special considerations or increased rate limits on a case-by-case basis. +Please :doc:`contact support ` to discuss your needs. + +If you believe you have been incorrectly limited or blocked, +please :doc:`contact support ` with details about your user agent and requesting IP block. +Please include the Cloudflare `Ray ID `_ from the blocking page if possible. From 68ac7344600779c54895d31dc55b124160428ec4 Mon Sep 17 00:00:00 2001 From: David Fischer Date: Wed, 20 May 2026 17:55:03 -0700 Subject: [PATCH 2/4] Minor grammar fixes --- docs/user/automated-access.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/user/automated-access.rst b/docs/user/automated-access.rst index d6d6115d3d2..0240e58323d 100644 --- a/docs/user/automated-access.rst +++ b/docs/user/automated-access.rst @@ -1,7 +1,7 @@ Automated access to Read the Docs ================================= -Read the Docs generally welcomes most respectful crawlers, spiders, and bots on documentation sites. +Read the Docs generally welcomes respectful crawlers, spiders, and bots on documentation sites. Agent use of Read the Docs is also acceptable. Please respect the following policies when automating access to Read the Docs: @@ -30,7 +30,7 @@ Please respect the following policies when automating access to Read the Docs: We frown upon large amounts of traffic impersonating real browsers or traffic distributed to get around rate limits. Distributed traffic is far more likely to result in a ban. * **Cloudflare verification**: - We use Cloudflare heavily (thanks Cloudflare for sponsoring |org_brand|). + We use Cloudflare heavily (thanks to Cloudflare for sponsoring |org_brand|). Cloudflare `verified bots `_ are less likely to be limited or blocked. * **Caching**: @@ -49,5 +49,5 @@ We are open to special considerations or increased rate limits on a case-by-case Please :doc:`contact support ` to discuss your needs. If you believe you have been incorrectly limited or blocked, -please :doc:`contact support ` with details about your user agent and requesting IP block. +please :doc:`contact support ` with details about your user agent and the requesting IP block. Please include the Cloudflare `Ray ID `_ from the blocking page if possible. From f08aca2eb463c922d63dc4da74826b6aff6bfa74 Mon Sep 17 00:00:00 2001 From: David Fischer Date: Wed, 20 May 2026 18:02:00 -0700 Subject: [PATCH 3/4] Fix etag spelling --- docs/user/automated-access.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user/automated-access.rst b/docs/user/automated-access.rst index 0240e58323d..b33b017e60c 100644 --- a/docs/user/automated-access.rst +++ b/docs/user/automated-access.rst @@ -35,7 +35,7 @@ Please respect the following policies when automating access to Read the Docs: are less likely to be limited or blocked. * **Caching**: Read the Docs implements most caching strategies. - Well-behaved crawlers can take advantage of cache headers, etags, date headers, and more. + Well-behaved crawlers can take advantage of cache headers, ETags, date headers, and more. Cached requests generally don't count against rate limits. Please help keep the web open by respecting these policies. From f1217405dd0907823fa73c064dd0c8111fc8dc4b Mon Sep 17 00:00:00 2001 From: David Fischer Date: Thu, 21 May 2026 07:45:52 -0700 Subject: [PATCH 4/4] Update docs/user/automated-access.rst Co-authored-by: Eric Holscher <25510+ericholscher@users.noreply.github.com> --- docs/user/automated-access.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user/automated-access.rst b/docs/user/automated-access.rst index b33b017e60c..45c7d182854 100644 --- a/docs/user/automated-access.rst +++ b/docs/user/automated-access.rst @@ -2,7 +2,7 @@ Automated access to Read the Docs ================================= Read the Docs generally welcomes respectful crawlers, spiders, and bots on documentation sites. -Agent use of Read the Docs is also acceptable. +Agent use of Read the Docs is also supported. Please respect the following policies when automating access to Read the Docs: