linkcheck inconsistent behavior from client

[fmigneault]

Describe the bug

Running on this project: https://github.com/crim-ca/weaver/

Local Ubuntu 20.04.6 LTS instance, Python 3.10.13 produces :

(         changes: line 1633) broken    https://github.com/geopython/pywps/commit/343d82539576b1e73eee3102654749c3d3137cff - 406 Client Error: Not Acceptable for url: https://github.com/geopython/pywps/commit/343d82539576b1e73eee3102654749c3d3137cff
(autoapi/weaver/base/index: line   72) redirect  https://stackoverflow.com/a/5191224 - with Found to https://stackoverflow.com/questions/5189699/how-to-make-a-class-property/5191224#5191224

On GitHub CI, Python 3.10.13 produces (https://github.com/crim-ca/weaver/actions/runs/7962895777/job/22037962351?pr=604) :

(         changes: line 1633) ok        https://github.com/geopython/pywps/commit/343d82539576b1e73eee3102654749c3d3137cff
(autoapi/weaver/base/index: line   72) broken    https://stackoverflow.com/a/5191224 - 403 Client Error: Forbidden for url: https://stackoverflow.com/a/5191224

Both locations run Sphinx 7.2.6

The errors for the same URLs are replicated over multiple runs, but somehow do not align on different machines.

How to Reproduce

Use the listed URLs in some RST document, and see if they get flagged by linkcheck.
The command make check-links can be used from https://github.com/crim-ca/weaver/ to test exactly the repo where the problem occurs.

Environment Information

Platform:              linux; (Linux-5.15.0-97-generic-x86_64-with-glibc2.31)
Python version:        3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0])
Python implementation: CPython
Sphinx version:        7.2.6
Docutils version:      0.18.1
Jinja2 version:        3.1.2
Pygments version:      2.17.1

> pip freeze
alabaster==0.7.13
alembic==1.12.1
amqp==5.2.0
anyascii==0.3.2
argcomplete==3.1.6
asciitree==0.3.3
astor==0.8.1
astroid==2.15.8
attrs==23.1.0
Babel==2.13.1
bandit==1.7.5
Beaker==1.12.1
beautifulsoup4==4.12.2
billiard==4.2.0
boto3==1.29.4
boto3-stubs==1.29.4
botocore==1.32.4
botocore-stubs==1.32.4
bump2version==1.0.1
CacheControl==0.13.1
celery==5.3.5
certifi==2023.11.17
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
cligj==0.7.2
codacy-coverage==1.3.11
colander==1.8.3
coloredlogs==15.0.1
contextlib2==21.6.0
cornice==6.0.1
cornice-swagger @ git+https://github.com/fmigneault/cornice.ext.swagger.git@b66bd4d85f5f5550f39d046742ae9799aee1f153
coverage==7.3.2
cryptography==41.0.5
cwl-upgrader==1.2.10
cwl-utils==0.31
cwltool==3.1.20230906142556
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.7
dnspython==2.4.2
doc8==0.11.2
docformatter==1.5.0
docker==6.1.3
docutils==0.18.1
dparse==0.6.3
duration==1.1.1
esgf-compute-api @ git+https://github.com/ESGF/esgf-compute-api.git@8faf829abb3ba0434715c4db44440cfb9c71b55f
execnet==2.0.2
fasteners==0.19
filelock==3.13.1
fiona==1.9.5
flake8==5.0.4
flynt==1.0.1
future==0.18.3
geotiff==0.2.10
gitdb==4.0.11
GitPython==3.1.40
greenlet==3.0.1
gunicorn==21.2.0
humanfriendly==10.0
humanize==4.8.0
hupper==1.12
idna==3.4
imagesize==1.4.1
importlib-resources==6.1.1
iniconfig==2.0.0
iso8601==2.1.0
isodate==0.6.1
isort==5.12.0
Jinja2==3.1.2
jmespath==1.0.1
json2xml==4.1.0
jsonschema==4.20.0
jsonschema-specifications==2023.11.1
kombu==5.3.4
lazy-object-proxy==1.9.0
lxml==4.9.3
Mako==1.3.0
markdown-it-py==3.0.0
MarkupSafe==2.1.3
mccabe==0.7.0
mdurl==0.1.2
mistune==2.0.5
mock==3.0.5
moto==4.2.14
msgpack==1.0.7
mypy==1.7.0
mypy-boto3-s3==1.29.3
mypy-extensions==1.0.0
networkx==3.2.1
numcodecs==0.12.1
numpy==1.26.2
oauthlib==3.2.2
OWSLib==0.29.3
packaging==21.3
parameterized==0.9.0
Paste==3.7.1
PasteDeploy==3.1.0
path==16.7.1
path.py==12.5.0
pbr==6.0.0
Pint==0.22
plaster==1.1.2
plaster-pastedeploy==1.0.1
platformdirs==4.0.0
pluggy==1.3.0
prompt-toolkit==3.0.41
prov==1.5.1
psutil==5.9.6
py==1.11.0
pycodestyle==2.9.1
pycparser==2.21
pydocstyle==6.3.0
pydot==1.4.2
pyflakes==2.5.0
Pygments==2.17.1
pylint==2.17.7
pylint-per-file-ignores==1.3.2
pylint-quotes==0.2.3
pymongo==4.6.0
pyparsing==3.1.1
pyproj==3.6.1
pyramid==2.0.2
pyramid-beaker==0.8
pyramid-celery @ git+https://github.com/crim-ca/pyramid_celery.git@3d4c5096661176b176cab4b7d31b282317c70301
pyramid-mako==1.1.0
pyramid-rewrite==0.2
pyramid-storage==1.3.0
pytest==6.2.5
pytest-fixture-config==1.7.0
pytest-httpserver==1.0.8
pytest-rerunfailures==12.0
pytest-server-fixtures==1.7.0
pytest-shutil==1.7.0
python-dateutil==2.8.2
pytz==2023.3.post1
pywps==4.6.0
PyYAML==6.0.1
rdflib==7.0.0
referencing==0.31.0
requests==2.31.0
requests-file==1.5.1
requests-magpie @ git+https://github.com/Ouranosinc/requests-magpie@b26ed3c2cee4395c1bd745c7e102a442ba2199ae
responses==0.24.1
restructuredtext-lint==1.4.0
retry==0.9.2
rich==13.7.0
rpds-py==0.13.1
ruamel.yaml==0.17.40
ruamel.yaml.clib==0.2.8
s3transfer==0.7.0
safety==2.3.5
schema-salad==8.4.20231117150958
shapely==2.0.2
shellescape==3.8.1
simplejson==3.19.2
six==1.16.0
smmap==5.0.1
snowballstemmer==2.2.0
soupsieve==2.5
Sphinx==7.2.6
sphinx-argparse==0.4.0
sphinx-autoapi==3.0.0
sphinx-autodoc-typehints==1.25.2
sphinx-paramlinks==0.6.0
sphinx-rtd-theme==1.3.0
sphinxcontrib-applehelp==1.0.7
sphinxcontrib-devhelp==1.0.5
sphinxcontrib-htmlhelp==2.0.4
sphinxcontrib-jquery==4.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.6
sphinxcontrib-redoc==1.6.0
sphinxcontrib-serializinghtml==1.1.9
SQLAlchemy==2.0.23
stevedore==5.1.0
stopit==1.1.2
termcolor==2.3.0
tifffile==2023.9.26
toml==0.10.2
tomli==2.0.1
tomlkit==0.12.3
translationstring==1.4
types-awscrt==0.19.13
types-s3transfer==0.7.0
typing_extensions==4.8.0
tzdata==2023.3
untokenize==0.1.1
urllib3==2.0.7
urlmatch==1.0.1
venusian==3.1.0
vine==5.1.0
waitress==2.1.2
wcwidth==0.2.11
-e git+ssh://git@github-crim/crim-ca/weaver.git@781dac1bb5429beba99580a4f1118224ef46b6a1#egg=weaver
WebOb==1.8.7
websocket-client==1.6.4
WebTest==3.0.0
Werkzeug==3.0.1
wrapt==1.16.0
WSGIProxy==0.2.2
WSGIProxy2==0.5.1
xmltodict==0.13.0
zarr==2.12.0
zope.deprecation==5.0
zope.interface==6.1

Sphinx extensions

https://github.com/crim-ca/weaver/blob/d911c161721c2e25ee4663d34a49d2b218261710/docs/source/conf.py#L54-L68

Additional context

Error seen on crim-ca/weaver#604 (but unrelated changes)
Last run that did not raise any issue by linkcheck as on crim-ca/weaver@d234e11 (ie: https://github.com/crim-ca/weaver/actions/runs/7890099969/job/21545963203)
No apparent change to documentation links between them (crim-ca/weaver@d234e11...cmd-err-handling).

[jayaddison]

Hi @fmigneault - unfortunately I think that this is most likely to be due to stackoverflow choosing to respond differently to HTTP requests depending on the origin. If that's so, then in the short-term may have to adapt by using linkcheck configuration options.

I'm going to convert this issue into a discussion topic since it isn't immediately clear that there is a bug or enhancement request for the Sphinx codebase here - but perhaps we'll discover one as a result.

[fmigneault]

As a demonstration about how inconsistent linkcheck can be regarding which status responses are obtained, I ran the 2 following PRs at roughly the same time:

• let sphinx pick a more recent user-agent for docs linkcheck crim-ca/weaver#609
• use typing Unpack for request options crim-ca/weaver#610

The first one failed a few times until I gave up, and applied an explicit ignore rule for the problematic URL.
The second PR, however, did not encounter any of the problem HTTP codes, even though it did not change anything regarding doc links.
This is comparison to crim-ca/weaver@c079e4e (at that time master for the 2 PRs), which was still indicating a broken link due to HTTP 405.

[fmigneault]

@jayaddison

I've been getting this issue again recently. It came back all of a sudden.

I get the following results.
It seems the option Sphinx injects is at cause.

sphinx/sphinx/builders/linkcheck.py

Line 496 in e44a40e

yield self._session.head, {'allow_redirects': True}

r = requests.head("https://stackoverflow.com/a/44776960"); print(r.status_code); print(json.dumps(dict(r.headers), indent=2))
302
{
  "Date": "Tue, 18 Aug 2026 19:33:28 GMT",
  "Connection": "keep-alive",
  "Location": "/questions/6358481/using-functools-lru-cache-with-dictionary-arguments/44776960#44776960",
  "CF-Ray": "a2d359b79b6f5933-YUL",
  "CF-Cache-Status": "DYNAMIC",
  "Cache-Control": "private",
  "Server": "cloudflare",
  "x-worker-origin-response-time": "75000000",
  "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
  "X-DNS-Prefetch-Control": "off",
  "content-security-policy": "upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com",
  "feature-policy": "microphone 'none'; speaker 'none'",
  "x-clacks-overhead": "GNU Terry Pratchett",
  "x-frame-options": "SAMEORIGIN",
  "x-request-guid": "5ea44a71-098c-463c-a6e7-3e9af5f761a8",
  "Set-Cookie": "prov=ea57382b-80e9-46f8-bf89-ea11c9d39d70; expires=Wed, 18 Aug 2027 19:33:28 GMT; domain=.stackoverflow.com; path=/; secure; samesite=none; httponly, __cflb=02DiuFA7zZL3enAQJD3AX8ZzvyzLcaG7wPF4Bu2Lt4xa8; HttpOnly; SameSite=None; Secure; Path=/; Expires=Wed, 19 Aug 2026 18:33:28 GMT, prov=ea57382b-80e9-46f8-bf89-ea11c9d39d70; Path=/; HttpOnly; Domain=stackoverflow.com, __cf_bm=tRWqLuTvhclG6VU1U0ymdZrBSn0r4eM33JIkBLfVkns-1787081608.901546-1.0.1.1-Jj8Pw0RVZtHnjIOEHdZOcXi1DxxVxy12VUU86Qi_fh57xJkJX71wzJxgxMZzqdbuEFmVVx7t54uIiG5oko84Q2A_vY.eN3CcQM.qTmnoD7a5SLwUxUJpuFDnIdOdxLocQdu1uTAwShh0GceB4GIjDQFFXl4HtW_eQoOW8IWk7ro; HttpOnly; SameSite=None; Secure; Path=/; Domain=stackoverflow.com; Expires=Tue, 18 Aug 2026 20:03:28 GMT"
}

r = requests.head("https://stackoverflow.com/a/44776960", allow_redirects=True); print(r.status_code); print(json.dumps(dict(r.headers), indent=2))
403
{
  "Date": "Tue, 18 Aug 2026 19:33:52 GMT",
  "Content-Type": "text/html; charset=UTF-8",
  "Connection": "close",
  "Accept-Ch": "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA",
  "Cf-Mitigated": "challenge",
  "Content-Security-Policy": "default-src 'none'; script-src 'nonce-8xQf0UCdfdlCY4q94nHsgu' 'unsafe-eval' https://challenges.cloudflare.com; script-src-attr 'none'; style-src 'unsafe-inline'; img-src 'self' https://challenges.cloudflare.com; connect-src 'self' https://challenges.cloudflare.com; frame-src 'self' https://challenges.cloudflare.com blob:; child-src 'self' https://challenges.cloudflare.com blob:; worker-src blob:; form-action http: https:; base-uri 'self'",
  "Server": "cloudflare",
  "Critical-Ch": "Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA",
  "Cross-Origin-Embedder-Policy": "require-corp",
  "Cross-Origin-Opener-Policy": "same-origin",
  "Cross-Origin-Resource-Policy": "same-origin",
  "Origin-Agent-Cluster": "?1",
  "Permissions-Policy": "accelerometer=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=(),xr-spatial-tracking=*",
  "Referrer-Policy": "same-origin",
  "Server-Timing": "chlray;desc=\"a2d35a4dbdf0a30e\"",
  "X-Content-Type-Options": "nosniff",
  "X-Frame-Options": "SAMEORIGIN",
  "set-cookie": "__cf_bm=PDY0GWsMnVy3gU6FTQ6fvMWWB8qUpkmOg7BbxL4LuLI-1787081632.9250543-1.0.1.1-Df.kisCAZfC.NmUZm6Sco3WFJjAfOxR8UceNsdp506iKOi_C6kZeEztmpCLRgqJFsSlArjrp7fkprQByM.AnASdo6RlaZ9XIV9Q5ec8LCuC5jNe_TbTgk.4QZ7qKArXgPOjLYewUyFEltpVgoGClAjacHhUvRdB6Eh4kmXFKPRM; HttpOnly; SameSite=None; Secure; Path=/; Domain=stackoverflow.com; Expires=Tue, 18 Aug 2026 20:03:52 GMT",
  "X-DNS-Prefetch-Control": "off",
  "Content-Encoding": "gzip",
  "CF-RAY": "a2d35a4dbdf0a30e-YUL"
}

Technically, 302 is "Found" so it could be considered sufficient.

Is there a way to avoid the failure caused by the redirect?
Only way I see is to completely ignore the link which is not ideal.

[jayaddison]

@fmigneault because the problem seems intermittent, it may be difficult to solve this reliably -- even so, please could you inspect the linkcheck_allowed_redirects setting documentation to figure out if that might help?

[fmigneault]

The 403 error happens consistently using:

requests.head("https://stackoverflow.com/a/44776960", allow_redirects=True)
# or using its full redirect URI:
requests.head("https://stackoverflow.com/questions/6358481/using-functools-lru-cache-with-dictionary-arguments/44776960", allow_redirects=True)

I think it is acloudflare policy that blocks it (ie: "Cf-Mitigated": "challenge" hinting at a CAPTCHA)
I tried with different User-Agent mocking a browser, same result. I am not sure how it detects it exactly.
The URIs are fine if using a browser.

Using the feature to allow any redirects, I get a mix of rate-limited/forbidden resutls.

• try linkcheck_allowed_redirects for stackoverflow doc URI requests crim-ca/weaver#1004
• https://github.com/crim-ca/weaver/actions/runs/32267011042/job/96209738775

[...]
-rate limited-   https://stackoverflow.com/a/44776960 | sleeping...
(autoapi/weaver/notify/index: line   42) broken    https://stackoverflow.com/a/55147077 - 429 Client Error: Too Many Requests for url: https://stackoverflow.com/a/55147077
(autoapi/weaver/base/index: line   80) broken    https://stackoverflow.com/a/5191224 - 429 Client Error: Too Many Requests for url: https://stackoverflow.com/a/5191224
(autoapi/weaver/utils/index: line  150) broken    https://stackoverflow.com/a/44776960 - 403 Client Error: Forbidden for url: https://stackoverflow.com/questions/6358481/using-functools-lru-cache-with-dictionary-arguments/44776960#44776960

[jayaddison]

The HTTP 429 rate limiting errors can only be raised (I think) if an HTTPError is raised by the requests library -- meaning, basically, an HTTP 4xx or 5xx response from the server.

In other words: for those first two cases, I don't think that an HTTP 302 redirect was issued.

I'm not completely sure about that, but that's my theory.