Skip to content

fix(proxy): raise ValueError for unknown IP_PROXY_PROVIDER_NAME instead of crashing with AttributeError#925

Open
kapil971390 wants to merge 1 commit into
NanmiCoder:mainfrom
kapil971390:fix/proxy-provider-unknown-name-error
Open

fix(proxy): raise ValueError for unknown IP_PROXY_PROVIDER_NAME instead of crashing with AttributeError#925
kapil971390 wants to merge 1 commit into
NanmiCoder:mainfrom
kapil971390:fix/proxy-provider-unknown-name-error

Conversation

@kapil971390

Copy link
Copy Markdown

问题

config.pyIP_PROXY_PROVIDER_NAME 被设置为一个不存在于 IpProxyProvider 字典中的值时(例如把 "kuaidaili" 拼写成 "kuaidalli"),IpProxyProvider.get() 会返回 None,这个 None 被静默地传入 ProxyIpPool.ip_provider

程序在启动时没有任何报错,直到调用 load_proxies() 时才崩溃:

AttributeError: 'NoneType' object has no attribute 'get_proxy'

此错误信息完全不提示根本原因,用户很难判断问题出在 IP_PROXY_PROVIDER_NAME 配置上。

复现步骤

  1. config/base_config.py 中将 IP_PROXY_PROVIDER_NAME 改为错误值,例如 "kuaidalli"
  2. ENABLE_IP_PROXY = True
  3. 启动任意爬虫
  4. create_ip_pool()load_proxies() 抛出 AttributeError: 'NoneType' object has no attribute 'get_proxy'

方案

create_ip_pool() 中,.get() 取到 None 时立即抛出 ValueError,并在错误信息中列出有效选项:

# proxy/proxy_ip_pool.py — before
ip_provider=IpProxyProvider.get(config.IP_PROXY_PROVIDER_NAME),  # None if unknown

# after
ip_provider = IpProxyProvider.get(config.IP_PROXY_PROVIDER_NAME)
if ip_provider is None:
    raise ValueError(
        f"Unknown proxy provider: '{config.IP_PROXY_PROVIDER_NAME}'. "
        f"Valid options: {list(IpProxyProvider.keys())}"
    )

修复后:

ValueError: Unknown proxy provider: 'kuaidalli'. Valid options: ['kuaidaili', 'wandouhttp', 'static']

测试

test/test_proxy_ip_pool.py 新增回归测试,覆盖拼写错误、空字符串、大小写错误及三个合法 provider 名称不受影响。


🤖 Generated with Claude Code

When IP_PROXY_PROVIDER_NAME in config.py is set to an unrecognised
value (e.g. typo "kuaidalli" instead of "kuaidaili"), IpProxyProvider.get()
returns None which is silently stored in ProxyIpPool.ip_provider.  The
crash surfaces later as:

  AttributeError: 'NoneType' object has no attribute 'get_proxy'

with no hint about what went wrong or how to fix it.

Add an explicit None-check in create_ip_pool() that raises ValueError
with the bad name and the list of valid options, so the root cause is
immediately visible.  Also add a regression test covering the four
invalid-name edge cases (typo, empty string, wrong case, unknown name)
and the three valid names to confirm they are unaffected.
@kapil971390 kapil971390 requested a review from NanmiCoder as a code owner June 26, 2026 06:44
@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. bug Something isn't working labels Jun 26, 2026
@kapil971390

kapil971390 commented Jun 26, 2026

Copy link
Copy Markdown
Author

Reproduction screenshots

Before fixip_provider=None stored silently, crash surfaces only on load_proxies():

mediacrawler-proxy-BEFORE

After fixValueError raised immediately with actionable message:

mediacrawler-proxy-AFTER

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant