Skip to content

docs(scraper-studio): document 3-job cap, propose PR-11 backoff + stu…#19

Open
anil-bd wants to merge 2 commits into
brightdata:mainfrom
anil-bd:docs/scraper-studio-document-cap-and-backoff
Open

docs(scraper-studio): document 3-job cap, propose PR-11 backoff + stu…#19
anil-bd wants to merge 2 commits into
brightdata:mainfrom
anil-bd:docs/scraper-studio-document-cap-and-backoff

Conversation

@anil-bd

@anil-bd anil-bd commented May 18, 2026

Copy link
Copy Markdown
Contributor

…b asks

The AI Flow caps concurrent scraper create generations per account (currently 3). Today the cap is undocumented in this skill, in the bdata scraper --help output, and in api-flow.md — users discover it by getting 429s in production.

This commit lands the documentation half of PR-11:

  • SKILL.md "Common mistakes" gets entry 9 naming the cap and pointing at the new CLI flags (--max-retries, --no-retry) that handle it.
  • SKILL.md "Troubleshooting" gets two new rows: one for the expected in-progress backoff stderr line, one for the retries-exhausted case (actionable, points at the dashboard).
  • references/api-flow.md documents the cap inline on the /dca/collectors/{id}/automate_template endpoint and notes the stub-collector pathology (template POST always succeeds, so a 429 here leaves a half-built collector behind).
  • New proposal at proposals/PR-11-backoff.md captures the full story: Half A (client-side auto-backoff + stub-recovery note — shipped in the cli repo) and Half B (open server-side asks — reject template POST when cap is full, expose DELETE /dca/collector/{id}).

No content change to prompts.md, recipes.md, or the rest of SKILL.md.

The CLI side ships separately on cli@feat/scraper-create-429-backoff.

anil-bd added 2 commits May 18, 2026 22:38
…b asks

The AI Flow caps concurrent `scraper create` generations per account
(currently 3). Today the cap is undocumented in this skill, in the
`bdata scraper --help` output, and in api-flow.md — users discover
it by getting 429s in production.

This commit lands the documentation half of PR-11:

* SKILL.md "Common mistakes" gets entry 9 naming the cap and pointing
  at the new CLI flags (--max-retries, --no-retry) that handle it.
* SKILL.md "Troubleshooting" gets two new rows: one for the
  expected in-progress backoff stderr line, one for the
  retries-exhausted case (actionable, points at the dashboard).
* references/api-flow.md documents the cap inline on the
  /dca/collectors/{id}/automate_template endpoint and notes the
  stub-collector pathology (template POST always succeeds, so a
  429 here leaves a half-built collector behind).
* New proposal at proposals/PR-11-backoff.md captures the full
  story: Half A (client-side auto-backoff + stub-recovery note —
  shipped in the cli repo) and Half B (open server-side asks —
  reject template POST when cap is full, expose DELETE
  /dca/collector/{id}).

No content change to prompts.md, recipes.md, or the rest of SKILL.md.

The CLI side ships separately on cli@feat/scraper-create-429-backoff.
…-document-cap-and-backoff

# Conflicts:
#	skills/scraper-studio/SKILL.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant