Skip to content

Enable MCP for all API endpoints - #160

Open
R1ckyH wants to merge 7 commits into
blacklanternsecurity:devfrom
R1ckyH:mcp-additions
Open

Enable MCP for all API endpoints#160
R1ckyH wants to merge 7 commits into
blacklanternsecurity:devfrom
R1ckyH:mcp-additions

Conversation

@R1ckyH

@R1ckyH R1ckyH commented Apr 21, 2026

Copy link
Copy Markdown

Summary

  • Add mcp=True to 38 remaining API endpoints across 10 modules
  • MCP coverage increases from 31/73 (42.5%) to 69/73 (94.5%)
  • Only 4 WebSocket endpoints excluded (agent dock, event/activity tail, event ingest)

New MCP endpoints by module

Module Endpoints
agents create_agent, delete_agent, get_agent, get_scan_status
activity list_activities, query_activities, count_activities
assets query_assets, count_assets
emails get_emails
events insert_event, get_event, list_events, query_events, count_events, archive_old_events
findings query_findings, count_findings, set_risk
presets create_preset, update_preset, delete_preset
scans get_scans, query_scans, count_scans, cancel_scan
targets count_targets, set_default_target, create_target, update_target, copy_target, delete_target, is_blacklisted, query_targets, list_ids
technologies list_technologies, query_technologies, count_technologies

@github-actions

github-actions Bot commented Apr 21, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@R1ckyH R1ckyH changed the title Enable MCP for all non-WebSocket API endpoints Enable MCP for all API endpoints Apr 21, 2026
@R1ckyH

R1ckyH commented Apr 21, 2026

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

bls-cla-bot Bot added a commit to blacklanternsecurity/CLA that referenced this pull request Apr 21, 2026
Add mcp=True to 38 remaining endpoints across 10 modules. Only
WebSocket endpoints (agent dock, event/activity tail, event ingest)
are excluded as they use a different protocol incompatible with MCP.

Newly enabled endpoints by module:
- agents: create_agent, delete_agent, get_agent, get_scan_status
- activity: list_activities, query_activities, count_activities
- assets: query_assets, count_assets
- emails: get_emails
- events: insert_event, get_event, list_events, query_events, count_events, archive_old_events
- findings: query_findings, count_findings, set_risk
- presets: create_preset, update_preset, delete_preset
- scans: get_scans, query_scans, count_scans, cancel_scan
- targets: count_targets, set_default_target, create_target, update_target, copy_target, delete_target, is_blacklisted, query_targets, list_ids
- technologies: list_technologies, query_technologies, count_technologies

Total MCP coverage: 69/73 (94.5%)
@TheTechromancer

Copy link
Copy Markdown
Contributor

@R1ckyH thanks for the PR! Have you got a chance to test these out? Do they behave well?

@R1ckyH

R1ckyH commented Apr 24, 2026

Copy link
Copy Markdown
Author

@R1ckyH thanks for the PR! Have you got a chance to test these out? Do they behave well?

I am doing some tests on it, currently not yet finished all, but I think MCP is just another way to call the http api.
Therefore, it shouldn't be an issue calling MCP with agents if the standard api are working fine

fastapi-mcp 0.3.3 only forwarded the Authorization header, causing 401 errors
when using custom auth headers like X-API-Key. Upgrade to 0.4.0 which adds
a headers allowlist parameter, and pass ["x-api-key"] to forward the
API key from MCP client requests to internal API calls.

Co-Authored-By: R1ckyH <R1ckyH@users.noreply.github.com>
@R1ckyH

R1ckyH commented Apr 28, 2026

Copy link
Copy Markdown
Author

@TheTechromancer, I have added a new commit. This should fix the MCP API key issue mentioned in the README.
I have tested that the API-key works in Claude code mcp after the modification.

NOTE: Authentication is https://github.com/blacklanternsecurity/bbot-server/issues/52 in Cursor, Cline, and it seems most other VS Code forks. A workaround is to disable authentication with --no-authentication when starting the server. Obviously, be careful with this and don't be a dumbass.

@R1ckyH

R1ckyH commented Apr 28, 2026

Copy link
Copy Markdown
Author

I have also added some bug fixes here

…rror

HTTPStreamRoute's wrapper used @functools.wraps(), which copies
attributes like __wrapped__ and __annotations__ from the original
async generator function onto the wrapper. This caused FastAPI to
detect the wrapper as an async generator and attempt to serialize
its output via orjson.dumps(), resulting in:

  TypeError: Type is not JSON serializable: async_generator

The wrapper is NOT an async generator — it's a regular async def
that returns a StreamingResponse. Replace @functools.wraps with
manual __name__, __qualname__, and __signature__ copying to give
FastAPI the metadata it needs for routing without the async generator
markers.

This fixes streaming endpoints (list_assets, list_scans, etc.) for
both direct HTTP access and MCP tool calls via fastapi_mcp.
@R1ckyH

R1ckyH commented Apr 29, 2026

Copy link
Copy Markdown
Author

ca927c1 — Fix streaming endpoints crashing with async_generator serialization error

Error:
All type="http_stream" endpoints (e.g. /scans/list, /assets/list, /events/list) return 500 with:

TypeError: Type is not JSON serializable: async_generator
  File "fastapi/routing.py", line 671, in app
    response = actual_response_class(content=gen, **response_args)
  File "starlette/responses.py", line 45, in __init__
    self.body = self.render(content)
  File "fastapi/responses.py", line 96, in render
    return orjson.dumps(...)

Cause:
HTTPStreamRoute.wrapped_function() wraps the original async generator into a regular async def that returns StreamingResponse. However, @functools.wraps(self.orig_function) copied attributes like __wrapped__ and __annotations__ from the original async generator function onto the wrapper. FastAPI detected these markers, treated the wrapper as an async generator, and tried to serialize the raw generator object through orjson.dumps() instead of letting StreamingResponse handle it.

Fix:
Replace @functools.wraps with manual copying of only the 3 attributes FastAPI actually needs for parameter routing — __name__, __qualname__, __signature__. This gives FastAPI correct endpoint introspection without the async generator identity markers.

R1ckyH added 2 commits April 29, 2026 20:12
…pe checks

- Add None guards to all query_* endpoints (query_scans, query_assets,
  query_events, query_findings, query_targets, query_technologies)
  matching the pattern already applied to count_* endpoints
- Fix get_emails: AssetFields now inherits CustomAssetFields so the
  emails field is properly registered on the Asset model, plus use
  getattr() fallback for safe attribute access
- Fix count_activities/query_activities: make query parameter optional
  with ActivityQuery | None = None and add None guard
- Fix in_scope/is_blacklisted/is_in_target: return False instead of
  crashing when no default target exists in the database

All 45 testable MCP endpoints now pass (0 failures).
@R1ckyH

R1ckyH commented May 8, 2026

Copy link
Copy Markdown
Author

Hey @TheTechromancer, after testing the MCP endpoints against a live server with a full scan workflow (create target → start scan → agent runs → query results), I found a few issues. Wanted to flag before this goes further:

1. get_scan_status calls a non-existent agent command

agents_api.py:98 sends "get_scan_status" to the agent, but the agent only implements these 6 commands: start_scan, cancel_scan, get_agent_status, finish_scan, kill_module, get_file. The endpoint always returns "Invalid command" — it has never worked.

2. get_scan_status uses UUID type for id param

agents_api.py:96 — This generates format: uuid in the MCP tool schema, which causes some MCP clients (including Claude Code) to reject valid UUIDs with -32602 Invalid request parameters. Other endpoints like get_scan use str and work fine.

3. Scope checks (in_scope, is_in_target, is_blacklisted) always return false when targets are created with only seeds

When a target is created with target=[] (empty) and seeds=["evilcorp.com"], BBOTTarget.in_scope() only checks the target list — it ignores seeds. So scope checks always return false:

target=[], seeds=["evilcorp.com"]    → in_scope("evilcorp.com") = False
target=["evilcorp.com"], seeds=None  → in_scope("evilcorp.com") = True  # seeds auto-derived from target

The fix could be in targets_models.py:110-111 — when target is empty but seeds is provided, the seeds should also define scope.


Would you prefer I commit the fixes in this PR, or should I open a separate PR for these?

@R1ckyH

R1ckyH commented May 8, 2026

Copy link
Copy Markdown
Author

One more thing to talk about here, when I am trying to do the MCP testing, I face many issues, such as the function not yet implemented. I want to know your attitude before I continue.

@singlerider singlerider left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@R1ckyH, answering 05-08: fix both here. get_scan_status and the seeds scope bug break the MCP surface this PR ships. Same for unimplemented endpoints, implement or drop mcp=True.

Pulled the branch, ran the suite on real mongo/redis (122 passed, 6 skipped), diffed OpenAPI against dev. Blockers inline, plus two with no line to hang on:

  • No CI has ever run here. All 8 pushes show CLAAssistant only.
  • PR body says 38 endpoints, branch has 69.

🟢 ca927c1: traceback, mechanism, why the narrower copy is right. I reproduced it from that comment alone. That is the standard for bug fixes here.

🟢 You ran a live workflow and reported what broke instead of pushing the green button. Three findings here are yours.

Approach is fine. Fix and I will re-review.

# that cause FastAPI to mishandle the StreamingResponse
wrapper.__name__ = self.orig_function.__name__
wrapper.__qualname__ = self.orig_function.__qualname__
wrapper.__signature__ = inspect.signature(self.orig_function)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking. Your diagnosis is right and I reproduced it: under fastapi 0.136.1 a @functools.wraps-decorated stream wrapper raises TypeError: 'async for' requires an object with __aiter__ method; without it, 200. Under 0.128.7 (what dev pins) both work, so this breakage arrives with the fastapi bump in this same lockfile.

But this copies __name__, __qualname__, and __signature__ and not __doc__. OpenAPI descriptions drop 21 -> 13, and 8 of the losses are MCP tools this PR adds: list_assets, query_assets, query_events, query_findings, query_scans, query_targets, query_technologies, query_activities. Those descriptions are the tool prompts the model reads, so this quietly degrades the feature the PR exists to deliver.

One line fixes it, verified still 200:

wrapper.__doc__ = self.orig_function.__doc__

Comment thread bbot_server/api/mcp.py
mcp_endpoints = MCP_ENDPOINTS
log.debug(f"Creating MCP server with endpoints: {','.join(mcp_endpoints)}")
mcp = FastApiMCP(fastapi_app, include_operations=list(mcp_endpoints))
mcp = FastApiMCP(fastapi_app, include_operations=list(mcp_endpoints), headers=["x-api-key"])

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking. config.py:77 already owns this as auth_header. Read it from config instead of pasting the literal.

Also note this replaces the library default of ["authorization"] rather than extending it. Confirm nothing depends on Authorization passthrough.

return agent_status

@api_endpoint("/scan_status", methods=["GET"], summary="Get the status of an agent's scan")
@api_endpoint("/scan_status", methods=["GET"], summary="Get the status of an agent's scan", mcp=True)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking. VALID_AGENT_COMMANDS is {cancel_scan, finish_scan, get_agent_status, get_file, kill_module, start_scan}. No get_scan_status, so line 98 always returns Invalid command, exactly as you found. Implement the agent-side command or drop mcp=True. Shipping it as a tool in this state is not an option.

While you are here, switch id from UUID to str per your own finding about clients rejecting format: uuid. get_scan already uses str.

@api_endpoint("/in_scope", methods=["GET"], summary="Check if a host or URL is in scope", mcp=True)
async def in_scope(self, host: str, target_id: UUID = None) -> bool:
bbot_target = await self._get_bbot_target(target_id)
if bbot_target is None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking, two things on this block and the two below it.

Fails open. Answering "is this host blacklisted?" with False because the target was not found is the wrong direction for a safety check. Neither has any other caller in the tree, so raising BBOTServerNotFoundError costs nothing.

Seeds-only targets. Your 05-08 finding belongs in this PR, since these three are mcp=True here and this is what exposes the bug to agents. BBOTTarget.in_scope() consults only the target list, so target=[] + seeds=["evilcorp.com"] reports its own seed out of scope. Fix at targets_models.py:110-111 and add a test pinning both shapes.

from bbot_server.applets.base import BaseApplet, api_endpoint, Field


class EmailsFields(CustomAssetFields):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking. This is a correct fix. The nested class AssetFields(BaseModel) was dead code, since the AST preloader in modules/__init__.py only collects module-level CustomAssetFields subclasses. Verified the effect: emails is absent from Asset.model_fields on dev, present here.

That is a change to the stored asset schema. It is not mentioned in the PR body and no test covers it. Call it out and add one.

model = Asset

@api_endpoint("/list", methods=["GET"], type="http_stream", response_model=Asset, summary="Stream all assets")
@api_endpoint("/list", methods=["GET"], type="http_stream", response_model=Asset, summary="Stream all assets", mcp=True)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking. uv run ruff format --diff exits 1 on this branch, 0 on dev. Adding mcp=True pushes this and 4 other files' decorators past the 119 char limit (activity, events, findings, technologies). CI runs this in the Lint step. uv run ruff format fixes it.

Comment thread pyproject.toml
"fastapi-mcp>=0.4.0",
"click==8.1.8",
"mcp==1.7.0",
"mcp==1.27.0",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking. uv lock --check fails on this branch and passes on dev. The merge resolved uv.lock backward:

  • bbot 3.0.0 (rev=dev) -> 0.0.0 (rev=3.0, d8f056e), while this file still declares rev = "dev"
  • blastdns dropped entirely
  • deepdiff 9.0.0 -> 8.6.2
  • bbot-server self-version 0.3.1 -> 0.2.0, contradicting line 3 of this file

Re-merge dev and regenerate with uv lock instead of hand-resolving the conflict.

@singlerider
singlerider changed the base branch from dev to stable August 24, 2026 17:40
@singlerider
singlerider changed the base branch from stable to dev August 24, 2026 22:29
@singlerider

Copy link
Copy Markdown
Collaborator

@R1ckyH I don't have access to push to your PR. Can you resolve the conflicts and address the above feedback?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants