Add Firecrawl web monitoring example

examples/firecrawl_web_monitor/README.md

@@ -0,0 +1,117 @@
+# Firecrawl Web Monitoring with ZenML
+
+This example turns each Firecrawl `monitor.page` event into a ZenML pipeline run. Firecrawl monitors and scrapes the page; ZenML preserves the event, diff, analysis, and report as independently versioned artifacts. An LLM interprets the change when an OpenAI key is stored in a ZenML secret and passed via `--llm-secret`, and a configured ZenML alerter can optionally post meaningful changes to Slack.
+
+```text
+Firecrawl monitor.page event (pulled via API or delivered by webhook)
+       |
+       v
+raw event -> normalized diff -> LLM analysis -> report -> optional Slack
+    v1             v1               v1           v1
+    v2             v2               v2           v2
+```
+
+The stable artifact names make every run easy to compare in the ZenML dashboard:
+
+- `firecrawl_monitor_event` stores the source webhook.
+- `web_page_change` stores the normalized diff and Firecrawl scrape IDs.
+- `web_change_analysis` stores the business interpretation.
+- `web_monitoring_report` joins the evidence and analysis for downstream use.
+
+## Quick start
+
+This example assumes that a [ZenML stack](https://docs.zenml.io/stacks) is already configured. From this directory, install the example dependencies and run the bundled realistic event:
+
+```bash
+uv pip install -e .  # or: pip install -e .
+zenml init
+zenml stack set <your-stack-name>
+python run.py
+```
+
+Run everything from this directory: `zenml init` sets the ZenML source root here (and the pipeline's Docker settings reference this `pyproject.toml`, so remote image builds pick up the right dependencies). Set the stack *after* `zenml init` — init scopes the active stack to this directory and resets it to `default`. If your stack uses an S3 artifact store, also run `zenml integration install s3 --uv` once before the first run.
+
+No Firecrawl, OpenAI, or Slack credentials are required for that first run. Without an LLM secret, the analysis uses Firecrawl's meaningful-change judgment and clearly labels itself as a fallback. Store the OpenAI key in ZenML to run the same pipeline with an LLM on both local and remote orchestrators:
+
+```bash
+zenml secret create firecrawl-monitoring \
+  --OPENAI_API_KEY=<your-key> \
+  --OPENAI_MODEL=gpt-5-mini
+python run.py --llm-secret firecrawl-monitoring
+```
+
+Run the example twice, optionally editing `sample_payload.json`, then inspect the artifact versions in the dashboard or from the terminal:
+
+```bash
+python history.py
+```
+
+## Monitor a real page
+
+Create a Firecrawl `scrape` monitor for the page you care about — no webhook infrastructure needed:
+
+```bash
+export FIRECRAWL_API_KEY=<your-firecrawl-key>
+python create_firecrawl_monitor.py \
+  --target-url https://news.ycombinator.com/newest \
+  --goal "Alert when new AI or developer-tooling stories are posted" \
+  --schedule "every 5 minutes"
+```
+
+A fast-changing page with a short cadence produces meaningful diffs within minutes — ideal for a first demo; dial it back with `--schedule hourly` or `--schedule daily` for real monitoring. A live monitor keeps running and consuming Firecrawl credits on every scheduled check until you delete it, and `--llm-secret` runs spend OpenAI tokens per analyzed change — so remember to clean up after a demo (see below). Once at least one check has completed, pull it straight from the Firecrawl API and analyze it — one pipeline run per monitored page:
+
+```bash
+python run.py --monitor-id <monitor-id>
+```
+
+This fetches the latest completed check (or a specific one with `--check-id`) and feeds each page result through the same pipeline as the bundled sample, so the artifact history mixes local experiments and real checks seamlessly. Firecrawl emits both unified markdown diffs and structured JSON field diffs; the pipeline preserves both, so a monitor configured for JSON change tracking can compare fields such as prices or availability without parsing prose.
+
+When you are done, delete the monitor so it stops running checks and billing:
+
+```bash
+curl -X DELETE https://api.firecrawl.dev/v2/monitor/<monitor-id> \
+  -H "Authorization: Bearer $FIRECRAWL_API_KEY"
+```
+
+## Production: trigger a pipeline snapshot
+
+For an event-driven setup, ZenML supports this natively — no custom receiver required. Publish the pipeline as a [snapshot](https://docs.zenml.io/concepts/snapshots) on a remote stack with a remote orchestrator, a container registry, and a remote artifact store (for example a [Kubernetes orchestrator](https://docs.zenml.io/stacks/stack-components/orchestrators/kubernetes)); the pipeline's Docker settings are already sourced from this `pyproject.toml`, so the code and artifact contracts do not change:
+
+```bash
+zenml stack set <your-remote-stack>
+zenml pipeline snapshot create pipelines.monitoring.firecrawl_web_monitor_pipeline \
+  --name firecrawl-web-monitor \
+  --config configs/snapshot.yaml
+```
+
+`configs/snapshot.yaml` provides the default parameters baked into the snapshot (the bundled sample payload); triggers can override them per step. Note that server-triggered runs of static pipelines take step-level parameters, not pipeline-level ones. Trigger a run from Python:
+
+```python
+from zenml.client import Client
+
+Client().trigger_pipeline(
+    snapshot_name_or_id="firecrawl-web-monitor",
+    run_configuration={
+        "steps": {
+            "ingest_firecrawl_event": {
+                "parameters": {"payload": <monitor-page-event>}
+            }
+        }
+    },
+)
+```
+
+Snapshots can also be [triggered from external systems](https://docs.zenml.io/user-guides/tutorial/trigger-pipelines-from-external-systems) with a single authenticated REST call, which is where Firecrawl's webhook plugs in.
+
+Tip for Apple Silicon: images build for `linux/amd64` under emulation, where `uv` can crash. If the build fails with exit code 139, override the installer with `settings: {docker: {python_package_installer: pip}}` in the snapshot config.
+
+## Optional Slack alerts
+
+ZenML's [Slack alerter](https://docs.zenml.io/stacks/stack-components/alerters/slack) is a stack component registered once with a Slack bot token and channel; any pipeline on the stack can then post to Slack without handling Slack credentials in code. Attach one to the active stack, then enable notifications for CLI runs:
+
+```bash
+python run.py --notify-slack
+```
+
+Only reports marked meaningful are sent. If notification is enabled but the active stack has no alerter, the step logs a warning and the run still succeeds.
+

examples/firecrawl_web_monitor/analysis.py

@@ -0,0 +1,104 @@
+"""Analysis helpers kept separate from ZenML orchestration for testability."""
+
+import json
+from typing import Any, Dict, Optional
+
+from models import ChangeAnalysis, PageChange
+
+DEFAULT_MODEL = "gpt-5-mini"
+
+
+def analyze_change(
+    change: PageChange,
+    goal: str,
+    api_key: Optional[str] = None,
+    model: str = DEFAULT_MODEL,
+) -> ChangeAnalysis:
+    """Analyze a change with OpenAI or a deterministic local fallback.
+
+    Args:
+        change: Normalized page change.
+        goal: Business goal that defines which changes matter.
+        api_key: Optional OpenAI API key loaded from a ZenML secret.
+        model: OpenAI model name.
+
+    Returns:
+        A structured analysis suitable for alerting and later comparison.
+    """
+    if api_key:
+        return _analyze_with_openai(
+            change=change, goal=goal, api_key=api_key, model=model
+        )
+    return _analyze_without_llm(change=change)
+
+
+def _analyze_with_openai(
+    change: PageChange, goal: str, api_key: str, model: str
+) -> ChangeAnalysis:
+    """Ask OpenAI for a structured interpretation of the diff."""
+    from openai import OpenAI
+
+    prompt = (
+        "Analyze this website change against the monitoring goal. "
+        "Return JSON with summary, impact, recommended_actions (a list), "
+        "meaningful (boolean), and confidence (low, medium, or high).\n\n"
+        f"Goal: {goal}\nURL: {change.url}\nStatus: {change.status}\n"
+        f"Unified diff:\n{change.unified_diff[:12000]}\n"
+        f"Structured diff:\n{json.dumps(change.structured_diff)[:6000]}"
+    )
+    response = OpenAI(api_key=api_key).chat.completions.create(
+        model=model,
+        messages=[
+            {
+                "role": "system",
+                "content": (
+                    "You are a precise web-change analyst. Base conclusions "
+                    "only on the supplied evidence and return valid JSON."
+                ),
+            },
+            {"role": "user", "content": prompt},
+        ],
+        response_format={"type": "json_object"},
+    )
+    content = response.choices[0].message.content
+    if not content:
+        raise RuntimeError("The LLM returned an empty analysis.")
+    result: Dict[str, Any] = json.loads(content)
+    result.pop("analyzer", None)
+    return ChangeAnalysis(
+        **result,
+        analyzer=model,
+    )
+
+
+def _analyze_without_llm(change: PageChange) -> ChangeAnalysis:
+    """Produce a useful offline result from Firecrawl's own evidence."""
+    judgment = change.firecrawl_judgment
+    if judgment:
+        actions = [item.reason for item in judgment.meaningful_changes]
+        return ChangeAnalysis(
+            summary=judgment.reason,
+            impact=(
+                "The change matches the configured monitoring goal."
+                if judgment.meaningful
+                else "The change appears to be noise for the monitoring goal."
+            ),
+            recommended_actions=actions or ["Review the recorded diff."],
+            meaningful=judgment.meaningful,
+            confidence=judgment.confidence,
+            analyzer="firecrawl-judgment-fallback",
+        )
+
+    changed_lines = [
+        line
+        for line in change.unified_diff.splitlines()
+        if line.startswith(("+", "-")) and not line.startswith(("+++", "---"))
+    ]
+    return ChangeAnalysis(
+        summary=f"Firecrawl reported {change.status} with {len(changed_lines)} changed lines.",
+        impact="An LLM was not configured, so business impact was not inferred.",
+        recommended_actions=["Review the recorded diff."],
+        meaningful=change.status in {"changed", "new", "removed"},
+        confidence="low",
+        analyzer="deterministic-fallback",
+    )

examples/firecrawl_web_monitor/configs/dev.yaml

@@ -0,0 +1,3 @@
+enable_cache: false
+tags:
+  - local

examples/firecrawl_web_monitor/configs/snapshot.yaml

@@ -0,0 +1,34 @@
+parameters:
+  payload:
+    success: true
+    type: monitor.page
+    id: 019df960-5f2a-75fb-a98b-bd2d32ca67d4
+    webhookId: f1e2d3c4-0000-0000-0000-000000000000
+    data:
+    - monitorId: 019df960-06e7-7383-9d89-82c0113dc31a
+      checkId: 019df960-5f2a-75fb-a98b-bd2d32ca67d4
+      url: https://example.com/pricing
+      status: changed
+      previousScrapeId: 019df94f-82c3-7e41-81f0-00c72b2d9c52
+      currentScrapeId: 019df960-73ee-7ac2-97a9-fb0e442c21f1
+      error: null
+      isMeaningful: true
+      judgment:
+        meaningful: true
+        confidence: high
+        reason: The Starter plan price increased from $19 to $24 per month.
+        meaningfulChanges:
+        - type: changed
+          before: "Starter \u2014 $19/mo"
+          after: "Starter \u2014 $24/mo"
+          reason: A competitor price change may require a positioning review.
+      diff:
+        text: "--- previous\n+++ current\n@@ -1,3 +1,3 @@\n # Pricing\n-Starter \u2014\
+          \ $19/mo\n+Starter \u2014 $24/mo\n"
+    metadata:
+      environment: demo
+  goal: Identify product, pricing, positioning, or release changes that require
+    a response from our product or go-to-market teams.
+  notify_slack: false
+  llm_secret_name: null
+enable_cache: false

examples/firecrawl_web_monitor/constants.py

@@ -0,0 +1,6 @@
+"""Shared defaults for the Firecrawl monitoring example."""
+
+DEFAULT_GOAL = (
+    "Identify product, pricing, positioning, or release changes that require "
+    "a response from our product or go-to-market teams."
+)

examples/firecrawl_web_monitor/create_firecrawl_monitor.py

@@ -0,0 +1,102 @@
+"""Create a Firecrawl page monitor whose checks the pipeline can pull."""
+
+import argparse
+import os
+from typing import Any, Dict, Optional
+
+import requests
+from constants import DEFAULT_GOAL
+
+
+def build_monitor_request(
+    target_url: str,
+    goal: str,
+    schedule: str,
+    webhook_url: Optional[str] = None,
+    webhook_token: Optional[str] = None,
+) -> Dict[str, Any]:
+    """Build a Firecrawl v2 monitor request.
+
+    Args:
+        target_url: Page that Firecrawl should monitor and scrape.
+        goal: Meaningful-change goal.
+        schedule: Firecrawl natural-language schedule.
+        webhook_url: Optional delivery endpoint. Without it, checks are
+            pulled via the API (``run.py --monitor-id``).
+        webhook_token: Optional bearer token for the webhook endpoint.
+
+    Returns:
+        Firecrawl monitor request body.
+    """
+    request: Dict[str, Any] = {
+        "name": f"ZenML monitor: {target_url}",
+        "schedule": {"text": schedule, "timezone": "UTC"},
+        "goal": goal,
+        "targets": [
+            {
+                "type": "scrape",
+                "urls": [target_url],
+                "scrapeOptions": {
+                    "formats": ["markdown"],
+                    "onlyMainContent": True,
+                },
+            }
+        ],
+    }
+    if webhook_url:
+        headers = (
+            {"Authorization": f"Bearer {webhook_token}"}
+            if webhook_token
+            else {}
+        )
+        request["webhook"] = {
+            "url": webhook_url,
+            "headers": headers,
+            "metadata": {"source": "zenml-firecrawl-example"},
+            "events": ["monitor.page"],
+        }
+    return request
+
+
+def main() -> None:
+    """Create a monitor using credentials from the environment."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--target-url", required=True)
+    parser.add_argument("--goal", default=DEFAULT_GOAL)
+    parser.add_argument("--schedule", default="hourly")
+    parser.add_argument(
+        "--webhook-url",
+        default=None,
+        help="Optional endpoint Firecrawl should POST monitor.page events "
+        "to, e.g. a relay that triggers a ZenML pipeline snapshot. Omit it "
+        "to pull checks with `run.py --monitor-id` instead.",
+    )
+    args = parser.parse_args()
+
+    api_key = os.getenv("FIRECRAWL_API_KEY")
+    if not api_key:
+        raise RuntimeError("Set FIRECRAWL_API_KEY before creating a monitor.")
+
+    response = requests.post(
+        "https://api.firecrawl.dev/v2/monitor",
+        headers={
+            "Authorization": f"Bearer {api_key}",
+            "Content-Type": "application/json",
+        },
+        json=build_monitor_request(
+            target_url=args.target_url,
+            goal=args.goal,
+            schedule=args.schedule,
+            webhook_url=args.webhook_url,
+            webhook_token=os.getenv("FIRECRAWL_WEBHOOK_TOKEN"),
+        ),
+        timeout=30,
+    )
+    response.raise_for_status()
+    body = response.json()
+    monitor_id = body.get("id") or body.get("data", {}).get("id", "<unknown>")
+    print(f"Created Firecrawl monitor: {monitor_id}")
+
+
+if __name__ == "__main__":
+    main()