feat: probe-cli — Site Archaeology 自动化工具

probe-cli/README.md

@@ -0,0 +1,155 @@
+# probe-cli
+
+Automates the [Site Archaeology protocol](https://github.com/better-world-ai/agent-cli-creator) from the x-cli ecosystem.
+
+One command replaces 30-60 minutes of manual `curl` calls with 30 seconds of auto-detection.
+
+## What it does
+
+Given any URL, probe-cli connects to the kimi-webbridge daemon and automatically:
+
+1. **Detects auth mechanism** — scans localStorage, sessionStorage, and cookies for token/session/csrf patterns
+2. **Extracts DOM elements** — finds all forms, inputs, buttons with their selectors
+3. **Captures network traffic** — filters XHR/Fetch requests, inspects auth headers, identifies API endpoints
+4. **Recommends a CLI pattern** — scores dom-scrape / api-reverse / form-submit / async-poll based on findings
+5. **Outputs a JSON site profile** — ready for the agent-cli-creator Phase 4 (implementation)
+
+## Protocol Mapping
+
+| Site Archaeology Step | probe-cli automation |
+|-----------------------|---------------------|
+| Step 1: Navigate | ✅ Auto-navigate to URL |
+| Step 2: Snapshot DOM | ✅ Auto-extract forms, inputs, buttons |
+| Step 3: Start network capture | ✅ Auto-start before navigate |
+| Step 4: Stop + list requests | ✅ Auto-stop after page settles |
+| Step 5: Inspect request detail | ✅ Auto-inspect auth headers |
+| Step 6: Verify with evaluate | ✅ Auth pattern detection + recommendation |
+
+## Install
+
+```bash
+# Build from source
+git clone https://github.com/RachelXiaolan/probe-cli.git
+cd probe-cli
+go build -o probe-cli .
+```
+
+## Usage
+
+```bash
+# Full probe (auth + DOM + network)
+probe-cli https://example.com
+
+# Quick mode (auth + DOM only, skip network capture)
+probe-cli https://example.com --quick
+
+# Custom session name
+probe-cli https://example.com --session mysite
+```
+
+**Prerequisites:** kimi-webbridge daemon running at `http://127.0.0.1:10086`
+
+## Output Format
+
+```json
+{
+  "ok": true,
+  "data": {
+    "url": "https://fish.audio/zh-CN/app/text-to-speech/",
+    "timestamp": "2026-06-01T12:00:00+09:00",
+    "auth": {
+      "detected": true,
+      "method": "bearer-localstorage",
+      "storage_type": "localStorage",
+      "token_keys": ["token"]
+    },
+    "dom": {
+      "forms": [],
+      "standalone_inputs": [
+        {"tag": "div", "content_editable": true, "role": "textbox"}
+      ],
+      "buttons": [
+        {"text": "生成语音", "type": "button"}
+      ]
+    },
+    "network": {
+      "total_requests": 42,
+      "api_endpoints": [
+        {
+          "method": "POST",
+          "url": "https://api.fish.audio/task",
+          "has_auth_header": true,
+          "auth_headers": ["Authorization: Bearer eyJhbG***"]
+        }
+      ],
+      "static_filtered": 37
+    },
+    "pattern": {
+      "type": "api-reverse",
+      "confidence": 0.85,
+      "reason": "Data loaded via XHR/Fetch API calls with bearer-localstorage auth. Replicate API calls in evaluate()."
+    }
+  }
+}
+```
+
+## How it works
+
+```
+┌─────────────────────────────────────────┐
+│ 1. Start network capture                │
+│ 2. Navigate to URL                      │
+│ 3. Wait 3s for page to settle           │
+├─────────────────────────────────────────┤
+│ 4. Inject auth detection JS             │
+│    → localStorage / cookie / sessionStorage scan │
+│ 5. Inject DOM extraction JS             │
+│    → forms, inputs, buttons             │
+├─────────────────────────────────────────┤
+│ 6. Stop network capture                 │
+│ 7. List + filter XHR/Fetch requests     │
+│ 8. Inspect auth headers per endpoint    │
+├─────────────────────────────────────────┤
+│ 9. Score patterns + recommend           │
+│ 10. Output site profile JSON            │
+└─────────────────────────────────────────┘
+```
+
+## Integration with agent-cli-creator
+
+After probe-cli outputs a site profile, the AI agent can skip directly to Phase 4 (Implementation) of the agent-cli-creator workflow:
+
+```
+Before: Agent reads site-exploration.md → manually runs 6 curl steps → 30-60 min
+After:  Agent runs probe-cli → gets site-profile.json → skip to Phase 4
+```
+
+The site profile contains everything the agent needs:
+- `auth.method` → tells the agent how to authenticate
+- `network.api_endpoints` → tells the agent which APIs to call
+- `dom.forms/inputs/buttons` → tells the agent which elements to interact with
+- `pattern.type` → tells the agent which template to use
+
+## Auth Detection Patterns
+
+| Detected Method | What it means | How CLI should handle |
+|----------------|---------------|----------------------|
+| `bearer-localstorage` | JWT/Bearer token in localStorage | Extract token via evaluate, add to fetch headers |
+| `csrf-cookie` | CSRF token in cookie (e.g. ct0) | Extract from cookie, add x-csrf-token header |
+| `cookie-only` | Session cookie, no extra headers | Just use evaluate (cookies sent automatically) |
+| `session-storage` | Auth token in sessionStorage | Extract via evaluate, add to headers |
+| `localstorage-other` | Non-token auth data in localStorage | Inspect keys for API key or session ID |
+| *(none detected)* | No login required, or complex auth | Manual archaeology needed |
+
+## CLI Pattern Types
+
+| Pattern | When | Implementation approach |
+|---------|------|------------------------|
+| `dom-scrape` | Data visible in DOM, no API | Snapshot + evaluate to extract text |
+| `api-reverse` | Data from XHR/Fetch with auth | Replicate fetch calls in evaluate |
+| `form-submit` | Forms to fill and submit | Fill fields + click submit via evaluate |
+| `async-poll` | POST action → poll GET for result | Submit + loop poll until done |
+
+## License
+
+MIT

probe-cli/browser/client.go

@@ -0,0 +1,102 @@
+package browser
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"time"
+)
+
+const DefaultDaemonURL = "http://127.0.0.1:10086"
+
+type Client struct {
+	baseURL string
+	session string
+	http    *http.Client
+}
+
+func NewClient(session string) *Client {
+	return &Client{
+		baseURL: DefaultDaemonURL,
+		session: session,
+		http:    &http.Client{Timeout: 90 * time.Second},
+	}
+}
+
+func (c *Client) Call(action string, args map[string]any) (json.RawMessage, error) {
+	body, _ := json.Marshal(map[string]any{
+		"action":  action,
+		"session": c.session,
+		"args":    args,
+	})
+	resp, err := c.http.Post(c.baseURL+"/command", "application/json", bytes.NewReader(body))
+	if err != nil {
+		return nil, fmt.Errorf("daemon unreachable: %w", err)
+	}
+	defer resp.Body.Close()
+
+	var result struct {
+		OK    bool            `json:"ok"`
+		Data  json.RawMessage `json:"data"`
+		Error *struct {
+			Code    string `json:"code"`
+			Message string `json:"message"`
+		} `json:"error"`
+	}
+	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
+		return nil, err
+	}
+	if !result.OK {
+		if result.Error != nil {
+			return nil, fmt.Errorf("%s: %s", result.Error.Code, result.Error.Message)
+		}
+		return nil, fmt.Errorf("unknown error from daemon")
+	}
+	return result.Data, nil
+}
+
+// Quick helpers
+
+func (c *Client) Navigate(url string) error {
+	_, err := c.Call("navigate", map[string]any{"url": url, "newTab": true})
+	return err
+}
+
+func (c *Client) Evaluate(code string) (json.RawMessage, error) {
+	return c.Call("evaluate", map[string]any{"code": code})
+}
+
+func (c *Client) Snapshot() (json.RawMessage, error) {
+	return c.Call("snapshot", map[string]any{})
+}
+
+func (c *Client) NetworkStart() error {
+	_, err := c.Call("network", map[string]any{"cmd": "start"})
+	return err
+}
+
+func (c *Client) NetworkStop() error {
+	_, err := c.Call("network", map[string]any{"cmd": "stop"})
+	return err
+}
+
+func (c *Client) NetworkList() (json.RawMessage, error) {
+	return c.Call("network", map[string]any{"cmd": "list"})
+}
+
+func (c *Client) NetworkDetail(requestID string) (json.RawMessage, error) {
+	return c.Call("network", map[string]any{"cmd": "detail", "requestId": requestID})
+}
+
+// Status checks if the kimi-webbridge daemon is running.
+func (c *Client) Status() (json.RawMessage, error) {
+	resp, err := c.http.Get(c.baseURL + "/status")
+	if err != nil {
+		return nil, fmt.Errorf("daemon unreachable at %s: %w", c.baseURL, err)
+	}
+	defer resp.Body.Close()
+	var result json.RawMessage
+	json.NewDecoder(resp.Body).Decode(&result)
+	return result, nil
+}

probe-cli/cmd/root.go

@@ -0,0 +1,78 @@
+package cmd
+
+import (
+	"fmt"
+	"os"
+
+	"github.com/spf13/cobra"
+
+	"probe-cli/browser"
+	"probe-cli/output"
+	"probe-cli/probe"
+)
+
+var (
+	sessionName string
+	quickMode   bool
+)
+
+var rootCmd = &cobra.Command{
+	Use:   "probe-cli <url>",
+	Short: "Auto-detect site auth, API endpoints, and DOM elements for x-cli development",
+	Long: `probe-cli automates the Site Archaeology protocol from agent-cli-creator.
+
+It connects to the kimi-webbridge daemon, navigates to the target URL,
+and auto-detects:
+  - Authentication mechanism (localStorage, cookie, sessionStorage, CSRF)
+  - API endpoints (XHR/Fetch with auth header analysis)
+  - DOM elements (forms, inputs, buttons)
+  - Recommended CLI pattern (dom-scrape, api-reverse, form-submit, async-poll)
+
+Output is a JSON site profile ready for CLI implementation.`,
+	Args: cobra.ExactArgs(1),
+	Run: func(cmd *cobra.Command, args []string) {
+		targetURL := args[0]
+
+		// Use URL hostname as session name if not specified
+		sess := sessionName
+		if sess == "" {
+			sess = "probe"
+		}
+
+		client := browser.NewClient(sess)
+
+		// Check daemon is running
+		if _, err := client.Status(); err != nil {
+			output.Error("daemon_unreachable", fmt.Sprintf(
+				"kimi-webbridge daemon not running at %s. Start it first.",
+				browser.DefaultDaemonURL,
+			))
+			os.Exit(1)
+		}
+
+		var profile *probe.SiteProfile
+		var err error
+
+		if quickMode {
+			profile, err = probe.RunQuick(client, targetURL)
+		} else {
+			profile, err = probe.Run(client, targetURL)
+		}
+
+		if err != nil {
+			output.Error("probe_failed", err.Error())
+			os.Exit(1)
+		}
+
+		output.Success(profile)
+	},
+}
+
+func Execute() error {
+	return rootCmd.Execute()
+}
+
+func init() {
+	rootCmd.Flags().StringVarP(&sessionName, "session", "s", "", "kimi-webbridge session name (default: probe)")
+	rootCmd.Flags().BoolVarP(&quickMode, "quick", "q", false, "skip network capture (DOM + auth only)")
+}

probe-cli/go.mod

@@ -0,0 +1,9 @@
+module probe-cli
+
+go 1.21
+
+require github.com/spf13/cobra v1.8.0
+
+require github.com/inconshreveable/mousetrap v1.1.0 // indirect
+
+require github.com/spf13/pflag v1.0.5 // indirect

probe-cli/go.sum

@@ -0,0 +1,10 @@
+github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
+github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
+github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
+github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0=
+github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho=
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
+github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

probe-cli/main.go

@@ -0,0 +1,13 @@
+package main
+
+import (
+	"os"
+
+	"probe-cli/cmd"
+)
+
+func main() {
+	if err := cmd.Execute(); err != nil {
+		os.Exit(1)
+	}
+}

probe-cli/output/output.go

@@ -0,0 +1,24 @@
+package output
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+)
+
+func Success(data any) {
+	printJSON(map[string]any{"ok": true, "data": data})
+}
+
+func Error(code, message string) {
+	printJSON(map[string]any{"ok": false, "error": map[string]any{"code": code, "message": message}})
+}
+
+func printJSON(v any) {
+	enc := json.NewEncoder(os.Stdout)
+	enc.SetEscapeHTML(false)
+	enc.SetIndent("", "  ")
+	if err := enc.Encode(v); err != nil {
+		fmt.Fprintf(os.Stderr, "output error: %v\n", err)
+	}
+}