Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions bing-cli/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Build output
/bing-cli
*.exe
*.dll
*.so
*.dylib

# Test binaries
*.test
*.out

# Editors
.vscode/
.idea/
.DS_Store
97 changes: 97 additions & 0 deletions bing-cli/ARCHAEOLOGY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Site Archaeology — Bing Search

Findings from applying the archaeology protocol against Bing (English + Chinese UI) on 2026-07-15. Merged from bing-cli-acc and updated for v2.

## Feature A: `search <query>`

**URL:** `GET https://www.bing.com/search?q=<urlencode(query)>&count=<N>&offset=<O>`

- No auth required for basic search.
- Results are SSR'd in the DOM — no XHR needed for page one.
- Chinese market (`cc=cn&setlang=zh-Hans`) vs international (`cc=us&setlang=en`).
- `count` max is ~50 per page. `offset` enables pagination.

### DOM Structure

Each organic result lives in:

```html
<li class="b_algo" iid="SERP.NNNN">
<style>...</style>
<h2><a href="https://www.bing.com/ck/a?...">Title text</a></h2>
<div class="b_attribution"><cite>https://example.com</cite></div>
<div class="b_caption"><p class="b_paractl">Snippet text...</p></div>
</li>
```

**Selectors:**

| Field | Selector | Notes |
|---------|----------------|-------|
| title | `#b_results > li.b_algo h2 a` textContent | |
| url | `#b_results > li.b_algo .b_attribution cite` textContent | Display URL (domain only) |
| snippet | `#b_results > li.b_algo .b_caption p` textContent | May include breadcrumb prefix |
| source | same as `cite` | |
| bingHref| `h2 a` getAttribute("href") | Encrypted /ck/a redirect — resolved on Go side |

### Extract Strategy

Async IIFE with 8 s polling deadline (google-cli pattern):

1. Check for consent interstitial (`location.host.startsWith('consent.')`).
2. Poll `#b_results > li.b_algo` every 500 ms until results appear.
3. Deduplicate by display URL (Set).
4. Clean snippet: `replace(/\s+/g, " ")`, strip trailing "Read more", slice to 300.

### Gotchas

- **Encrypted href**: The `<h2 a>` `href` is a Bing redirect (`/ck/a?...&u=<encrypted>`). The CLI follows each redirect via HTTP GET and extracts the real URL from the response HTML body using a regex.
- **Inline `<style>` in results**: Each `li.b_algo` contains inline `<style>`. Extract by selector, never `innerHTML`.
- **Snippet prefix**: Chinese UI snippets sometimes start with `› ` (breadcrumb). Whitespace normalization handles it.
- **Result count**: `#b_results > li.b_algo` typically returns 10–12 per page.
- **Consent interstitial**: EU / anonymous browsers may hit `consent.bing.com`. Detected and reported as `consent_required`.

### Known Failure Modes

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| `consent_required` | EU consent interstitial | Accept once in Chrome, retry |
| `no_results` / empty items | Bing reflowed selectors | Re-run archaeology against `li.b_algo` |
| `daemon_unreachable` | kimi-webbridge not running | Start Kimi Desktop App |
| `extension_not_connected` | WebBridge extension not installed | Install from kimi.com/features/webbridge |
| CDP context errors | Tab still loading | Handled by evaluateWithRetry (3 retries) |

---

## Feature B: `result <url>`

**Delivery model:** Arbitrary page; DOM extraction via `evaluate`.

### Extraction

```json
{
"url": "location.href after redirects",
"title": "document.title",
"description": "meta[name=description] → meta[property='og:description'] → ''",
"text": "document.body.innerText capped at 5000 chars"
}
```

### Gotchas

- `document.body.innerText` begins with header/nav text — acceptable for MVP.
- JS-heavy pages: if `text.length < 50`, retry once with a 2 s settle delay.
- No network capture required.

---

## Daemon `evaluate` Protocol (Critical for Go Client)

- Code runs as a top-level expression — `return <expr>` triggers `SyntaxError`.
- The value of the last expression is the return value.
- For async, wrap explicitly: `(async () => { ... })()`.
- Daemon wraps every return as `{"type": "<type>", "value": <v>}`.
- When code ends with `JSON.stringify(...)`, `type` is `"string"`.
- Go client **must** unwrap this envelope before unmarshalling.
- `EvaluateJSON` handles the unwrap automatically.
46 changes: 46 additions & 0 deletions bing-cli/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# bing-cli Architecture

## Design Principles

- **CLI stays simple** — one action per command, no orchestration. Agent composes calls.
- **Raw JSON by default** — primary consumer is AI agents, not humans.
- **No API key** — uses kimi-webbridge for real browser sessions → cookies → no bot detection.

## Dedup Strategy

| Flag | Behavior |
|------|----------|
| *(none)* | Raw Bing top 10, no filtering |
| `-u` | Title dedup (>80% char overlap) + max 3 per domain |

Only `-u` triggers dedup. Default is pass-through — agent decides if results are too noisy and retries with `-u`.

## Market Strategy

Default is `-m us` (international). Bing's `cc` parameter controls the market. Only use `-m cn` when Chinese-local sources (Baidu/Zhihu/etc) are specifically needed.

## Platform Patterns

### General Search Engines (Bing, Google, Baidu, Sogou)
- Same strategy: default pass-through, `-u` for dedup
- Domain diversity matters — results from multiple sources

### Platform-Internal Search (Xiaohongshu, Douyin, WeChat)
- All results from same domain → `-u` meaningless
- Need different dedup (content-based, not domain-based)
- Separate CLI per platform

## Project Structure

```
bing-cli/
├── main.go # Entry point
├── browser/ # WebBridge HTTP client (reusable across all CLI tools)
│ └── client.go
├── output/ # JSON output helpers
│ └── output.go
├── bing/ # Bing-specific search logic
│ └── search.go
└── cmd/ # CLI commands (cobra)
└── root.go
```
103 changes: 103 additions & 0 deletions bing-cli/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# bing-cli

CLI wrapper around Bing Search backed by the [kimi-webbridge](https://www.kimi.com/features/webbridge) browser daemon. Runs inside your real Chrome session — no API key needed — and emits JSON on stdout.

## Commands

| Command | Usage | Returns |
|---|---|---|
| `search` | `bing-cli search <query> [-n N] [-m cn\|us] [-o N] [-r]` | `{query, count, results: [{title, url, snippet, source}]}` |
| `result` | `bing-cli result <url> [-r]` | `{url, title, description, text}` |

All output:

```json
{"ok": true, "data": ...}
```
or on failure (non-zero exit):

```json
{"ok": false, "error": {"code": "...", "message": "..."}}
```

### Agent consumption (recommended)

Use `--raw` (`-r`) to get clean JSON without the status wrapper — saves tokens:

```bash
bing-cli search "query" -r -n 10
# → [{"title":"...","url":"...","snippet":"...","source":"..."}]
```

## Prerequisites

**kimi-webbridge daemon** running on `127.0.0.1:10086` — install from https://www.kimi.com/features/webbridge.

## Install

**Binary** (Windows): copy `bing-cli.exe` into a directory on your `PATH`.

**Build from source** (requires Go 1.26+):

```bash
go build -o bing-cli .
./bing-cli search "claude code"
```

## Quick test

```bash
# English search
bing-cli search "claude code" --count 3

# Chinese search
bing-cli search "牛仔裤面料" --market cn -r -n 5

# International with pagination
bing-cli search "rust async patterns" -m us -r -n 10 -o 10

# Fetch page content
bing-cli result "https://example.com/article"
```

## Flags (search)

| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--market` | `-m` | `""` (auto) | `cn` (China), `us` (International) |
| `--count` | `-n` | `10` | Results per page (max 50) |
| `--offset` | `-o` | `0` | Pagination offset |
| `--raw` | `-r` | `false` | Bare JSON array — no wrapper |

## Layout

```
bing-cli/
├── main.go # entrypoint
├── browser/client.go # kimi-webbridge HTTP client (+ EvaluateJSON, Status)
├── output/output.go # JSON contract: {ok, data} / {ok, error}
├── bing/
│ ├── search.go # search command backend (DOM extractor + /ck/a resolver)
│ ├── result.go # result command backend (page content extraction)
│ └── common.go # evaluateWithRetry + isTransientContextError
├── cmd/
│ └── root.go # Cobra CLI definition (search + result)
├── ARCHAEOLOGY.md # DOM-selector field notes
└── README.md # this file
```

## Robustness

- **CDP retry**: 3-level backoff (300ms → 700ms → 1500ms) for transient context errors.
- **Consent detection**: Detects Bing privacy interstitials and reports a clear error.
- **Async DOM polling**: Waits up to 8 s for results to render (defense against lazy loading).
- **URL dedup**: Filters duplicate display URLs from the result set.
- **/ck/a redirect resolution**: Follows Bing's encrypted redirect pages to extract real target URLs.
- **JS-heavy page retry**: For `result`, retries with a 2 s settle delay if body text is < 50 chars.

## Notes

- Bing encrypts the real link target in the `href` of each result's `<h2 a>` (a proprietary redirect through `/ck/a`). The CLI automatically follows each redirect and extracts the real URL from the response HTML.
- Selectors pinned to `#b_results > li.b_algo` + `h2 a` + `.b_attribution cite` + `.b_caption p`. If Bing reflows, re-run the archaeology protocol and update `bing/search.go`.
- No login needed for basic search.
- `cn.bing.com` auto-redirect may happen based on IP when no `--market` flag.
41 changes: 41 additions & 0 deletions bing-cli/bing/common.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
package bing

import (
"strings"
"time"

"bing-cli/browser"
)

// evaluateWithRetry calls client.EvaluateJSON and retries on transient CDP
// context errors with a three-level backoff: 300ms → 700ms → 1500ms.
// Non-transient errors are returned immediately.
func evaluateWithRetry(client *browser.Client, code string, v any) error {
delays := []time.Duration{300 * time.Millisecond, 700 * time.Millisecond, 1500 * time.Millisecond}
var lastErr error
for i, d := range delays {
if i > 0 {
time.Sleep(d)
}
err := client.EvaluateJSON(code, v)
if err == nil {
return nil
}
lastErr = err
if !isTransientContextError(err) {
return err
}
}
return lastErr
}

// isTransientContextError returns true when the error message indicates a
// temporary CDP execution-context race (tab is still loading / has navigated).
func isTransientContextError(err error) bool {
if err == nil {
return false
}
msg := err.Error()
return strings.Contains(msg, "Cannot find default execution context") ||
strings.Contains(msg, "Execution context was destroyed")
}
73 changes: 73 additions & 0 deletions bing-cli/bing/result.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
package bing

import (
"fmt"

"bing-cli/browser"
)

// PageResult holds the extracted content from a fetched page.
type PageResult struct {
URL string `json:"url"`
Title string `json:"title"`
Description string `json:"description"`
Text string `json:"text"`
}

// resultExtractJS extracts page metadata and body text.
// For JS-heavy pages where text hasn't rendered yet, the caller should
// retry once after a 1.5 s wait if text length < 50.
const resultExtractJS = `(() => {
const meta = document.querySelector('meta[name="description"]');
const og = document.querySelector('meta[property="og:description"]');
return JSON.stringify({
url: location.href,
title: document.title,
description: meta ? meta.getAttribute('content') : (og ? og.getAttribute('content') : ''),
text: (document.body ? document.body.innerText : '').slice(0, 5000)
});
})()`

// FetchResult navigates to url and extracts title, description, and text.
// JS-heavy pages get a single retry after a 1.5 s settle delay.
func FetchResult(client *browser.Client, targetURL string) (*PageResult, error) {
if err := client.Navigate(targetURL); err != nil {
return nil, fmt.Errorf("navigate: %w", err)
}

var page PageResult
if err := evaluateWithRetry(client, resultExtractJS, &page); err != nil {
return nil, fmt.Errorf("extract: %w", err)
}

// JS-heavy pages: if body text is suspiciously short, wait and retry once.
if len(page.Text) < 50 {
// Use a raw Evaluate call so we can sleep inside JS; the daemon
// doesn't support time.Sleep in Go, so we re-evaluate after a
// short settle via a simple JS delay.
settleJS := `(async () => {
await new Promise(r => setTimeout(r, 2000));
return JSON.stringify({
url: location.href,
title: document.title,
description: (() => {
const m = document.querySelector('meta[name="description"]');
if (m) return m.getAttribute('content');
const og = document.querySelector('meta[property="og:description"]');
return og ? og.getAttribute('content') : '';
})(),
text: (document.body ? document.body.innerText : '').slice(0, 5000)
});
})()`
var retry PageResult
if err := client.EvaluateJSON(settleJS, &retry); err != nil {
// Fall back to the original (possibly short) result.
return &page, nil
}
if len(retry.Text) > len(page.Text) {
page = retry
}
}

return &page, nil
}
Loading