Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ jobs:
env:
CARGO_TERM_COLOR: always
BUILD_PROFILE: debug
WHITAKER_INSTALLER_VERSION: '0.2.5'
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8
- name: Setup Rust
Expand All @@ -26,6 +27,24 @@ jobs:
**/*.md
!**/target/**
!**/dist/**
- name: Cache whitaker-installer
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cargo/bin/whitaker-installer
~/.cache/cargo-binstall
key: whitaker-installer-${{ runner.os }}-${{ runner.arch }}-${{ env.WHITAKER_INSTALLER_VERSION }}
- name: Install the Whitaker Dylint suite
run: |
if ! command -v whitaker-installer >/dev/null 2>&1; then
if cargo binstall --version >/dev/null 2>&1; then
cargo binstall --no-confirm --locked "whitaker-installer@${WHITAKER_INSTALLER_VERSION}"
else
echo "cargo-binstall unavailable; building whitaker-installer from crates.io"
cargo install --locked whitaker-installer --version "${WHITAKER_INSTALLER_VERSION}"
fi
fi
whitaker-installer
- name: Lint
run: make lint
- name: Test and Measure Coverage
Expand Down
1 change: 0 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,6 @@ project:

### Testing


- Use `rstest` fixtures for shared setup.
- Replace duplicated tests with `#[rstest(...)]` parameterized cases.
- Prefer `mockall` for ad hoc mocks/stubs.
Expand Down
90 changes: 90 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ description = "Fetch Rust crate documentation from docs.rs and convert to LLM-fr
async-openai = "0.26"
async_zip = { version = "0.0.17", features = ["tokio", "deflate", "bzip2"] }
bytes = "1.9"
cap-std = "3"
clap = { version = "4.5", features = ["derive"] }
eyre = "0.6"
futures-util = "0.3"
Expand Down
4 changes: 3 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
TARGET ?= rustxt

CARGO ?= cargo
WHITAKER ?= whitaker
BUILD_JOBS ?=
RUST_FLAGS ?= -D warnings
CARGO_FLAGS ?= --all-targets --all-features
Expand All @@ -26,9 +27,10 @@ test: ## Run tests with warnings treated as errors
target/%/$(TARGET): ## Build binary in debug or release mode
$(CARGO) build $(BUILD_JOBS) $(if $(findstring release,$(@)),--release) --bin $(TARGET)

lint: ## Run Clippy with warnings denied
lint: ## Run Clippy and the Whitaker Dylint suite with warnings denied
RUSTDOCFLAGS="$(RUSTDOC_FLAGS)" $(CARGO) doc --no-deps
$(CARGO) clippy $(CLIPPY_FLAGS)
RUSTFLAGS="$(RUST_FLAGS)" $(WHITAKER) --all -- $(CARGO_FLAGS)

fmt: ## Format Rust and Markdown sources
$(CARGO) fmt --all
Expand Down
18 changes: 12 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# rustxt

A command-line tool that fetches Rust crate documentation from [docs.rs](https://docs.rs) and converts it to clean, LLM-friendly Markdown. Optionally summarizes documentation using GPT-4.1 for quick comprehension.
A command-line tool that fetches Rust crate documentation from
[docs.rs](https://docs.rs) and converts it to clean, LLM-friendly Markdown.
Optionally summarizes documentation using GPT-4.1 for quick comprehension.

## What it does

Expand All @@ -12,7 +14,8 @@ Instead of dumping a list of links, rustxt gives you **comprehensive, readable d
- Parses key type details including method signatures
- Optionally summarizes everything with GPT-4.1 for quick understanding

Perfect for feeding crate documentation into LLMs, offline reading, or just getting a quick overview of an unfamiliar crate.
Perfect for feeding crate documentation into LLMs, offline reading, or just
getting a quick overview of an unfamiliar crate.

## Installation

Expand Down Expand Up @@ -84,7 +87,7 @@ shell completions, and automatic help generation.

## CLI Options

```
```text
Usage: rustxt [OPTIONS] <CRATE_NAME>

Arguments:
Expand All @@ -108,7 +111,8 @@ Options:
- Public types (structs, enums, traits) with their documentation
- Method signatures and descriptions for key types
4. **Convert**: Transforms HTML docblocks to clean Markdown
5. **Summarize** (optional): Sends the extracted documentation to GPT-4.1 for a concise, actionable summary
5. **Summarize** (optional): Sends the extracted documentation to GPT-4.1
for a concise, actionable summary
6. **Output**: Formats everything as Markdown to stdout

## Configuration
Expand All @@ -122,12 +126,14 @@ export OPENAI_API_KEY=sk-your-key-here
```

If the API key is not set, rustxt will:

- Work normally with `--no-summary`
- Print a warning and fall back to raw documentation without `--no-summary`

### Model

The tool uses `gpt-4.1` by default, which supports up to 1 million tokens of context - enough for even the largest crate documentation.
The tool uses `gpt-4.1` by default, which supports up to 1 million tokens
of context - enough for even the largest crate documentation.

## Use cases

Expand Down Expand Up @@ -170,7 +176,7 @@ cargo clippy

## Project structure

```
```text
src/
main.rs # CLI entry point and orchestration
error.rs # Error types (FetchError, ParseError, SummaryError)
Expand Down
25 changes: 5 additions & 20 deletions src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,19 +28,11 @@ pub enum FetchError {
}

/// Errors that can occur when parsing rustdoc HTML.
///
/// Parsing itself degrades gracefully on malformed HTML, so the only
/// failure mode is reading documentation files from disk.
#[derive(Debug, thiserror::Error)]
pub enum ParseError {
/// The HTML structure is invalid or unexpected.
#[error("Invalid HTML structure: {0}")]
InvalidHtml(String),

/// A required element was not found in the HTML.
#[error("Missing expected element: {selector}")]
MissingElement {
/// The CSS selector or element description that was not found.
selector: String,
},

/// I/O error when reading HTML files.
#[error("Failed to read file: {0}")]
Io(#[from] io::Error),
Expand All @@ -49,22 +41,15 @@ pub enum ParseError {
/// Errors that can occur during GPT-4.1 summarization.
#[derive(Debug, thiserror::Error)]
pub enum SummaryError {
/// OpenAI API returned an error.
/// `OpenAI` API returned an error.
#[error("OpenAI API error: {0}")]
ApiError(String),

/// Rate limited by OpenAI, should retry after specified duration.
#[error("Rate limited, retry after {retry_after_secs} seconds")]
RateLimited {
/// Number of seconds to wait before retrying.
retry_after_secs: u64,
},

/// Missing API key in environment.
#[error("OPENAI_API_KEY environment variable not set")]
MissingApiKey,

/// OpenAI client error.
/// `OpenAI` client error.
#[error("OpenAI client error: {0}")]
Client(#[from] async_openai::error::OpenAIError),
}
Expand Down
Loading
Loading