diff --git a/docs/rfcs/0001-notte-project-scaffolding-and-deploy.md b/docs/rfcs/0001-notte-project-scaffolding-and-deploy.md new file mode 100644 index 0000000..6867e5f --- /dev/null +++ b/docs/rfcs/0001-notte-project-scaffolding-and-deploy.md @@ -0,0 +1,717 @@ +# RFC 0001 — A `notte` project: scaffolding, bundling, and declarative deploys + +| | | +|---|---| +| **Status** | Draft — for discussion, nothing implemented | +| **Author** | Lucas Giordano | +| **Date** | 2026-08-27 | +| **v1 scope** | functions, secrets, schedules. Managed-auth connections later. | + +--- + +## Context + +We have built the same framework twice, by hand: + +| | `anything-api/marketplace` | `monorepo/apps/back/managed-auth` | +|---|---|---| +| What | 2,049 `.py` functions across 670 domains | 9 connectors = 18 functions (login + verifier) | +| Config | one 2.2 MB `marketplace/manifest.json` | one `connectors/.json` per connector | +| Deploy | `scripts/marketplace-catalog.ts` (2,404 lines TS) | `scripts/deploy.py` (996 lines, stdlib-only) | +| Driver | `make marketplace push prod` | `make deploy google.com staging` | +| Env selection | `$(filter …,$(MAKECMDGOALS))` + failing no-op rules | `%::` catch-all so URLs with `:` survive | +| Create vs update | manifest lockfile: `envs[env].code_sha256 != local` | server decides from `revision` + per-file sha256 | +| Shared code | **none — 0 relative imports in 2,049 files**; `has_value`/`clean_*` reimplemented hundreds of times | `contract.py` spliced in by one regex; `_verification_code()` hand-copied across 4 connectors | +| CI | designed for, never wired up | full 6-job promotion pipeline | + +Both converged on the same shape. Both spent their worst code on the same two problems: **faking subcommands in `make`**, and **not having a bundler**. Meanwhile the CLI already owns auth, config, output formatting, and every `functions` API call these scripts shell out to. + +Goal: fold the framework into `notte`, so a functions repo is `notte stack init` + `notte stack deploy`, and so `from ._shared.http import get` actually works. + +--- + +## Scope + +Three kinds of thing, and only the first two belong in git: + +| | Owner | In v1 | +|---|---|---| +| **Code** — function sources, shared modules | git, bundled by the CLI | ✅ | +| **Config** — function name/description/shared, cron, *which* secrets are required | `notte.toml` | ✅ | +| **Secret values** | gitignored `.env.`, pushed explicitly | ✅ (`notte secrets push/diff`) | +| **Runtime/user data** — managed-auth connections, sessions, personas, vaults, runs | the API; never reconciled from a file | ❌ (operational commands only) | + +Managed-auth **templates** are already declarative and could join later (see below). Managed-auth **connections** should probably never be — creating one runs a real browser login, spends money, and provisions a vault + a profile as side effects. + +--- + +## The hard constraint nobody can design around + +There are **two** gates, with different scopes, and conflating them is easy — an earlier draft of this document did exactly that. Getting the distinction right matters, because it changes which arguments are load-bearing: + +| | Upload | Runtime | +|---|---|---| +| Where | `POST /functions`, `?restricted=` **defaults to `True`** (`notte_api/functions/endpoints.py:353,411`) | `workflows-lambda/worker.py:891` calls `run_script(..., restricted=False)` | +| RestrictedPython AST policy | **enforced** — `FORBIDDEN_NODES`, `FORBIDDEN_CALLS`, the lot | **not applied at all** — `worker.py:608` takes the `restricted=False` branch and uses plain `compile()` | +| Import allowlist | enforced statically over the source | **still enforced**, dynamically: `__import__` is `runner.safe_import`, which calls `check_valid_import(name)` on every import (`worker.py:572-578`) | +| Which list | `notte_api.ast.ALLOWED_IMPORTS` | `_LAMBDA_ALLOWED_IMPORTS` + `{httpx, httpcloak}`, minus `tempfile` (`worker.py:520`) | + +So the AST policy is an upload-time quality gate, while **the import allowlist is a genuine runtime guard** — enforced by name, on every import, by the last guard standing once `restricted=False`. + +**1. Local imports are structurally impossible server-side.** +`notte-api/src/notte_api/ast.py`, `visit_ImportFrom`: +```python +if node.module is None: + raise SyntaxError("Relative imports are not allowed") +``` +`from . import util` dies there. `from .util import x` survives that check (module is `"util"`) but then hits `check_valid_import("util")` → not in `ALLOWED_IMPORTS` → rejected. Plain `import util` likewise. **There is no server-side path to multi-file. Bundling must be client-side.** + +**2. Off-the-shelf bundlers don't help — and `os`/`sys` isn't the reason.** + +An earlier draft said stickytape and pinliner were impossible because RestrictedPython forbids `sys`, `exec`, `compile`, `__import__` and `os`. **That was the wrong mechanism** — the AST policy isn't applied at execution time, so pointing at `FORBIDDEN_CALLS` proves nothing about what a deployed function can do. The real blockers are further down, and they bite in this order: + +1. **`import tempfile` fails first.** `worker.py:520` does `ALLOWED_IMPORTS.discard("tempfile")`, so stickytape's `mkdtemp()` dies before a single module is written. +2. **`import util` — the entire point — is rejected by name at run time.** stickytape exists to make a written-to-disk module importable; that import goes through `safe_import("util")` → `check_valid_import("util")` → not in the allowlist. The one thing it does is the one thing that's gated. +3. `os`, `sys` and `shutil` are on the **runtime** denylist too (`_LAMBDA_DISALLOWED_STDLIB_IMPORTS`), so none of this is an upload-time gate a query parameter can switch off. + +**On the obvious counter-proposal — "just allow `os` and `sys`":** it is neither necessary nor sufficient. It doesn't touch (1) or (2). Making stickytape work means disabling `safe_import`, i.e. permitting arbitrary imports by name at run time. That is a materially bigger decision than allowing two modules, and unlike the original framing it carries a real security dimension, because `safe_import` is the only import guard left once `restricted=False`. It deserves to be decided on its own merits, not adopted as a side effect of a bundling convenience. + +**Three reasons to flatten that don't depend on any allowlist.** These are the actual argument, and the earlier draft buried them beneath a claim that turned out to be wrong: + +- **The artifact stops being readable.** stickytape emits a prelude plus every module as an escaped bytes literal passed to `__stickytape_write_module`. The deployed file is what the console renders, what `functions show` downloads, and what marketplace's `push`/`check` diff against. A blob kills the diff model, and makes tracebacks worse rather than better. +- **stickytape disclaims itself.** Its README: *"bodged together… for a specific use case"*, no `from __future__` imports, `__file__` unreliable, dynamic imports need manual flags. That is an unmaintained third-party dependency sitting in the deploy path. +- **It's Python.** `notte stack` requires an interpreter anyway, so this is no longer a dependency argument — it is a separation one. Flattening stays a pure text transformation with no interpreter in it, which is what lets it be tested offline against thousands of files and reasoned about without a runtime. Python does the part that genuinely needs Python: validating the result. + +So: the bundler must be a **static flattener** emitting plain, ordinary Python — one module namespace, no runtime machinery. **That recommendation stands even in a fully permissive runtime**, which is how it should have been argued in the first place. + +**3. Dependencies are a fixed allowlist, so there is no dependency resolution to build.** +`ALLOWED_IMPORTS = set(sys.stdlib_module_names) - DISALLOWED_STDLIB_IMPORTS | {notte, notte_sdk, notte_core, notte_browser, notte_agent, notte_llm, pydantic, loguru, requests, httpx, httpcloak, playwright, gspread, google, litellm, bs4, pipedream, tqdm, typing_extensions}`. + +No `pip install`, no `requirements.txt`, no PEP-723 block, and no dependency resolver to write. The set is closed, so `notte stack sync` installs the intersection of that list with what your functions import — which makes the venv itself the check, since anything outside it fails to resolve. + +Two more upload-time contracts: +- `extract_env_requirements(source)` (`notte_api/functions/requirements.py`) AST-scans for literal `os.environ[...]`/`os.getenv(...)`/`.get`/`.setdefault`/`.pop` plus aliases, unions with an optional module-level `NOTTE_REQUIRED_SECRETS = [...]` list, subtracts reserved names, and persists to `functions.required_secrets`. Env vars are read via `from notte_sdk.types import os` (bare `import os` is blocked). **This is a ready-made input for the secrets planner.** +- `response_format` (a JSON Schema) is accepted only if `check_run_returns_pydantic_model` finds `def run(...) -> Model` with `class Model(BaseModel)` in the same file. + +--- + +## What the CLI has today + +- Cobra, all commands in `internal/cmd` as package globals (`internal/cmd/root.go`). +- `internal/config/config.go`: one global `~/.notte/cli/config.json` = `{api_key, api_url}`, plus bare-string state files `current_session`, `current_function`. **No per-directory config anywhere in the repo** — `notte stack init` would introduce the first. +- `internal/auth/env.go` **already has an environment notion**: `KeyringKeyForEnv(label)` namespaces keyring entries, and `hostToEnvLabel` maps `api.notte.cc`/`us-prod`→prod, `us-staging`→staging, `us-dev*`→dev. That's the hook `--env` hangs off. +- `internal/cmd/functions.go` covers list/create/show/update/delete/fork/run/runs/schedule/secrets. Create/update send a single multipart `file` part. No zip/tar/directory support anywhere in the repo. +- **Unexposed wins already in the generated client:** `response_format` and `restricted` (create/update), `version` (update — server-assigned `v%Y%m%d_%H%M%S`), `decryption_key` (download). The marketplace script had to hand-roll `fetch` + re-derive `sha256("api_key:{k}:workflow_id:{id}:dumb")[:64]` because `--decryption-key` doesn't exist. +- No `//go:embed` in the repo. Scaffolding would be the first. + +Two things to verify rather than assume: +- `marketplace-catalog.ts` carries a `detectCliError()` workaround because *"the CLI reports API errors as a JSON body on stdout with exit 0"*. `internal/output/json.go` looks like it writes errors to stderr and `Execute()` exits 1, so this may be path-specific — reproduce before designing around it. +- The README documents `notte functions schedule --cron "0 9 * * *"` — five fields. `validate_and_format_cron` wants the six-field AWS EventBridge form (`cron(m h dom mon dow year)`). Either the README is wrong or the server is lenient; worth checking, and worth validating client-side either way. + +--- + +## Prior art, and what each one gets right + +| | Project marker | Config | Per-unit dir | Shared code | Envs | +|---|---|---|---|---|---| +| **Supabase** | `supabase/` + `config.toml` | TOML, `[functions.]` | `functions//index.ts` | `_shared/` (underscore = not a function) | `supabase link --project-ref`, `[remotes]` | +| **Vercel** | `.vercel/project.json` (gitignored) | `vercel.json` | `api/.py` — file *or* dir | bundler resolves it | preview vs production; `vercel promote ` | +| **Cloudflare** | `wrangler.toml` | TOML | one worker per config | esbuild | `[env.staging]` + `--env staging` | +| **dbt** | `dbt_project.yml` | YAML | `models/**.sql` | `macros/` | `profiles.yml` targets + `--target prod` | +| **Modal** | none | Python | `-m src.app` module mode | `Image.add_local_python_source` (explicit since 1.0) | — | +| **Val Town** | `.vt/` | — | one val | — | `vt clone` / `push` / `watch` | + +Worth stealing: + +1. **Underscore = not a unit** (Supabase `_shared`). Zero config, instantly legible. +2. **File *or* directory** (Vercel). `functions/quick.py` for a one-liner; promote to `functions/quick/` when it grows helpers. +3. **`--env` + a config block per env** (wrangler, dbt). Deletes both Makefile hacks outright. +4. **`promote` moves the *artifact*, not the source** (Vercel). Byte-identical staging→prod. +5. **Explicit local sources** (Modal 1.0). Modal *removed* automounting because it was unpredictable. Ours is explicit by construction — only what's reachable from `main.py` via relative import gets bundled. +6. **Secret values via a gitignored `.env`, never in the config** (Supabase). `config.toml` is safe to commit; `supabase secrets set` pushes from `.env`. + +And four ideas from your own frameworks that beat anything in that table: + +7. **The manifest is a lockfile with per-environment state; the path is the identity.** + ```jsonc + {"envs":{"prod":{"function_id":"a0b8…","functions_version":"v20260821_162138","code_sha256":"ea2e…"}, + "dev":{"function_id":"7cc4…","functions_version":"v20260826_081715","code_sha256":"ea2e…"}}, + "path":"1001tracklists.com/fetch_batch_tracklists.py", …} + ``` + Per-env `code_sha256` is what lets **one tree serve all three environments** — a tree-wide hash would mean pushing to prod marks dev as up to date. No env-scoped id ever appears in a filename. +8. **The write and the bookkeeping are separate failure domains.** `entryAfterPush` advances `code_sha256` to the pushed bytes even when the confirmation read-back fails; only version strings go stale. Tying the hash to the read-back meant *"a transient error on the second request minted a duplicate upstream version on the next run."* +9. **A run is authoritative only for what it inspected.** `--limit 20` must never orphan the other 2,029. A *complete* walk that no longer lists something is the only thing that may remove it. And **never delete remote things absent from the tree** — report them (`reportExtraRemote`). +10. **Preview → guard → apply** (managed-auth): dry-run returns `target_state_sha256`; the apply sends it back as `expected_target_state_sha256`. Optimistic concurrency for free. + +--- + +## Proposed layout + +``` +my-functions/ +├── notte.toml # committed. intent: envs, defaults, per-function config +├── notte.lock.json # committed, machine-written. per-env ids + hashes +├── .env.dev / .env.prod # GITIGNORED. secret values only +├── .notte/ # gitignored. build output, caches +│ └── build/prod/amazon_search.py +├── pyrightconfig.json # written by `notte stack init` +├── AGENTS.md # written by `notte stack init` — the authoring contract +└── functions/ # ← real python package, importable from repo root + ├── __init__.py + ├── _shared/ # `_` prefix = library, never deployed + │ ├── __init__.py + │ ├── http.py + │ └── contract.py + ├── amazon_search/ # a function + │ ├── __init__.py + │ ├── main.py # entrypoint: must define exactly one `run()` + │ ├── parse.py # local helper, bundled in + │ └── test_main.py # colocated test, never bundled + └── quick_check.py # single-file function — no directory needed +``` + +`functions/amazon_search/main.py`: +```python +from pydantic import BaseModel +from .parse import parse_rows # sibling +from .._shared.http import fetch_json # shared + +class Response(BaseModel): + items: list[dict] + +def run(query: str = "laptop") -> Response: + return Response(items=parse_rows(fetch_json(query))) +``` + +**Why `functions/` at the repo root and not `notte/functions/`.** `notte` is a real PyPI package and is in `ALLOWED_IMPORTS`. A top-level `notte/` directory in a repo whose root lands on `sys.path` — which happens the moment you run pytest — shadows it, and pyright resolves your empty directory instead of the SDK. `functions/` has no such collision. Make it configurable (`[project] functions_dir = "functions"`) so you *can* have `notte/functions/`, and document the caveat there rather than defaulting into it. + +Everything is a real package (`__init__.py` generated by `notte stack new`), so relative imports resolve identically for pyright, pytest, and the bundler. Discovery rule, one sentence: **anything directly under `functions_dir` whose name doesn't start with `_` is a function — a `/main.py` or a `.py`.** + +--- + +## Config file format + +Two files, two owners, two formats. That split matters more than which format wins. + +### Which format for the hand-written config + +| | Comments | Editor schema/autocomplete | Go parsing | Deep nesting | Ecosystem | +|---|---|---|---|---|---| +| **JSON** | ✗ — fatal for a config you want to annotate | ✓✓ best in class: `$schema`, works in every editor with zero setup | stdlib | fine | JS/TS (`vercel.json`, `tsconfig.json`) | +| **JSONC / JSON5** | ✓ | partial | third-party | fine | VS Code only; no cross-editor story | +| **YAML** | ✓ | ✓ via `# yaml-language-server: $schema=` | third-party | ✓✓ | k8s, CI | +| **TOML** | ✓ | ✓ via taplo's `#:schema` directive + Even Better TOML | third-party | ✗ awkward past 2 levels | Python & Rust (`pyproject.toml`, `Cargo.toml`, `wrangler.toml`, `fly.toml`, `netlify.toml`) | + +**Recommendation: TOML, as `notte.toml`.** Three reasons, in order: + +1. **The users are Python developers.** They read `pyproject.toml` every day. A functions CLI whose config looks like `pyproject.toml` needs no explanation. +2. **Comments are the point.** Both existing frameworks are ~40% prose comments explaining *why* — `deploy.py`'s `NOTTE_API_URLS` carries a war story about `NOTTE_API_URL` silently pointing `pull prod` at dev. JSON cannot hold that, and it's exactly what belongs next to `[env.prod] api_url = …`. +3. **The nesting here is shallow by construction** — `[env.prod]`, `[functions.amazon_search]`. That's TOML's sweet spot, and it's the one place TOML is weak, so the weakness never bites. + +YAML is rejected on safety: whitespace significance, and the Norway problem (`no` → `false`) in a config whose values include country codes — `proxy_country = "no"` is a real value in the managed-auth template schema. + +Bare `.notte` is rejected outright: no extension means no highlighting, no schema association, no declared format until you open it. But `.notte/` as a **directory** for gitignored local state is right, and mirrors `.vercel/`, `.vt/`, `.terraform/`. + +Also rejected: `[tool.notte]` inside `pyproject.toml` (most Python-native, but the repo isn't a distributable package and it couples project config to a file uv/pip also own); `.notte.toml` (hidden files are for user-level config — you want this discoverable in `ls`); `notte.config.toml` (the JS `*.config.*` convention disambiguates; there's nothing to disambiguate from). + +### Ship a JSON Schema regardless + +TOML's schema story is real but needs a directive on line 1: +```toml +#:schema https://notte.cc/schema/notte-v1.json +``` +`notte stack init` writes it; `notte schema` prints it for vendoring. Validate against the same schema in `notte stack check`, so a typo'd key errors instead of doing nothing — TOML's failure mode for an unknown key is silence. + +### The rule that follows from Go's TOML libraries + +Every Go TOML library either drops comments on write or exposes them read-only; `pelletier/go-toml` v2 explicitly **dropped document editing from its requirements**. Comment-preserving edits need a niche lossless-AST library (`creachadair/tomledit`, `smm-h/go-toml-edit`). + +Make it a rule instead of a dependency: **`notte.toml` is never machine-rewritten.** `notte stack init` and `notte stack new` *render templates* (text, not a marshal round-trip); after that only humans edit it. Everything the CLI writes back goes to `notte.lock.json` — JSON, stdlib-parsed, machine-owned. That's precisely the line `marketplace/manifest.json` failed to draw: it mixed generated state (`pulled_at`, `run_count`) with copy (`name`, `description`, `categories`) that a human wants to own, and the result is a 2.2 MB file every sync dirties. + +### One interpolation syntax + +Config must reference things it may not contain. Supabase solved this with `env("MY_KEY")`. Use a uniform `${namespace:key}` so there's one rule: + +```toml +[env.prod] +api_url = "https://api.notte.cc" +api_key = "${env:NOTTE_API_KEY_PROD}" # resolved at use, never stored + +[env.preview] +extends = "dev" +headers = { "x-db-preview" = "${git:branch}" } +``` +Namespaces: `env:` (process environment, optionally from a gitignored `.env.`), `git:` (`branch`, `sha`, `short_sha`), `keyring:` (the existing `KeyringKeyForEnv` entries). **Fail loudly on an unresolved reference** — `deploy.py`'s `--db-preview` guard exists because a header silently ignored by staging meant *"a silent wrong write"*, and unresolved-to-empty-string is the same bug class. + +--- + +## Naming conventions + +| Thing | Choice | Why not the alternative | +|---|---|---| +| Project config | `notte.toml` | See above. | +| Lockfile | `notte.lock.json` | One entry per line — marketplace proved this keeps diffs reviewable at 2,049 entries. Separate from `notte.toml` because it's machine-written. | +| Local state | `.notte/` (gitignored) | Mirrors `.vercel/`. Build output lives here, never next to sources. | +| Secret values | `.env.` (gitignored) | Supabase's split. Never in `notte.toml`. | +| Package root | `functions/`, configurable | See layout section. | +| Entrypoint | `main.py` | Not `function.py` (redundant inside `functions//`), not `index.py` (a JS import), not `route.py` — a Notte function has exactly one `run()`; there is no route/handler split to encode. | +| Shared code | `_`-prefixed anything; `_shared/` by convention | Supabase's rule, and it doubles as the "not a function" marker. | +| Function name | directory or file stem, `[a-z0-9_-]+` | Never contains the function id — ids are per-env. | +| Build output | `.notte/build//.py` | Per-env because per-env config can change the bytes. | +| Envs | `dev`, `staging`, `prod`, `preview`, `local` | Both frameworks already use exactly these. | +| Deploy verb | `deploy` | Not `push`. `push` implies a round trip; bundling makes the tree source-of-truth and the flow one-way. Reserve `pull` for adoption/import. | + +--- + +## Command surface + +``` +notte stack init [dir] # scaffold notte.toml, functions/, pyrightconfig, .gitignore, AGENTS.md +notte stack init --from-session # bootstrap from `sessions workflow-code` — record, then scaffold +notte stack new # one function directory from a template + +notte stack deploy [] # build → diff → confirm → create/update → schedule → write lock +notte stack check [] # build + validate + diff vs remote. writes NOTHING. the CI gate. +notte stack status # what's drifted, and what a `_shared` edit would touch +notte stack pull # adopt existing remote functions into the tree + lock + +notte stack sync # create/refresh .notte/venv: Python 3.12, latest notte-sdk, + # plus the allowlisted packages your functions import +notte stack doctor # what is installed, resolved SDK vs latest, ty version, org +``` + +Eight commands. `` is a name, a glob, `all`, or a path — so `notte stack deploy functions/amazon_search` tab-completes. + +### Why `stack`, and the two rules that keep it honest + +The namespace exists because `notte functions` today is **id-centric and imperative** — `--function-id`, a global `~/.notte/cli/current_function`, one file at a time — while these are **directory-centric and declarative**. Two mental models under one name is the confusion worth preventing. Whether a command needs Python is not: validation degrades rather than being required, so a toolchain prefix would advertise a dependency that does not exist. No CLI in the prior art groups by toolchain anyway — Supabase needs Docker for `db` and `functions serve` and marks neither; `docker compose` and `git lfs` are namespaced because they are separate binaries. + +`stack` over the alternatives: `project` is generic and says nothing; `workspace` is **taken**, since Notte already means org by it (`workspaceIdFromUrl` in marketplace resolves an org id); `app` implies a running application; `fn`/`fns` is too close to `functions` to disambiguate anything. And `functions` itself becomes actively wrong the moment managed-auth connectors join, because you would be deploying connectors from a namespace called functions. `stack` absorbs them, which is the point — it names where this is going rather than only where it is. + +The one real cost is that in Pulumi and CloudFormation — the prior art users arrive with — **a stack is an environment** (`pulumi stack select dev`). Two rules keep that from surfacing, and they are constraints rather than preferences: + +1. **No stack selector, ever.** No `notte stack select prod`, no `notte stack prod deploy`. `--env` is the only way an environment is ever named. The ambiguity requires both spellings to exist. +2. **No `notte stack destroy`.** It is the command a Pulumi user reaches for, and it would promise resource-lifecycle semantics this design explicitly does not own. If teardown is ever needed, give it a name that does not imply cascade. + +### Environments are opt-in + +Almost every user deploys to prod and nothing else. Multi-environment support exists for `marketplace` and `managed-auth`, both internal, so it has to be **possible** without being **prominent**: + +- **`--env` defaults to `prod`** and appears in no quickstart example. +- **`notte stack init` scaffolds no `[env.*]` blocks.** Just `[project]`. An environments section gets added when a second environment actually exists. +- **`notte stack status` hides the environment column** for a single-environment project. +- **The lockfile keeps its per-environment shape regardless.** That costs a single-environment user one key, and marketplace established that a tree-wide hash silently marks dev up to date when you push to prod. + +An earlier draft had this inverted — scaffolding three `[env.*]` blocks and three API keys into every new project. That generalised from the two internal frameworks in the wrong direction, and would have left a first-time user believing three credentials were a prerequisite for deploying anything. + + +### Why `pull` is v1 and not a migration nicety + +It is tempting to file `pull` under "only needed to adopt an existing tree." That is wrong twice over. + +**An org with existing functions is the normal case, not the migration case.** Functions already arrive from `sessions workflow-code`, from the Anything API build agent, and from the console. The expected Notte flow is *author in the browser, then decide you want it in git* — which is `pull`, and which is also why `notte stack init --from-session` is really just a single-function `pull` wearing a different name. The machinery is required either way. + +**Without it, `deploy` is unsafe in a non-empty org.** Create-vs-update is decided from the lock: no `function_id` for this env → create. A fresh `notte stack init` against an org that already has `amazon_search` produces a lock that believes nothing exists, so the first `deploy` **creates a second `amazon_search`** rather than updating the first. `functions.name` has no unique constraint (`text NOT NULL DEFAULT 'default'`), so the API accepts it silently, and now two functions share a name while callers hold the id of the one that stopped being updated. That is the worst failure mode in this document, and `pull` is what prevents it. + +Which implies a companion rule: **`deploy` must refuse to create a function whose name already exists remotely but is absent from the lock**, and say `run 'notte stack pull' first`. Fail closed, same as the credential rule below. `--force-create` exists for the genuine case where you do want a second one. + +**`pull` is not the inverse of `deploy`, and must not pretend to be.** What comes back over the wire is the *artifact*, and a bundled artifact cannot be un-flattened into the package that produced it. So: + +- A function with no entry in the lock is written as a **single-file function** — `functions/.py` — because that is genuinely what it is. It can be promoted to a directory with helpers later, by hand, at which point it starts bundling. +- A function already in the lock and already deployed *from this tree* is **left alone**. Overwriting `functions/amazon_search/{main,parse}.py` with one flattened file would destroy the sources to "sync" them, which is the opposite of the intent. If its `artifact_sha256` doesn't match what's deployed, that's drift — report it, and let `status`/`deploy` handle it. +- Following marketplace: a run is authoritative only for what it inspected, so `--limit` or a failed download never prunes; and remote functions absent from the tree are **reported, never deleted**. + +**There is no bulk download, and no download command at all.** This is the part that makes `pull` a real batch job rather than one request, and it needs saying because the shape isn't obvious: + +- `GET /functions` returns `PaginatedResponseFunctionResponse{Items []FunctionResponse}`, and `FunctionResponse` **has no URL field**. Only `FunctionWithLinkResponse` carries `Url`, and that comes from `GET /functions/{id}`. +- So the code for each function costs **two more requests**: one to get the signed URL, one to fetch it. A full pull is `⌈N/100⌉ + 2N` — roughly 4,120 requests for marketplace's 2,049 functions. It ran that at concurrency 48. +- The URL is a Fernet token for Notte-managed functions, decrypted with a key derived client-side: `sha256(f"api_key:{key}:workflow_id:{id}:dumb")[:64]`, passed as `?decryption_key=`. +- **`notte functions download` does not exist.** `functions show` already calls `FunctionDownloadUrl`, prints the metadata, and throws the URL away. So today this is entirely hand-rolled — marketplace reimplements the key derivation and the fetch, which is how one secret-derivation rule ended up living in two repos. + +**This needs no backend change — parallelise it.** An earlier draft asked for `url` on the list endpoint to halve the request count. That ask is unnecessary, and marketplace is the evidence: it runs the full walk at `concurrency: 48` (`marketplace-catalog.ts:2103`, overridable with `--concurrency`) over the largest corpus that exists, and the comment on its retry classifier records the measurement: + +> *"Nothing hit a 429 while this was being measured, but a full pull is several times larger than any sample taken, and a retry is much cheaper than a half-written tree."* + +4,120 requests, no rate limiting. The retry logic is defensive rather than a response to observed throttling. For a realistic project — tens of functions, not thousands — `1 + 2N` at concurrency 48 is a second or two. Adding a backend dependency to optimise that would be trading real coordination cost for an imperceptible win. + +So what's actually needed is client-side, and all of it is CLI work: + +- **`notte functions download --function-id `** as a primitive in its own right, deriving the decryption key internally rather than exposing a flag users must understand. `pull` becomes a bounded-concurrency loop over it, and marketplace's hand-rolled fetch and duplicated key derivation both get deleted. +- Retry with backoff on 429/5xx honouring `Retry-After` — cheap, and the failure it prevents is a half-written tree. +- A **complete** page walk before anything is reported as a remote extra. A listing that stops early is indistinguishable from one where functions were deleted. + +**And `check` shouldn't pay this cost at all in the common case.** The lock already stores `artifact_sha256` per env, so the default gate — *"you changed sources and didn't deploy"* — is a local build plus a hash comparison, with no downloads and no page walk. `--verify-remote` does the full download walk to catch out-of-band edits made in the console. marketplace always downloads in `check` because it is a mirror with no separate source hash to trust; we have one, so we can be cheap by default and thorough on request. That also matches its own observation that a scheduled `check` against prod is a staleness alarm rather than a build gate. + +### Deferred, and why + +An earlier draft proposed eighteen commands. Roughly half were aspirational or gated on backend work that doesn't exist yet, and a large surface is its own cost — it has to be documented, tab-completed, kept coherent, and lived with. Everything below is deliberately *not* in v1: + +| Command | Why deferred | +|---|---| +| `run`, `logs`, `secrets`, `schedule` | already exist under `notte functions`; project-aware twins aren't needed on day one | +| `build` | folded into `deploy` and `check`. Expose separately only once someone actually wants the artifact without the diff | +| `promote`, `rollback` | need `--version` and `versions[]` exposed on the CLI first | +| `dev` | genuinely valuable, but it's a second execution model and deserves its own design rather than a line in this table | +| `whoami`, `auth login --env` | blocked on `GET /me` (backend ask #3). These ship *with* that endpoint — `auth login --env` is a prerequisite for the fail-closed credential rule below, not an optional extra | + +The credential resolution rules in the next section apply to all six commands. `--env` is not deferred — it just defaults to prod and stays out of the way. + +`notte.toml`: +```toml +#:schema https://notte.cc/schema/notte-v1.json + +[project] +name = "anything-api" +functions_dir = "functions" + +[functions.amazon_search] +name = "Amazon AE product search" +description = "…" +shared = true +cron = "cron(0 9 * * ? *)" +secrets = ["AMAZON_PARTNER_TAG"] # in addition to what the AST scan finds +``` + +That is the whole file for the common case: no environments, no credentials, prod implied. A project that genuinely has several adds them, and only then: + +```toml +[env.dev] +api_url = "https://us-dev.notte.cc" +api_key = "${env:NOTTE_API_KEY_DEV}" + +[env.staging] +api_url = "https://us-staging.notte.cc" +api_key = "${env:NOTTE_API_KEY_STAGING}" + +[env.preview] +extends = "dev" +headers = { "x-db-preview" = "${git:branch}" } # generalizes managed-auth's preview mode +``` + +### Credentials resolve *from* the environment, never beside it + +API keys are **not** literals in `notte.toml`. More importantly, **the key and the URL must be resolved as one unit.** Selecting an env fixes `api_url`, and every candidate key is then derived from that same `api_url` — there is no step in the chain that can hand back a credential belonging to a different endpoint: + +1. `--api-key` (explicit, and the operator owns the consequences) +2. the `api_key = "${env:…}"` reference declared **inside that env's own block** +3. the keyring entry under `KeyringKeyForEnv(hostToEnvLabel(api_url))` — the label computed from the resolved URL, not from ambient state +4. **stop.** Fail with `no credential for env 'staging' (https://us-staging.notte.cc) — set NOTTE_API_KEY_STAGING or run 'notte auth login --env staging'`. + +The two fallbacks the global CLI uses today — a bare `NOTTE_API_KEY` and `~/.notte/cli/config.json` — are **deliberately not in this chain**, because neither is tied to an endpoint. A developer with `NOTTE_API_KEY` exported for prod running `notte stack deploy --env staging` would otherwise authenticate to staging with a prod key: it fails closed if the orgs differ, but it succeeds and writes to the *wrong org* whenever they don't. `notte stack deploy` is the command where that matters most. + +This is the same bug `marketplace-catalog.ts` already documents having hit from the other direction — its `NOTTE_API_URLS` table exists precisely because reusing a helper that read ambient `NOTTE_API_URL` made `pull prod` silently read dev. Its `createNotteRunner` then passes `NOTTE_API_KEY` and `NOTTE_API_URL` to the subprocess explicitly, together, never ambient. Same rule, enforced one level up. + +`notte auth login --env ` and `notte whoami --env ` are the paired ergonomics that make failing closed tolerable, and `notte stack status` should print the resolved org for each configured env so a misconfiguration is visible before a deploy rather than after. + +--- + +## The bundler + +### Algorithm + +1. Parse `main.py`; collect relative imports (`from .x import a`, `from ..y.z import b`). +2. Resolve to files inside `functions_dir`; recurse. Anything outside the package, or non-relative, is left alone and checked against the allowlist. +3. Topologically sort. Cycle → error naming the cycle. +4. Emit: header, then `from __future__ import annotations` if any module had it (exactly one, first statement), then hoisted+deduped third-party imports, then each dependency's body in topological order with its relative-import lines **replaced** (see aliases below), then `main.py`'s body last. +5. Hash the artifact. Write `.notte/build//.py` + a source map. + +**Aliased relative imports keep their binding.** A relative import line is not simply deleted — it is replaced in place by an assignment per aliased name: + +```python +from .parse import parse_rows as pr, clean # source +pr = parse_rows # artifact (clean needs nothing) +``` + +Deleting the line outright would drop `pr` and the artifact would die with `NameError` at run time, which is the worst possible failure mode: it passes the bundler, passes upload validation, and fails in production. Unaliased names need no assignment because the flattened definition already carries that name, and dependency bodies are emitted before the body that imports them, so the right-hand side is always bound by the time the assignment runs. + +**Collisions are an error, not a rename.** If `_shared/http.py` and `parse.py` both define `clean`, fail with `_shared/http.py:12 and parse.py:8 both define 'clean' — rename one`. This is the pivotal simplification: **no reference rewriting is ever needed**, so no full-fidelity Python parser is needed, and the artifact stays byte-readable — which matters because that artifact is what the console shows and what tracebacks point at. + +The collision set is every top-level binding **plus every alias introduced above** — `from .parse import clean as fetch` collides with a `fetch` defined in `_shared/http.py` exactly as a second `def fetch` would, and must be reported the same way. + +Rejected in v1, each with a fix-it message: +- `from . import mod` then `mod.f()` → *"use `from .mod import f`"*. Neither existing codebase does this. +- `from .x import *` → *"star imports can't be flattened"*. +- Import cycles. +- Relative imports inside a function body or `if TYPE_CHECKING`. + +### Where it runs + +| | Go-native | Shell out to Python/`uv` | Server-side | +|---|---|---|---| +| Parse fidelity | Purpose-built tokenizer. Sufficient **because collisions error out**. `go-python/gpython` is a Python 3.4 grammar — no f-strings, no walrus, no `match` — so it isn't an option. | Perfect: Python's own `ast`. | Perfect. | +| Runtime deps | None. Brew binary works in CI, in a bare container, everywhere. | Needs `python3`/`uv` present. | None. | +| local build offline | Yes | Yes | **No** — you lose local preview and the CI gate | +| Agreement with `ScriptValidator` | Must mirror the allowlist as data (drifts) | Same problem — the validator lives in `notte-api`, not in a pip package | Authoritative by construction | +| Cost | ~600 lines Go + tests | ~200 lines Python, `//go:embed`-ed, run via `uv run --script` | Backend work: accept a tar, bundle, validate | +| Failure mode | A weird import form is rejected with a clear message | "python3: not found" on a machine where the CLI otherwise works | Slow loop; can't check in a PR without credentials | + +**Recommendation: Go flattens, Python validates.** Not either column alone — each does the half it is actually good at, and neither reimplements the other. + +**Go does the flattening**, with no toolchain requirement. Resolution, topological sort, concatenation, import hoisting and alias preservation are a text transformation, and the collisions-error-out rule shrinks the parsing problem to import discovery plus top-level binding extraction, all line-oriented at indent zero. Validated against `anything-api/marketplace`: 2,524 production files, zero bundle errors, zero artifacts failing `py_compile`, zero lost definitions. + +**Python validates**, when it is available — and this is where the real leverage is. Mirroring the server's rules in Go means maintaining a copy of `ALLOWED_IMPORTS`, a denylist, and a stdlib set generated from a pinned CPython. Every one of those drifts. With an interpreter present you stop mirroring and run the real thing: + +| Problem | Cost of mirroring in Go | With Python present | +|---|---|---| +| Tokenizer edge cases | a hand-written scanner | `ast`, the parser the server uses | +| Import allowlist drift | vendored copy of two lists | import the real `ScriptValidator` | +| stdlib version mismatch | pinned generator, version-guarded | the runtime's own interpreter | +| Semantic errors | not detectable at all | `ty check` | + +That last row is the one the flattener cannot cover by itself. A *missed* collision produces valid Python with the wrong meaning, so `py_compile` passes — but a type checker reports it as a redefinition. The gate closes exactly the hole the bundler can leave open. + +### `notte stack` requires Python; the rest of the CLI does not + +`notte sessions`, `notte page` and friends stay pure Go with no toolchain. `notte stack` requires an interpreter, and requires rather than prefers it. + +An earlier draft made it optional, degrading to a vendored copy of the server's rules when uv was absent. That copy was the problem. Mirroring `ALLOWED_IMPORTS`, a denylist, and a stdlib set generated from a pinned CPython means maintaining three things that drift from a backend this repo does not control — and two of them drifted within a week of being written, one of them shipping a list from CPython 3.14 that would have rejected `telnetlib`, `cgi` and 17 other modules the 3.12 runner actually has. + +Requiring Python deletes the mirror rather than maintaining it: + +| Deleted | Replaced by | +|---|---| +| a vendored `ALLOWED_IMPORTS` + denylist | the real `ScriptValidator.parse_script(source, restricted=True)` | +| a stdlib set generated from a pinned CPython | the interpreter's own `sys.stdlib_module_names` | +| `make generate-stdlib` and its script | nothing | +| "is this check current?" | not a question that can be asked | + +The ask is small because **uv downloads the interpreter itself**, so the requirement is "have uv" — one binary. `managed-auth` already works this way. It needs one line in `--help` and a good error, not a redesign. + +### The runtime describes itself: `GET /functions/health` + +Everything below depends on `nottelabs/monorepo#2394`, which makes the runner report its own contract: + +```json +{ "status": "ok", "python_version": "3.12.0", "runtime_digest": "sha256:…", + "packages": [{"import_name": "notte_sdk", "package": "notte-sdk", "version": "1.4.4", + "installed": true, + "source": "git+https://github.com/nottelabs/notte@#subdirectory=packages/notte-sdk"}], + "stdlib_modules": ["asyncio", "json", "…"], + "reserved_env_names": ["NOTTE_API_KEY", "…"] } +``` + +This removes the vendored copies entirely rather than reducing them. No allowlist in Go, no generated stdlib list, no hand-pinned CPython version, no scraped SDK commit. + +Three properties of it that the client has to respect: + +- **`source` is read from the install's PEP 610 `direct_url.json`, not from build args.** The runner installs `notte-sdk` and `notte-core` from git SHAs, so a version number is not an identity — and build args record intent while dist-info records what is actually in the image. `sync` installs from `source` when present and from the index when it is `null`. +- **`runtime_digest` covers the contract fields only** — python version, packages with sources, allowed imports, reserved names — deliberately not status, latency or reachability. So it moves if and only if a venv built against the previous answer would now be wrong. It is also the ETag, but there is no `If-None-Match` handling: compare the value, do not expect a 304. +- **`status: "degraded"` is a normal state, not an error.** Between an API deploy and the runner image rebuild, `/runtime` 404s on a perfectly healthy runner, so `python_version`, `packages` and `runtime_digest` are absent and the digest is `null`. Rules that follow: never block a deploy on it, never overwrite a good cached report with a partial one, and note that `reserved_env_names` is still populated because it is the API's rule rather than the runner's — so `secrets push` validation keeps working throughout. + +Caching is for container cycling rather than latency: `runtime_info()` is cached per container, so the round trip is ~295 ms cold and ~12 ms warm. Cache the report in `.notte/` keyed by `runtime_digest`; refresh on `sync` and `doctor`. + +### The venv is the enforcement, and the validator owns only structure + +`notte stack sync` builds `.notte/venv` with the reported `python_version`, and installs the reported packages **at their reported sources** — the intersection with what the functions actually import, not the whole list. + +Once the venv mirrors the runner image, **`ty`'s `unresolved-import` *is* the allowlist violation.** No third-party allowlist has to exist client-side, because the environment enforces it. An import of something not reported is an error at sync time naming the file and line, never a silent install. + +**The SDK's own validator cannot be trusted for imports, and this is not a hypothetical.** `notte_core.ast.ScriptValidator` in the published `notte-sdk 1.8.31` carries an explicit 41-entry allowlist that omits `httpcloak`, `httpx`, `bs4` and `tqdm`, and *includes* `tempfile`, which the runner discards. Running it against `marketplace/99.co/list_condos_by_letter.py` — deployed and serving traffic — rejects it outright, along with the ~333 other functions that import `httpcloak`. The runner installs `notte_core` from a git SHA rather than from the index, so the published package and the running code are different code under the same version number. + +So the split is: + +| Owner | Checks | +|---|---| +| `GET /functions/health` + the venv | which imports are allowed, at which versions | +| `notte_core.ast.ScriptValidator` | structure: one top-level `run()`, forbidden nodes, forbidden calls, relative imports, stdlib denials, `variables` extraction | +| `ty` | semantics, including the redefinitions a flattener could silently introduce | + +The validator's structural checks were verified against the published SDK and are correct there. One known gap: it **accepts two top-level `run()` definitions** where the server rejects them, so the CLI checks that itself. + +Worth fixing independently of this CLI: anyone calling `parse_script` from the published SDK today gets wrong answers about production functions. + +**`sync` does not generate a `pyproject.toml`.** It is tempting — the project would become a normal Python project and `pytest` would work unconfigured — but a generated `pyproject.toml` is clobbered the moment someone adds `pytest` or `ruff` to it. That is exactly the mistake `marketplace/manifest.json` made by mixing generated state with hand-owned content. `pyproject.toml` stays entirely the user's; the CLI owns `.notte/` and nothing else in the repo root. + +### `ty`, and the way it fails open + +Use `ty`, not basedpyright. `anything-api` already gates its build agent on `ty check client.py` before every create or update, so there is in-house precedent and a working configuration; and ty's 10–100x advantage on *cold* checks is precisely the case a CLI hits on every invocation. It is on `0.0.x` with no stable API, which argues for pinning the version, not for avoiding it. + +The trap is documented in `anything-api`'s `ty-config.ts`, and it is worth quoting because the CLI would walk straight into it: + +> ty does NOT resolve imports against the environment it was pip-installed into. With no `VIRTUAL_ENV`, no `.venv` and no `--python`, it falls back to the first `python` on PATH — the system 3.9 in the sandbox, not the 3.11 the snapshot installed notte-sdk into. So every generated client.py reported `unresolved-import` for `requests`, `pydantic` AND `notte_sdk`, and **the agent deployed straight through the mandatory type check.** + +A type checker that cannot resolve imports does not fail. It emits `unresolved-import` and everything downstream reads green — the worst shape a gate can have, since it looks like coverage and provides none. So: + +- **Write a `ty.toml` naming the interpreter explicitly.** Never rely on ambient `python`. Under `uv` the CLI owns the venv, so this is a known path rather than a probe — but the config must still be written. +- **`unresolved-import` for an allowlisted package is a hard error**, never an environment artefact to wave through. That is already the rule in the build agent's prompt. +- ty treats a wrong `environment.python` as fatal for the whole run, and rejects an `extra-paths` entry that is not a directory. Both are worse than the bug being fixed, so both are checked before the file is written. + +### Which `notte-sdk` to check against + +Latest. The runner image pins `notte-sdk` to a commit extracted from `notte-api/uv.lock` at build time (`build-docker.sh:54`), which sounds like it could lag PyPI — but the monorepo has a hard check that the latest SDK is installed before every release, so the lock is bumped and the image rebuilt on every SDK release. Latest tracks the runtime. + +So the CLI pins nothing and resolves fresh, and **`deploy` fails on SDK skew**, because a green check against an SDK the runtime does not run is exactly the drift this section removes. Two guards keep that from being outage-shaped: + +- **Never fail on unreachable PyPI.** Fall back to the uv cache and say so. An offline runner or a registry blip must not block a hotfix. +- **`--allow-sdk-skew`**, with the message naming installed versus latest. A door exists, or people route around the tool entirely. + +One consequence worth stating plainly: because the runtime moves, **a deployed function can break without anyone touching it.** A scheduled `notte stack check --verify-remote` is the only thing that would notice, which makes it more than the staleness alarm it is described as above. + +### Two hashes, because bundling breaks the round trip + +marketplace's core invariant — *"files are byte-identical to what prod serves"* — cannot survive a bundler. Replace it with two hashes in the lock: + +- `source_sha256` — over the canonicalized *set* of contributing source files (sorted `path:sha256` pairs). Drives create-vs-update. +- `artifact_sha256` — the bundled bytes. Drives the diff shown before confirming, against what's deployed now. + +And now that byte-fidelity is gone, a generated header is free and should be there: +```python +# generated by notte 0.1.0 — do not edit +# sources: functions/amazon_search/{main,parse}.py, functions/_shared/http.py +# source-sha256: 4f2a… +``` + +--- + +## Secrets + +The API shape (`notte-api/src/notte_api/secrets/endpoints.py`, table `tenant_secrets`): + +- `POST /secrets {namespace, name, value}` → 201. Namespaces: `function_env`, `llm_provider`. +- `GET /secrets?namespace=` → metadata only: `{id, namespace, name, key_hint, created_at, last_used_at}`. +- `GET /secrets/{name}?namespace=` → **plaintext value**. +- `DELETE /secrets/{secret_id}` → 204. + +Four facts that shape the design: + +1. **`(namespace, name)` is UNIQUE** — enforced by partial indexes, scoped org-or-user. That is the best natural key in the whole API and it makes secrets genuinely declarable. +2. **`function_env` names must match `^[A-Z_][A-Z0-9_]*$`**, ≤128, and must not be in `{NOTTE_API_KEY, NOTTE_API_URL, NOTTE_BASE_URL, NOTTE_ENV, ENVIRONMENT, NOTTE_DB_PREVIEW_BRANCH}`. Validate all of that client-side. +3. **`POST` is not an upsert.** It's a bare insert; a duplicate is a **409**. And **delete is by UUID, not name.** So "update a secret" means LIST → map name→id → DELETE → POST. Non-atomic, and there's a window where the secret doesn't exist. +4. **Secrets are org-scoped, not per-function.** Since each env is a different key/org, per-env separation comes free — but it also means the CLI's current placement of these under `notte functions secrets` is misleading. Promote to top-level `notte secrets`. + +### The model + +**Names in git, values in a gitignored `.env.`** (the same split Supabase uses). + +The declared set for an env = union of every deployed function's `required_secrets` (which the server already computes from the AST scan and returns on `FunctionResponse`) **plus** any explicit `[functions.x] secrets = [...]` for names computed at runtime that no scanner can see. + +``` +$ notte secrets diff --env prod + missing on prod (declared, not set): + AMAZON_PARTNER_TAG required by amazon_search + STOCKX_TOKEN required by stockx_search, stockx_bid + set on prod, not declared: + OLD_SCRAPER_KEY (not deleted — pass --prune to remove) +``` + +- `notte secrets push --env prod` reads `.env.prod`, POSTs what's missing. For a name that exists with a different `key_hint`, it DELETEs then POSTs — and **says so explicitly**, because that sequence is not atomic. +- **Diff on `key_hint`, never on values.** Reading a value back writes an `audit_events` row (`entity_type="tenant_secret"`, `action="read"`) and bumps `last_used_at`. A `diff` that silently audits every secret on every CI run is a bad neighbour. +- **Never prune by default.** Same rule as `reportExtraRemote`: report extras, delete only on `--prune`. +- Never print values, in any output mode. + +### Deploy preflight + +`preflight_required_secrets` already raises **422** with the missing names at run time, before a session starts (so nothing is charged). Pull that forward: after `notte stack deploy` writes a function, read back `required_secrets`, diff against the env, and warn — **especially before setting a cron**, since otherwise the first sign of trouble is a 09:00 job failing on a Sunday. + +--- + +## Schedules + +`POST /functions/{id}/schedule {cron, variables}` is genuinely good for reconciliation: an upsert, idempotent for the same cron, guarded by a Postgres advisory lock plus optimistic `schedule_revision` CAS, with EventBridge rollback compensation. It validates `variables` keys against the function's declared `variables`. + +**The blocker: there is no read endpoint.** The data exists — `functions.schedule_cron`, `schedule_variables`, `schedule_state`, `schedule_paused_reason`, `schedule_paused_at`, `schedule_revision` are all on the row and all on `SupabaseFunctionResponse` — but the API's `FunctionResponse` model drops them, so FastAPI never serialises them, so the generated Go client can't see them. The CLI has `schedule`/`unschedule` and nothing else. + +Consequence: `notte stack status` cannot show the current cron, and `notte stack deploy` cannot distinguish "already correct" from "about to change". Two options: + +- **Preferred:** add the three fields to `FunctionResponse` (`functions/endpoints.py:50-63`) — additive, mirrors exactly how `published` and `required_secrets` were added, and regenerates through OpenAPI → Go client. Then cron reconciles properly. +- **Until then:** record the last-applied cron in `notte.lock.json` and treat the lock as truth, printing a caveat that a change made in the console is invisible to `status`. + +Two rules regardless: + +- **Validate the cron client-side.** It must be six-field AWS EventBridge form (`cron(m h dom mon dow year)`) or `rate(...)`. A five-field crontab string is the natural thing to write and the docs currently show one. +- **Never fight `schedule_state`.** The system pauses schedules for `credit_exhausted` or `function_inactive`. A reconciler that re-POSTs because state ≠ enabled will thrash against the billing system. Reconcile on `cron` + `variables` only; surface `schedule_paused_reason` in `status` as information. + +--- + +## Managed auth — deferred, and why it will be easy + +**Templates are already the declarative design you're describing.** `POST /managed-auth/templates/import?dry_run=true` returns a real field-level diff — `{slug, action: create|update|no_change, previous_revision, revision, metadata_changes[], login_changed, verifier_changed, bundle_sha256, target_state_sha256}` — and the apply is guarded by `expected_target_state_sha256` from the preview. `GET /managed-auth/templates/{slug}/export` round-trips it. `slug` is UNIQUE. It is strictly the best-designed surface in the API, and `connectors/.json` is already a git-versioned manifest. + +Three things to know before adopting it: +- The router is **`include_in_schema=False`** (`main.py:719`), so managed-auth is absent from the OpenAPI spec and therefore absent from the generated Go client entirely. Flip that, or hand-write the client. +- Import is gated to one hard-coded org: `_require_connector_organization()` demands `org_id == "4dbf683a-…"`. Fine for you; a blocker if customers should ever declare their own connectors. +- A **connection** is not a template. Creating one runs a real browser login, spends money, and provisions a vault + a browser profile as side effects; `PATCH` covers only `label` and `schedule`; `credentials`, `vault_id`, `mailbox_id`, `two_fa_method`, `domain` are all create-only and immutable; and most of its fields (`status`, `last_login_at`, `last_failure_code`) are observed runtime state. Reconciling a connection means delete-and-re-login. **Keep connections imperative.** What a CLI can usefully add is operational commands (`list`, `check`, `reauthenticate`, `reset-profile`) and CI fixtures — the `ci-longlived-` / `ci-smoke--` pattern `smoke.py` already implements, including its hard refusal to delete anything with the long-lived prefix. + +Related: vaults, personas, and profiles all have server-generated UUIDs with non-unique names, so none of them are name-addressable. Profiles are the closest — `GET /profiles?name=` supports filtering, which makes find-or-create viable. Vault *credentials* are keyed by `(vault_id, root domain of url)`, which is a real natural key and effectively an upsert. Worth knowing, not worth building yet. + +--- + +## The DX ideas worth building + +**Source maps.** Emit `.notte/build//.map.json` mapping artifact line ranges → `source:line`, and have `notte logs` / `run-metadata` rewrite tracebacks through it. A traceback that says `line 612` in a 900-line concatenated file is the single worst thing about any bundler, and nobody in Python serverless fixes it. Highest-leverage item here. + +**Blast radius in `notte stack status`.** `_shared/contract.py` is inlined into every function that imports it, so editing it changes N artifacts. managed-auth papered over exactly this with `scripts/check_revision_bumps.py` (122 lines) plus a note in four separate docs. The CLI knows the import graph: +``` +$ notte stack status + functions/_shared/contract.py changed → 9 functions affected + ✗ google_login drifted (source 4f2a… ≠ deployed 8c31…) + ✗ bluesky_login drifted + … +``` +Auto-derived. `check_revision_bumps.py` and the manual `revision` field both disappear. + +**`notte promote` moves bytes, not source.** Download the artifact deployed to staging, upload those exact bytes to prod, record both hashes. Guarantees what you tested is what ships — stronger than re-running the build, and it's Vercel's model. + +**`notte stack check` as the CI gate `anything-api` designed and never wired up.** Writes nothing, exits non-zero on drift; `notte stack init` drops the GitHub Action in. Heed marketplace's warning: on a PR it's a genuine gate; on a schedule against prod it's a *staleness alarm*, since the catalog changes whenever anyone publishes. + +**`notte dev `.** Run the entrypoint locally against real cloud sessions, `--var` → `run()` kwargs. The current inner loop is deploy-to-test. This is `supabase functions serve` / `wrangler dev`, and it's where `drive_login.py` and the whole `login/*.py` exploratory-recording corpus want to live. + +**Confirm on the diff, not on the destination.** An earlier draft had `[env.prod] confirm = true`, which makes sense only when prod is the exceptional target. For almost every user it is the *only* target, so it degrades into friction on every deploy. The diff-then-confirm step already covers the real risk and keys off *what changed* rather than *where it is going*, which is the better signal anyway. marketplace's `push` refuses without a TTY, naming all three ways out (`--yes`, `--apply`, `--dry-run`); keep that wording. + +**Expose what's already in the generated client.** `--version` on update and `versions[]` on show → `notte rollback --to v20260821_162138`. `--decryption-key`, or just derive it automatically → deletes the duplicated `sha256("api_key:{k}:workflow_id:{id}:dumb")[:64]` that currently lives in two repos. + +**Agent-native scaffolding.** `notte stack init` writes `AGENTS.md` with the real contract — `run()` returns a `BaseModel`, the import allowlist, `from notte_sdk.types import os`, six-field cron — and registers the `notte-browser` skill. Encore does exactly this (`encore app create` asks which AI tool and writes the rules file). The material exists as `notte-skills/plugins/notte-cli/skills/notte-browser/references/function-management.md`; it needs the project layout added and to stay in sync (it's a submodule). + +**`notte init --from-session `.** `sessions workflow-code` already emits a deployable `run()`. Record a workflow in a browser → scaffold a project around it. An onboarding path nothing else in the prior-art table can offer. + +**Testing.** Two layers. + +*For user projects:* `test_*.py` colocated in the function dir, never bundled, run with pytest; `notte stack check --test` runs them. managed-auth's `ShippedConnectorsTest` — asserting properties of the real checked-in catalog, e.g. *"every connector still parses once inlined"* — generalizes into `notte stack check` itself and stops being something each repo hand-writes. + +*For the CLI itself:* the bundler is the part where a wrong answer is silent, so it ships with a golden-file suite before it ships at all — `internal/bundle/testdata//{in/,want.py}`, one directory per case, following `marketplace-catalog.ts`'s convention of naming each test after the invariant it protects. The cases that must exist on day one: + +| Case | Asserts | +|---|---| +| `alias-preserved` | `from .parse import f as g` emits `g = f`; the artifact defines `g` | +| `alias-collides` | an alias colliding with another module's top-level name is reported, not silently shadowed | +| `collision-reported` | two modules defining `clean` fail with both file:line locations | +| `topo-order` | a dependency's body precedes every body that imports it | +| `diamond` | a module reached by two paths is emitted exactly once | +| `cycle-rejected` | the error names the cycle | +| `future-annotations` | emitted once, first statement, even when three modules declare it | +| `import-hoist-dedup` | `import requests` in four modules yields one line | +| `star-import`, `from-dot-import`, `import-in-function` | each rejected with its fix-it message | +| `disallowed-import` | `import os` fails locally with the `notte_sdk.types` hint, before any upload | +| `deterministic` | bundling twice byte-identical — `artifact_sha256` is load-bearing for the whole diff model | +| `source-map` | every artifact line maps to a real `source:line` | + +`alias-preserved` and `alias-collides` exist because that gap was found in review of this document rather than in a test — which is the argument for the table. + +--- + +## Migration + +**`managed-auth`** — the cleaner fit for the bundler. `contract.py` → `functions/_shared/contract.py`; `login/bluesky_login.py` → `functions/bluesky_login/main.py`; `verifier/bluesky.py` → `functions/bluesky_verify/main.py`. `from contract import LoginResult` becomes `from .._shared.contract import LoginResult`, and `inline_contract()` — including its `re.sub` escape-expansion war story — is deleted. `login/email_2fa.py` becomes `functions/_shared/email_2fa.py` and the four hand-copied `_verification_code()` loops collapse into one import. `revision` and `check_revision_bumps.py` are replaced by `source_sha256`. What doesn't map in v1: the connector concept (one slug = a login + verifier deployed transactionally) and `/managed-auth/templates/import`. Keep a thin `deploy.py` for the template bundle, let `notte` own the two functions, and revisit when managed-auth joins the project model. + +**`marketplace`** — 2,049 files, zero relative imports, so bundling is a no-op for every one of them. The value is deleting `marketplace-catalog.ts`: the `MAKECMDGOALS` filtering, `createNotteRunner` + `detectCliError`, `redact()`, the org preflight, the decryption-key derivation, the `pool`/`retry` helpers — all CLI-native. `manifest.json` maps almost field-for-field onto `notte.lock.json`; `envs[env].{function_id, functions_version, versions, code_sha256}` is already the right shape. The gap it exposes: **`name`, `description`, `categories` are only editable upstream** — `push` only ever sends `--file`. `[functions.]` should own them and `notte stack deploy` should push them, which is a real capability gain over what exists. + +--- + +## Backend asks (ordered by how much they unblock) + +*Landed while this RFC was in review: `GET /functions/health` (`nottelabs/monorepo#2394`) reports the runner's Python version, allowed imports, package versions and install sources, reserved env names, and a content digest. It removes every vendored copy of the runtime's rules — see the validation section above.* + +1. **Add `schedule_cron` / `schedule_variables` / `schedule_state` to `FunctionResponse`** (`functions/endpoints.py:50-63`). ~2 lines, additive, exactly how `published` and `required_secrets` were added. Without it, cron cannot be reconciled — only blindly re-applied. +2. **Make secrets updatable by name**: `PUT /secrets/{namespace}/{name}` as an upsert, and `DELETE` by `(namespace, name)`. Today a secret rotation is LIST → DELETE-by-uuid → POST, which is three calls and a window where the secret doesn't exist. +3. **`GET /me`** → `{user_id, org_id, org_name, org_role, plan_type}`. A `notte.toml` committed to git gets applied by different keys; without this, "am I about to deploy to the right org?" is unanswerable. It also deletes marketplace's hack of reading the org id out of the first path segment of a signed download URL. +4. **`POST /functions?dry_run=true`** returning a managed-auth-style diff (`action`, `previous_version`, `changed`, `target_state_sha256`). Enables preview→guard→apply for functions and makes `notte stack check` server-authoritative. +5. *(later, for managed auth)* Flip `include_in_schema` on the managed-auth router so the Go client can be generated, and decide whether template import stays gated to the connectors org. + +--- + +## Open questions + +1. **Should `safe_import` remain the runtime import guard?** Execution already runs with `restricted=False`, so the RestrictedPython AST policy is off and `safe_import` is the only thing standing between a deployed function and arbitrary imports. It is also, incidentally, what makes off-the-shelf bundlers unusable. Relaxing it would make them work — but that trade should be evaluated on its own merits, not taken as a side effect of a bundling convenience, and this RFC does not need it either way. +2. **`notte functions` vs the project commands.** The current commands are function-id-centric with global `~/.notte/cli/current_function` state; the new ones are project-centric. Proposal: `GetCurrentFunctionID()` gains a fourth source — the project lock, resolved from cwd — ahead of the global state file, so both surfaces stay coherent. +3. **Is `preview` (dev + `x-db-preview: `) Notte-wide or managed-auth-specific?** Modeled above as generic `[env.*] headers`, which may be over-general. +4. **Function grouping.** managed-auth needs "these two deploy together, transactionally." Does a `[bundle]` concept belong in the model now, or is it deferred with managed auth? diff --git a/docs/rfcs/0002-connectors-in-notte-stack.md b/docs/rfcs/0002-connectors-in-notte-stack.md new file mode 100644 index 0000000..d16d5aa --- /dev/null +++ b/docs/rfcs/0002-connectors-in-notte-stack.md @@ -0,0 +1,226 @@ +# RFC 0002 — Managed-auth connectors in `notte stack` + +| | | +|---|---| +| **Status** | Draft — design only, nothing implemented | +| **Date** | 2026-08-30 | +| **Depends on** | [RFC 0001](0001-notte-project-scaffolding-and-deploy.md) | + +--- + +## Context + +`apps/back/managed-auth` deploys nine connectors with **1,502 lines of bespoke machinery**: + +| | Lines | What it does | +|---|---|---| +| `scripts/deploy.py` | 996 | bundling, contract checks, env resolution, target selection, the template import | +| `contract.py` | 271 | the shared runtime contract, spliced into every source by regex | +| `Makefile` | 113 | positional-goal parsing with a `%::` catch-all so URLs containing `:` survive | +| `scripts/check_revision_bumps.py` | 122 | enforces that a changed connector had its `revision` incremented | + +RFC 0001 already absorbs most of that for plain functions. What it does not cover is the thing managed-auth actually is: **a connector is two functions plus metadata, deployed together**. So `deploy.py` survives today for one reason — `POST /managed-auth/templates/import` — while carrying a thousand lines that the CLI now duplicates. + +Two goals, and the second is the more interesting one: + +1. Delete the machinery. A connector should be a directory, not a manifest plus a regex. +2. **Let customers ship their own authenticated connectors**, not just Notte internally. Today `_require_connector_organization()` hard-codes a single org UUID, so the capability exists but only we can reach it. + +--- + +## The shape + +The entrypoint filename declares the kind. That is the whole rule. + +``` +functions/ + _shared/ + contract.py LoginResult, VerifierResult, classify_login_failure + email_2fa.py the mailbox polling loop, written once + — both become an importable package later; see below + amazon_search/ + main.py → a function + bluesky/ + login.py → a connector: run(session_id) -> LoginResult + verifier.py → run(session_id) -> VerifierResult + helpers.py connector-local, bundled into both roles +``` + +**Discovery gains one sentence, and it is depth-independent.** *Any* directory under `functions/` containing `main.py` is a function; any containing `login.py` and `verifier.py` is a connector. Neither, or only one of the pair, is an error naming both options — the same treatment a directory without `main.py` already gets. + +Depth-independence is what lets grouping be a convention rather than a rule. `functions/bluesky/` and `functions/auth/bluesky/` both work, so a project with nine connectors can stay flat and one with fifty can group, without the CLI reserving the name `auth` or anyone migrating. Verified that the bundler already resolves three- and four-dot relative imports, so the deeper layout needs no change to it. + +The slug is the directory name, never the path: `functions/auth/bluesky/` deploys as `bluesky`. The catalog slug is globally unique, so a grouping directory must not leak into it. + +**This costs nothing in the bundler.** Each role is an ordinary entrypoint, and the existing flattener already handles two of them in one directory sharing local helpers. Verified: + +``` +bluesky/login.py -> [bluesky/helpers.py _shared/contract.py bluesky/login.py] +bluesky/verifier.py -> [bluesky/helpers.py _shared/contract.py bluesky/verifier.py] +``` + +Relative imports stay two dots, exactly as in a function: `from .._shared.contract import LoginResult`. + +### On grouping directories + +An earlier draft argued against `functions/auth//` on the grounds that it pushes shared imports to three dots. That objection was too strong: the extra dot is cosmetic, and it was outweighed by something the flat rule genuinely costs — **kind is invisible in a listing**. `ls functions/` showing `amazon_search/ bluesky/ google/ hn_scraper/` tells you nothing about which are connectors. At nine connectors that is fine; at fifty mixed with twenty functions it is not. + +Sibling top-level directories (`functions/` beside `auth/`) were also considered and are the weaker option. The bundler roots at `functions_dir`, so siblings force the root up to the repository, putting `notte.toml`, `.notte/` and `pyrightconfig.json` inside the Python package root. They also split shared code: `contract.py` would have to live in one tree and be reached from the other as `from ..functions._shared.http import`, which is worse than the depth it was avoiding. + +Depth-independence makes the choice the project's rather than the CLI's. + +--- + +## The shared contract should be a package, not files in the tree + +`contract.py` and `email_2fa.py` are not really *the user's* code. They are Notte's runtime contract — `LoginResult`, `VerifierResult`, the ~55 ordered failure phrases in `classify_login_failure`, and the mailbox polling loop. Copying them into every project is how the current design ends up with seven hand-copied 2FA loops and a regex that splices 271 lines into every artifact. + +The alternative is an importable package the runner ships, so a connector reads: + +```python +from notte_managed_auth import LoginResult, classify_login_failure +from notte_managed_auth.email import read_verification_code +``` + +**This works, and the part that looked like it would block it does not.** Three facts, checked rather than assumed: + +| | | +|---|---| +| The server's contract check | only `parsed.variables == ["session_id"]` (`managed_auth/service.py:1396`). Nothing about the return type | +| `response_format` | connectors never send it, so `check_run_returns_pydantic_model`'s "declared in the same file" rule never applies to them | +| managed-auth's own check | a string comparison that the annotation reads literally `LoginResult` (`deploy.py:379`) — which an imported name satisfies | + +So `contract.py` does not need flattening at all. It becomes an ordinary allowlisted import that survives into the artifact as an import, exactly like `pydantic`. + +What that changes: + +- **The 271-line splice disappears**, and with it the coupling where editing one shared file changes all nine bundle hashes. +- **The seven hand-copied 2FA loops disappear** without needing `_shared` to hold them. +- **`_shared/` gets much lighter**, which incidentally weakens the last argument against grouping directories — there is less left to reach across a dot. +- **Connectors become nearly single-file**, which is what they looked like before the contract was forced inline. + +The requirement is that the package ships in the runner image and appears in its allowlist. That is now *checkable* rather than assumed: `GET /functions/health` lists what the runner has, and today it has no auth package — `google, gspread, httpcloak, httpx, litellm, loguru, notte, notte_agent, notte_browser, notte_core, notte_sdk, playwright, pydantic, requests, typing_extensions`. Adding one would show up there, versioned, with its install source. + +### Timing + +**Not yet.** The library is worth extracting once the shape has stopped moving, and the way to find that out is to keep building connectors against it internally. `classify_login_failure`'s phrase list in particular is still growing, annotated `(observed)` as each one is read off a live site — a published package would freeze an interface that is still learning. + +The natural sequence: keep `_shared/contract.py` in-tree while connectors are built, watch which parts stop changing, and extract when the churn stops. A month or two of real use is a reasonable read. Documenting it as a library is what makes third-party connectors possible at all, so it is a prerequisite for the customer-facing question below, not a parallel track. + +--- + +## Metadata + +`connectors/.json` disappears into `notte.toml`: + +```toml +[connectors.bluesky] +name = "Bluesky" +domain = "bsky.app" +category = "Social" +color = "#0085ff" +description = "Decentralized social" +method = "Email & password" +allowed_domains = [] +supports_totp = false +proxy_country = "us" + +login = { name = "Bluesky managed login", description = "Signs in with the connection's username and password." } +verifier = { name = "Bluesky login verifier", description = "Read-only check for the authenticated settings link." } +``` + +Three fields from today's manifest are deliberately absent: + +- **`slug`** — it is the directory name. The old format required them to match and enforced it in code. +- **`login.path` / `verifier.path`** — the layout says where they are. +- **`revision`** — see below. + +--- + +## Revisions become derived + +Today `revision` is a hand-maintained integer, and `check_revision_bumps.py` (122 lines) exists to catch the case where someone edits a connector and forgets to increment it. It also encodes a tax the current design cannot avoid: because `contract.py` is spliced into every bundle, **editing one shared file means editing all nine manifests**, and the script special-cases exactly that. + +The lockfile already stores `source_sha256` per environment. So: + +> **The revision is the count of times the bundle hash has changed**, tracked in `notte.lock.json` and incremented by `deploy` when it moves. + +That deletes `check_revision_bumps.py` outright, removes the shared-file tax, and keeps the server contract unchanged — it still receives a monotonically increasing integer, and still refuses a stale or same-revision-different-content import. + +A `--revision` override stays for the rare case of adopting an existing connector whose upstream revision is already ahead. + +--- + +## Deploy + +`deploy.py`'s import flow is the good part of it and should move into the CLI rather than be discarded: + +1. Build both roles, plus the template block from `notte.toml`. +2. `POST /managed-auth/templates/import?dry_run=true` — returns `action`, `metadata_changes[]`, `login_changed`, `verifier_changed`, and `target_state_sha256`. +3. Show that diff and confirm, exactly as `notte stack deploy` already does for functions. +4. Apply with `expected_target_state_sha256` from the preview — optimistic concurrency, unchanged. + +Two properties worth preserving explicitly, because both were learned the hard way: + +- **The pair is transactional.** A connector whose login updates and whose verifier fails is worse than one that did not deploy, and the server already flips both inside one transaction. The CLI must not split them. +- **Function ids are never rotated on update.** Run rows and console links point at them. + +`notte stack deploy` grows no new flags: a connector is just another unit in the plan, printed as one line with its two roles. + +--- + +## The per-role contract + +A function's contract is `run()` returning a `BaseModel` declared in the same file. A connector role is stricter, and the server enforces it: + +| | Requirement | +|---|---| +| Parameters | exactly `session_id`, no more, no `*args`/`**kwargs` | +| Return | literally `LoginResult` for login, `VerifierResult` for verifier | +| Body | no module-level `run()` call — that is the shape of an exploratory recording | + +`ScriptValidator.parse_script` already rejects a script whose module-level variables are not exactly `["session_id"]`. The return-type check is `managed-auth`'s own, in `check_contract()`, and it is an AST check the CLI can run against the artifact in the same pass as everything else. + +**This is the point at which flattening earns itself here.** The annotation must be literally `LoginResult`, and after inlining `contract.py` that class *is* in the same file — which is exactly what the API's "declared in the same file" rule wants, and what the regex splice was faking. + +--- + +## What this deletes + +| | Today | After | +|---|---|---| +| `scripts/deploy.py` | 996 lines | the template-import call only, or nothing if it moves into the CLI | +| `scripts/check_revision_bumps.py` | 122 lines | gone — revisions are derived | +| `Makefile` | 113 lines | gone — `notte stack` has subcommands | +| `connectors/*.json` | 9 files | `[connectors.*]` in `notte.toml` | +| `contract.py` regex splice | `inline_contract()` | an ordinary import | +| the 2FA polling loop | **hand-copied into 7 login files** | imported — from `_shared/` now, from the package later | + +That last row is the clearest sign the current model is wrong. `login/email_2fa.py` exists and is imported by exactly two *undeployed* recordings, because a deployed connector is a single file and cannot import it. Seven shipped connectors carry a copy instead. + +--- + +## The customer-facing question + +The user-facing goal is that **customers ship their own authenticated connectors**, not just us. The machinery already supports it; the gates do not: + +1. **`_require_connector_organization()`** hard-codes one org UUID. It needs a customer-scoped equivalent, and a decision about whether customer connectors are private to their org or can be published. +2. **The managed-auth router is `include_in_schema=False`** (`main.py:719`), so it is absent from the OpenAPI spec and therefore from the generated Go client. Nothing here can be built until that flips or the client is hand-written. +3. **Templates are a global catalog** with a `slug` unique across all orgs. Customer connectors need either namespacing or a separate scope, or the first customer to claim `shopify` takes it. + +Worth deciding early, because it changes whether `[connectors.*]` describes a *catalog entry* or a *private connector*, and those want different metadata. + +--- + +## Open questions + +1. **Where does `{{MAILBOX_ID}}` templating live?** The API substitutes `{{TARGET_URL}}`, `{{DOMAIN}}`, `{{MAILBOX_ID}}`, `{{PROFILE_ID}}`, `{{VAULT_ID}}` — but only for custom connections; a catalog connector ships the literal string and relies on a backend fallback. Should the CLI know about these at all, or is a placeholder just text it passes through? +2. **Should `notte stack check` run the connector contract?** It is a different shape from the function contract, so either `check` grows a per-kind rule or connectors get their own validation pass. +3. **Do connectors need `notte stack dev`?** Driving a real login locally is exactly what `scripts/drive_login.py` does today, and it is how selectors and refusal phrases get discovered. That is a strong argument for `dev` being connector-shaped first. +4. **Does the smoke suite move too?** `scripts/smoke.py` (1,000 lines) drives real logins against a live environment with a shared test account, a mailbox mutex and a long-lived connection per environment. It is genuinely managed-auth-specific and probably stays — but it is the other half of the workflow. + +--- + +## What I would not do + +**Do not model connections.** A connection is one workspace's instance of a connector: real credentials, a browser profile, a vault, and a status that only the runtime can know. Creating one runs a real browser login and spends money. RFC 0001 already puts it on the imperative side, and nothing here changes that. diff --git a/go.mod b/go.mod index c01093f..cf79a96 100644 --- a/go.mod +++ b/go.mod @@ -2,11 +2,18 @@ module github.com/nottelabs/notte-cli go 1.25.5 -require github.com/oapi-codegen/runtime v1.1.2 +require ( + github.com/99designs/keyring v1.2.2 + github.com/BurntSushi/toml v1.6.0 + github.com/muesli/termenv v0.16.0 + github.com/oapi-codegen/runtime v1.1.2 + github.com/spf13/cobra v1.8.0 + github.com/spf13/pflag v1.0.5 + golang.org/x/term v0.3.0 +) require ( github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 // indirect - github.com/99designs/keyring v1.2.2 // indirect github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect github.com/danieljoos/wincred v1.1.2 // indirect @@ -18,10 +25,6 @@ require ( github.com/lucasb-eyer/go-colorful v1.2.0 // indirect github.com/mattn/go-isatty v0.0.20 // indirect github.com/mtibben/percent v0.2.1 // indirect - github.com/muesli/termenv v0.16.0 // indirect github.com/rivo/uniseg v0.4.7 // indirect - github.com/spf13/cobra v1.8.0 // indirect - github.com/spf13/pflag v1.0.5 // indirect golang.org/x/sys v0.30.0 // indirect - golang.org/x/term v0.3.0 // indirect ) diff --git a/go.sum b/go.sum index 0d61342..f8e79c3 100644 --- a/go.sum +++ b/go.sum @@ -2,6 +2,8 @@ github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 h1:/vQbFIOMb github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4/go.mod h1:hN7oaIRCjzsZ2dE+yG5k+rsdt3qcwykqK6HVGcKwsw4= github.com/99designs/keyring v1.2.2 h1:pZd3neh/EmUzWONb35LxQfvuY7kiSXAq3HQd97+XBn0= github.com/99designs/keyring v1.2.2/go.mod h1:wes/FrByc8j7lFOAGLGSNEg8f/PaI3cgTBqhFkHUrPk= +github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk= +github.com/BurntSushi/toml v1.6.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho= github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= github.com/apapsch/go-jsonmerge/v2 v2.0.0 h1:axGnT1gRIfimI7gJifB699GoE/oq+F2MU7Dml6nw9rQ= github.com/apapsch/go-jsonmerge/v2 v2.0.0/go.mod h1:lvDnEdqiQrp0O42VQGgmlKpxL1AP2+08jFMw88y4klk= @@ -26,6 +28,7 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d/go.mod h1:2PavIy+JPciBPrBUjwbNvtwB6RQlve+hkpll6QSNmOE= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/lucasb-eyer/go-colorful v1.2.0 h1:1nnpGOrhyZZuNyfu1QjKiUICQ74+3FNCN69Aj6K7nkY= github.com/lucasb-eyer/go-colorful v1.2.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0= @@ -35,6 +38,7 @@ github.com/mtibben/percent v0.2.1 h1:5gssi8Nqo8QU/r2pynCm+hBQHpkB/uNK7BJCFogWdzs github.com/mtibben/percent v0.2.1/go.mod h1:KG9uO+SZkUp+VkRHsCdYQV3XSZrrSpR3O9ibNBTZrns= github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc= github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk= +github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs= github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= github.com/oapi-codegen/runtime v1.1.2 h1:P2+CubHq8fO4Q6fV1tqDBZHCwpVpvPg7oKiYzQgXIyI= github.com/oapi-codegen/runtime v1.1.2/go.mod h1:SK9X900oXmPWilYR5/WKPzt3Kqxn/uS/+lbpREv+eCg= @@ -49,6 +53,8 @@ github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spkg/bom v0.0.0-20160624110644-59b7046e48ad/go.mod h1:qLr4V1qq6nMqFKkMo8ZTx3f+BZEkzsRUY10Xsm2mwU0= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.3.0 h1:NGXK3lHquSN08v5vWalVI/L8XU9hdzE/G6xsrze47As= +github.com/stretchr/objx v0.3.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= @@ -60,6 +66,7 @@ golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/term v0.3.0 h1:qoo4akIqOcDME5bhc/NgxUdovd6BSS2uMsVjB56q1xI= golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b h1:QRR6H1YWRnHb4Y/HeNFCTJLFVxaq6wH4YuVdsUOr75U= gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= diff --git a/internal/auth/env.go b/internal/auth/env.go index b5b9c27..566e16b 100644 --- a/internal/auth/env.go +++ b/internal/auth/env.go @@ -35,6 +35,22 @@ func ResolveEnvLabel(apiURL string) string { return host } +// IsKnownEnvHost reports whether an API URL maps to one of the canonical +// environments, as opposed to a hostname used verbatim as its own label. +// +// The difference matters to callers that want to detect a contradiction: a URL +// that is definitively prod disagreeing with a section called staging is a +// misconfiguration, while a self-hosted or preview host simply has no +// canonical label to disagree with. +func IsKnownEnvHost(apiURL string) bool { + u, err := url.Parse(apiURL) + if err != nil || u.Host == "" { + return false + } + _, ok := hostToEnvLabel[u.Hostname()] + return ok +} + // KeyringKeyForEnv returns the env-qualified keyring key for the given label. func KeyringKeyForEnv(envLabel string) string { return KeyringKey + ":" + envLabel diff --git a/internal/auth/keyring.go b/internal/auth/keyring.go index e3a45b6..9b10898 100644 --- a/internal/auth/keyring.go +++ b/internal/auth/keyring.go @@ -101,7 +101,16 @@ func deleteFromSystemKeyring(key string) error { // On first read after upgrade, it falls back to the legacy "api_key" entry and // auto-migrates it to the env-qualified key. func GetKeyringAPIKey() (string, error) { - envLabel := ResolveEnvLabel(GetCurrentAPIURL()) + return GetKeyringAPIKeyForEnv(ResolveEnvLabel(GetCurrentAPIURL())) +} + +// GetKeyringAPIKeyForEnv retrieves the key stored for a named environment. +// +// Callers that already know which environment they are targeting must use this +// rather than GetKeyringAPIKey, which infers the label from whatever +// NOTTE_API_URL happens to be. Inferring it is how a credential for one +// environment reaches another. +func GetKeyringAPIKeyForEnv(envLabel string) (string, error) { envKey := KeyringKeyForEnv(envLabel) // Try env-qualified key first diff --git a/internal/bundle/bundle.go b/internal/bundle/bundle.go new file mode 100644 index 0000000..4fb1c0e --- /dev/null +++ b/internal/bundle/bundle.go @@ -0,0 +1,321 @@ +// Package bundle flattens a Python package into the single file the Notte +// functions API accepts. +// +// The API rejects every form of local import: `from . import x` is refused +// outright by the upload validator, and `from .util import x` then fails the +// import allowlist by name — at run time as well, because the Lambda runner +// keeps __import__ bound to safe_import even though it disables the +// RestrictedPython AST policy. So a multi-module function has to become one +// module before it is uploaded, and it has to do so statically: the sys.modules +// prelude every off-the-shelf bundler emits would need imports that are +// themselves blocked. +// +// Flattening is concatenation in dependency order with relative imports +// removed. Names that would collide are an error rather than something to +// mangle, which is what keeps a full Python parser out of this package: nothing +// is ever rewritten, so nothing has to be understood well enough to rewrite. +// It also keeps the artifact readable, which matters because the artifact is +// what the console shows and what tracebacks point at. +// +// This package deliberately does not check imports against the runtime +// allowlist. It used to, by vendoring a copy of the server's lists and a stdlib +// set generated from a pinned CPython — and both drifted within a week of being +// written. `notte stack` requires Python, so the real ScriptValidator runs +// against the artifact instead, and a copy that can disagree with it is worse +// than no copy at all. +package bundle + +import ( + "fmt" + "io/fs" + "path" + "strings" +) + +// Result is a successful bundle. +type Result struct { + // Code is the artifact: valid Python defining exactly the names the + // entrypoint's module would have had. + Code string + // Sources are the contributing files in emission order, entrypoint last. + Sources []string + // SourceSHA256 covers the set of inputs. It answers "does this need + // deploying?" and is stable across bundler changes that do not change the + // sources. + SourceSHA256 string + // ArtifactSHA256 covers Code. It answers "what changed upstream?" and is + // what a remote diff compares against. + ArtifactSHA256 string + // Map resolves an artifact line back to the file and line it came from. + Map *SourceMap +} + +// Error is a bundling failure carrying the location that caused it. +type Error struct { + Path string + Line int + Msg string + Hint string +} + +func (e *Error) Error() string { + loc := e.Path + if e.Line > 0 { + loc = fmt.Sprintf("%s:%d", e.Path, e.Line) + } + if e.Hint != "" { + return fmt.Sprintf("%s: %s — %s", loc, e.Msg, e.Hint) + } + return fmt.Sprintf("%s: %s", loc, e.Msg) +} + +func errAt(p string, line int, msg, hint string) *Error { + return &Error{Path: p, Line: line, Msg: msg, Hint: hint} +} + +// Options tunes the emitted artifact. +type Options struct { + // Header is prepended verbatim. Callers pass provenance here; the bundler + // does not invent one so that output stays a pure function of the input. + Header string +} + +// module is one parsed source file. +type module struct { + path string // slash-separated, relative to the package root + src string + lines []string + stmts []Stmt + imports []Import + deps []string // resolved paths of relative imports, in source order +} + +// Bundle flattens the package reachable from entrypoint into one file. +// +// fsys is rooted at the functions directory, and entrypoint is a path within +// it such as "amazon_search/main.py". Only relative imports are followed; +// absolute ones are hoisted and left for the allowlist check. +func Bundle(fsys fs.FS, entrypoint string, opts Options) (*Result, error) { + mods := map[string]*module{} + order, err := collect(fsys, entrypoint, mods, nil) + if err != nil { + return nil, err + } + if err := checkCollisions(order, mods); err != nil { + return nil, err + } + return emit(order, mods, opts) +} + +// collect loads the entrypoint and everything it reaches, returning modules in +// dependency-first order. stack carries the current resolution path so a cycle +// can be reported as the cycle it is rather than as a stack overflow. +func collect(fsys fs.FS, p string, mods map[string]*module, stack []string) ([]string, error) { + for i, s := range stack { + if s == p { + return nil, errAt(p, 0, "import cycle: "+strings.Join(append(stack[i:], p), " -> "), + "break the cycle by moving the shared names into their own module") + } + } + if _, done := mods[p]; done { + return nil, nil + } + + m, err := load(fsys, p) + if err != nil { + return nil, err + } + mods[p] = m + + var order []string + for _, dep := range m.deps { + sub, err := collect(fsys, dep, mods, append(stack, p)) + if err != nil { + return nil, err + } + order = append(order, sub...) + } + return append(order, p), nil +} + +func load(fsys fs.FS, p string) (*module, error) { + raw, err := fs.ReadFile(fsys, p) + if err != nil { + // A relative import naming a package rather than a module lands here, + // because the flattener resolves every target to a single .py file. + // `from .pkg import sub` where sub is pkg/sub.py has no single file to + // inline, so it is refused rather than emitted as a binding that does + // not exist. + if dir := strings.TrimSuffix(p, ".py"); dirExists(fsys, dir) { + return nil, errAt(p, 0, "cannot inline a package", + fmt.Sprintf("%s is a directory; import the module directly, e.g. from .%s. import ", + dir, path.Base(dir))) + } + return nil, errAt(p, 0, "cannot read module", "") + } + src := string(raw) + m := &module{ + path: p, + src: src, + lines: strings.Split(src, "\n"), + stmts: Scan(src), + } + + for _, s := range m.stmts { + im, ok := ParseImport(s) + if !ok { + continue + } + if !s.TopLevel() { + // A relative import inside a function body would have to be + // rewritten in place, and rewriting is the thing this design + // avoids. Absolute ones are harmless where they are. + if im.Kind == ImportRelative { + return nil, errAt(p, s.StartLine, "relative import inside an indented block", + "move it to the top of the file") + } + continue + } + // Emission drops or rewrites whole lines, so an import sharing a line + // with anything else would take its neighbour with it. Rejecting is + // cheap for the author to fix and PEP 8 asks for it anyway; a sub-line + // rewriter is a lot of machinery for `import os; x = 1`. + if sharesLine(m.stmts, s) { + return nil, errAt(p, s.StartLine, "import shares a line with another statement", + "put each import on its own line") + } + m.imports = append(m.imports, im) + + switch im.Kind { + case ImportFuture: + if len(im.Names) != 1 || im.Names[0].Name != "annotations" { + return nil, errAt(p, s.StartLine, "only 'from __future__ import annotations' is allowed", "") + } + case ImportRelative: + if im.Star { + return nil, errAt(p, s.StartLine, "star imports cannot be flattened", + "import the names explicitly") + } + if im.Module == "" { + return nil, errAt(p, s.StartLine, "'from . import ' cannot be flattened", + fmt.Sprintf("use 'from .%s import ' instead", firstName(im))) + } + dep, err := resolve(p, im) + if err != nil { + return nil, err + } + m.deps = append(m.deps, dep) + } + } + return m, nil +} + +// sharesLine reports whether any other statement begins inside this one's span. +func sharesLine(stmts []Stmt, target Stmt) bool { + for _, other := range stmts { + if other == target { + continue + } + if other.StartLine >= target.StartLine && other.StartLine <= target.EndLine { + return true + } + } + return false +} + +// dirExists reports whether a package directory sits where a module was +// expected, which is the difference between a typo and an unsupported import. +func dirExists(fsys fs.FS, dir string) bool { + info, err := fs.Stat(fsys, dir) + return err == nil && info.IsDir() +} + +func firstName(im Import) string { + if len(im.Names) > 0 { + return im.Names[0].Name + } + return "mod" +} + +// resolve turns a relative import into a path within the package root. +// +// Level 1 is the importing module's own package, level 2 its parent, and so +// on — the same rule Python uses, so a layout that resolves here resolves in +// the editor too. +func resolve(from string, im Import) (string, error) { + pkg := path.Dir(from) + if pkg == "." { + pkg = "" + } + for i := 1; i < im.Level; i++ { + if pkg == "" { + return "", errAt(from, im.Stmt.StartLine, + "relative import goes above the functions directory", "") + } + pkg = path.Dir(pkg) + if pkg == "." { + pkg = "" + } + } + rel := strings.ReplaceAll(im.Module, ".", "/") + ".py" + if pkg == "" { + return rel, nil + } + return pkg + "/" + rel, nil +} + +// checkCollisions rejects two modules defining the same module-level name. +// +// Concatenation makes the later definition win silently, so this is reported +// rather than resolved. Aliases count: `from .parse import clean as fetch` +// introduces `fetch` exactly as a def would. +func checkCollisions(order []string, mods map[string]*module) error { + type owner struct { + path string + line int + } + seen := map[string]owner{} + + for _, p := range order { + m := mods[p] + for _, s := range m.stmts { + if !s.TopLevel() { + continue + } + for _, name := range bindingsOwnedBy(s) { + if prev, dup := seen[name]; dup && prev.path != p { + return errAt(p, s.StartLine, + fmt.Sprintf("%s:%d and %s:%d both define %q", prev.path, prev.line, p, s.StartLine, name), + "rename one of them") + } + seen[name] = owner{path: p, line: s.StartLine} + } + } + } + return nil +} + +// bindingsOwnedBy are the names a statement introduces into the flattened +// namespace. +// +// An unaliased relative import is excluded: `from .parse import clean` refers +// to the very definition that will be concatenated in, so counting it would +// report every shared helper as colliding with itself. Absolute imports are +// excluded because they are hoisted and deduplicated, so four modules importing +// requests produce one binding rather than four. +func bindingsOwnedBy(s Stmt) []string { + im, isImport := ParseImport(s) + if !isImport { + return TopLevelBindings(s) + } + if im.Kind != ImportRelative { + return nil + } + var out []string + for _, n := range im.Names { + if n.Alias != "" { + out = append(out, n.Alias) + } + } + return out +} diff --git a/internal/bundle/corpus_test.go b/internal/bundle/corpus_test.go new file mode 100644 index 0000000..ae24969 --- /dev/null +++ b/internal/bundle/corpus_test.go @@ -0,0 +1,73 @@ +package bundle + +import ( + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// TestRealCorpus bundles every .py file in a real functions tree and checks +// that it survives: the bundler accepts it, the artifact compiles, and no +// top-level definition is lost. +// +// Opt-in via NOTTE_CORPUS because it needs a checkout and takes a couple of +// minutes. It is the highest-value test here by some distance — a hand-written +// suite covers the cases its author thought of, while anything-api/marketplace +// is 2.5k files of production Python written by other people and by an agent, +// full of constructs nobody would think to write down. Point it at a +// marketplace checkout before trusting a change to the scanner: +// +// NOTTE_CORPUS=~/path/to/anything-api/marketplace go test ./internal/bundle -run TestRealCorpus +func TestRealCorpus(t *testing.T) { + root := os.Getenv("NOTTE_CORPUS") + if root == "" { + t.Skip("set NOTTE_CORPUS") + } + var files []string + _ = filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error { + if err == nil && !d.IsDir() && strings.HasSuffix(p, ".py") { + files = append(files, p) + } + return nil + }) + t.Logf("found %d files", len(files)) + + py, _ := exec.LookPath("python3") + var bundleErrs, compileErrs, mismatched int + for _, f := range files { + rel, _ := filepath.Rel(root, f) + res, err := Bundle(os.DirFS(root), filepath.ToSlash(rel), Options{}) + if err != nil { + bundleErrs++ + if bundleErrs <= 8 { + t.Logf("BUNDLE ERR %s: %v", rel, err) + } + continue + } + // Zero relative imports in this corpus, so every def must survive. + orig, _ := os.ReadFile(f) + for _, line := range strings.Split(string(orig), "\n") { + if strings.HasPrefix(line, "def ") && !strings.Contains(res.Code, line) { + mismatched++ + if mismatched <= 5 { + t.Logf("LOST DEF %s: %q", rel, line) + } + break + } + } + if py != "" { + tmp := filepath.Join(t.TempDir(), "a.py") + _ = os.WriteFile(tmp, []byte(res.Code), 0o644) + if out, err := exec.Command(py, "-m", "py_compile", tmp).CombinedOutput(); err != nil { + compileErrs++ + if compileErrs <= 8 { + t.Logf("COMPILE ERR %s: %s", rel, out) + } + } + } + } + t.Logf("RESULT files=%d bundleErrs=%d compileErrs=%d lostDefs=%d", + len(files), bundleErrs, compileErrs, mismatched) +} diff --git a/internal/bundle/emit.go b/internal/bundle/emit.go new file mode 100644 index 0000000..69e2844 --- /dev/null +++ b/internal/bundle/emit.go @@ -0,0 +1,263 @@ +package bundle + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "sort" + "strings" +) + +// SourceMap maps 1-based artifact lines back to where they came from. +// +// Without it a traceback points into a concatenated file and the reader has to +// guess which module line 612 belonged to. Entries are sorted by ArtifactLine +// and cover every emitted source line; generated lines (the header, hoisted +// imports, alias assignments) have an empty Path. +type SourceMap struct { + Entries []MapEntry `json:"entries"` +} + +// MapEntry is one contiguous run of artifact lines from one source file. +type MapEntry struct { + ArtifactLine int `json:"artifact_line"` + Path string `json:"path"` + SourceLine int `json:"source_line"` + Count int `json:"count"` +} + +// Lookup resolves an artifact line to its source location. ok is false for +// generated lines, which have no source. +func (sm *SourceMap) Lookup(artifactLine int) (path string, line int, ok bool) { + i := sort.Search(len(sm.Entries), func(i int) bool { + return sm.Entries[i].ArtifactLine > artifactLine + }) - 1 + if i < 0 { + return "", 0, false + } + e := sm.Entries[i] + if e.Path == "" || artifactLine >= e.ArtifactLine+e.Count { + return "", 0, false + } + return e.Path, e.SourceLine + (artifactLine - e.ArtifactLine), true +} + +// writer accumulates artifact lines and the map alongside them. +type writer struct { + lines []string + entries []MapEntry +} + +// generated appends lines with no source counterpart. +func (w *writer) generated(lines ...string) { + w.lines = append(w.lines, lines...) +} + +// fromSource appends one source line and records where it came from, extending +// the previous run when it is contiguous. +func (w *writer) fromSource(path string, srcLine int, text string) { + artifactLine := len(w.lines) + 1 + if n := len(w.entries); n > 0 { + last := &w.entries[n-1] + if last.Path == path && last.ArtifactLine+last.Count == artifactLine && last.SourceLine+last.Count == srcLine { + last.Count++ + w.lines = append(w.lines, text) + return + } + } + w.entries = append(w.entries, MapEntry{ + ArtifactLine: artifactLine, + Path: path, + SourceLine: srcLine, + Count: 1, + }) + w.lines = append(w.lines, text) +} + +// emit concatenates the modules in dependency order. +// +// Per module: relative imports become alias assignments or vanish, absolute +// imports are lifted to a single deduplicated block at the top, and every other +// line is copied verbatim so the artifact still reads like the sources. +func emit(order []string, mods map[string]*module, opts Options) (*Result, error) { + w := &writer{} + + if opts.Header != "" { + for _, line := range strings.Split(strings.TrimRight(opts.Header, "\n"), "\n") { + w.generated(line) + } + w.generated("") + } + + // `from __future__` must precede every other statement, so it is emitted + // once here no matter which module asked for it. + if anyFutureAnnotations(order, mods) { + w.generated("from __future__ import annotations", "") + } + + if hoisted := hoistImports(order, mods); len(hoisted) > 0 { + w.generated(hoisted...) + w.generated("") + } + + for _, p := range order { + m := mods[p] + drop, replace := rewritePlan(m) + + w.generated("# ── " + p + " ──") + started := false + for i, text := range m.lines { + lineNo := i + 1 + if lineNo == len(m.lines) && text == "" { + continue // trailing newline artefact of the split + } + if drop[lineNo] { + continue + } + if aliases, ok := replace[lineNo]; ok { + w.generated(aliases...) + started = true + continue + } + // Removing a module's imports strands the blank lines that + // separated them from the first definition, under the header + // comment. Skip forward to real content. + if !started && strings.TrimSpace(text) == "" { + continue + } + started = true + w.fromSource(p, lineNo, text) + } + w.generated("") + } + + code := strings.Join(w.lines, "\n") + if !strings.HasSuffix(code, "\n") { + code += "\n" + } + + return &Result{ + Code: code, + Sources: order, + SourceSHA256: sourceHash(order, mods), + ArtifactSHA256: sha256Hex(code), + Map: &SourceMap{Entries: w.entries}, + }, nil +} + +// rewritePlan decides, per physical line, what emission does with it. +// +// drop covers lines that leave entirely (hoisted absolute imports, __future__, +// unaliased relative imports). replace maps the first line of an aliased +// relative import to the assignments that preserve its bindings. +func rewritePlan(m *module) (drop map[int]bool, replace map[int][]string) { + drop = map[int]bool{} + replace = map[int][]string{} + + for _, im := range m.imports { + switch im.Kind { + case ImportAbsolute, ImportFrom, ImportFuture: + for l := im.Stmt.StartLine; l <= im.Stmt.EndLine; l++ { + drop[l] = true + } + case ImportRelative: + // The definition arrives by concatenation, so the unaliased name is + // already bound. An alias is not, and deleting the line without + // recreating it is a NameError at run time that nothing before + // production would catch. + var assigns []string + for _, n := range im.Names { + if n.Alias != "" { + assigns = append(assigns, fmt.Sprintf("%s = %s", n.Alias, n.Name)) + } + } + for l := im.Stmt.StartLine; l <= im.Stmt.EndLine; l++ { + drop[l] = true + } + if len(assigns) > 0 { + delete(drop, im.Stmt.StartLine) + replace[im.Stmt.StartLine] = assigns + } + } + } + return drop, replace +} + +func anyFutureAnnotations(order []string, mods map[string]*module) bool { + for _, p := range order { + for _, im := range mods[p].imports { + if im.Kind == ImportFuture { + return true + } + } + } + return false +} + +// hoistImports collects every absolute import, deduplicated and sorted. +// +// Sorting is what makes the artifact reproducible: the same sources must yield +// the same bytes, because ArtifactSHA256 drives the remote diff. +func hoistImports(order []string, mods map[string]*module) []string { + seen := map[string]bool{} + var plain, from []string + for _, p := range order { + for _, im := range mods[p].imports { + if im.Kind != ImportAbsolute && im.Kind != ImportFrom { + continue + } + text := normalizeImport(im) + if seen[text] { + continue + } + seen[text] = true + if im.Kind == ImportAbsolute { + plain = append(plain, text) + } else { + from = append(from, text) + } + } + } + sort.Strings(plain) + sort.Strings(from) + return append(plain, from...) +} + +// normalizeImport rebuilds an import from its parsed form so that two spellings +// of the same import deduplicate. +func normalizeImport(im Import) string { + names := make([]string, 0, len(im.Names)) + for _, n := range im.Names { + if n.Alias != "" { + names = append(names, n.Name+" as "+n.Alias) + } else { + names = append(names, n.Name) + } + } + sort.Strings(names) + if im.Kind == ImportAbsolute { + return "import " + strings.Join(names, ", ") + } + if im.Star { + return "from " + im.Module + " import *" + } + return "from " + im.Module + " import " + strings.Join(names, ", ") +} + +// sourceHash covers the inputs rather than the output, so it is stable when the +// bundler changes but the sources do not. +func sourceHash(order []string, mods map[string]*module) string { + paths := append([]string(nil), order...) + sort.Strings(paths) + h := sha256.New() + for _, p := range paths { + // hash.Hash.Write is documented never to return an error. + _, _ = fmt.Fprintf(h, "%s:%s\n", p, sha256Hex(mods[p].src)) + } + return hex.EncodeToString(h.Sum(nil)) +} + +func sha256Hex(s string) string { + sum := sha256.Sum256([]byte(s)) + return hex.EncodeToString(sum[:]) +} diff --git a/internal/bundle/errors_test.go b/internal/bundle/errors_test.go new file mode 100644 index 0000000..388622d --- /dev/null +++ b/internal/bundle/errors_test.go @@ -0,0 +1,232 @@ +package bundle + +import ( + "strings" + "testing" + "testing/fstest" +) + +// mapFS builds an in-memory package. Keys are paths under the functions +// directory; values are file bodies. +func mapFS(files map[string]string) fstest.MapFS { + fsys := fstest.MapFS{} + for p, body := range files { + fsys[p] = &fstest.MapFile{Data: []byte(body)} + } + return fsys +} + +// wantErr bundles and requires failure, returning the message. +func wantErr(t *testing.T, files map[string]string) string { + t.Helper() + res, err := Bundle(mapFS(files), "fn/main.py", Options{}) + if err == nil { + t.Fatalf("expected an error, got a bundle:\n%s", res.Code) + } + return err.Error() +} + +func mustContain(t *testing.T, got string, wants ...string) { + t.Helper() + for _, w := range wants { + if !strings.Contains(got, w) { + t.Fatalf("error %q does not mention %q", got, w) + } + } +} + +// Two modules defining the same name: concatenation would silently let the +// later one win. +func TestCollisionIsReportedWithBothLocations(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from .a import a\nfrom .b import b\n\n\ndef run():\n return a() + b()\n", + "fn/a.py": "def clean(s):\n return s\n\n\ndef a():\n return clean(1)\n", + "fn/b.py": "def clean(s):\n return s\n\n\ndef b():\n return clean(2)\n", + }) + mustContain(t, msg, "clean", "fn/a.py", "fn/b.py", "rename") +} + +// An alias occupies a name exactly as a definition does. +func TestAliasCollidesWithDefinition(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from .a import helper as fetch\nfrom .b import fetch as other\n\n\ndef run():\n return fetch, other\n", + "fn/a.py": "def helper():\n return 1\n", + "fn/b.py": "def fetch():\n return 2\n", + }) + mustContain(t, msg, "fetch") +} + +// The same name imported unaliased by two modules is one definition, not a +// collision. A naive binding count reports every shared helper as conflicting. +func TestSharedHelperImportedTwiceIsNotACollision(t *testing.T) { + res, err := Bundle(mapFS(map[string]string{ + "fn/main.py": "from .a import a\nfrom .b import b\n\n\ndef run():\n return a() + b()\n", + "fn/a.py": "from .shared import shared\n\n\ndef a():\n return shared()\n", + "fn/b.py": "from .shared import shared\n\n\ndef b():\n return shared()\n", + "fn/shared.py": "def shared():\n return 1\n", + }), "fn/main.py", Options{}) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if n := strings.Count(res.Code, "def shared()"); n != 1 { + t.Fatalf("shared emitted %d times, want 1", n) + } +} + +func TestImportCycleIsReported(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from .a import a\n\n\ndef run():\n return a()\n", + "fn/a.py": "from .b import b\n\n\ndef a():\n return b()\n", + "fn/b.py": "from .a import a\n\n\ndef b():\n return a()\n", + }) + mustContain(t, msg, "cycle", "fn/a.py", "fn/b.py") +} + +func TestSelfImportIsReportedAsCycle(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from .main import run\n\n\ndef run():\n return 1\n", + }) + mustContain(t, msg, "cycle") +} + +func TestStarImportIsRejected(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from .helpers import *\n\n\ndef run():\n return 1\n", + "fn/helpers.py": "def h():\n return 1\n", + }) + mustContain(t, msg, "star", "explicitly") +} + +// `from . import mod` then `mod.f()` needs the module to survive as an object, +// which flattening cannot provide. The message has to name the alternative. +func TestFromDotImportIsRejectedWithAFix(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from . import parse\n\n\ndef run():\n return parse.clean(1)\n", + "fn/parse.py": "def clean(s):\n return s\n", + }) + mustContain(t, msg, "from .parse import") +} + +func TestRelativeImportInsideFunctionIsRejected(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "def run():\n from .parse import clean\n return clean(1)\n", + "fn/parse.py": "def clean(s):\n return s\n", + }) + mustContain(t, msg, "indented", "top of the file") +} + +// An absolute import inside a function is harmless where it is: it needs no +// rewriting, so there is no reason to reject it. +func TestAbsoluteImportInsideFunctionIsAllowed(t *testing.T) { + if _, err := Bundle(mapFS(map[string]string{ + "fn/main.py": "def run():\n import json\n return json.dumps({})\n", + }), "fn/main.py", Options{}); err != nil { + t.Fatalf("unexpected error: %v", err) + } +} + +func TestNonAnnotationsFutureImportIsRejected(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from __future__ import division\n\n\ndef run():\n return 1\n", + }) + mustContain(t, msg, "__future__") +} + +func TestMissingModuleIsReported(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from .nope import x\n\n\ndef run():\n return x\n", + }) + mustContain(t, msg, "fn/nope.py") +} + +// Climbing above the functions directory has no meaning; it must not silently +// resolve to something outside the tree. +func TestImportAboveRootIsReported(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from ...outside import x\n\n\ndef run():\n return x\n", + }) + mustContain(t, msg, "above the functions directory") +} + +func TestMissingEntrypointIsReported(t *testing.T) { + _, err := Bundle(mapFS(map[string]string{"fn/other.py": "x = 1\n"}), "fn/main.py", Options{}) + if err == nil { + t.Fatal("expected an error") + } + mustContain(t, err.Error(), "fn/main.py") +} + +// Errors carry a location so the message can be acted on without searching. +func TestErrorCarriesPathAndLine(t *testing.T) { + _, err := Bundle(mapFS(map[string]string{ + "fn/main.py": "import requests\n\nfrom .helpers import *\n", + "fn/helpers.py": "def h():\n return 1\n", + }), "fn/main.py", Options{}) + if err == nil { + t.Fatal("expected an error") + } + be, ok := err.(*Error) + if !ok { + t.Fatalf("error is %T, want *bundle.Error", err) + } + if be.Path != "fn/main.py" || be.Line != 3 { + t.Fatalf("location = %s:%d, want fn/main.py:3", be.Path, be.Line) + } +} + +// `import json; import re` used to parse as a single import of a module named +// "json;", drop the whole line, and never hoist re — a NameError from an +// artifact that compiled and passed upload validation. +func TestSemicolonSeparatedImportsAreRejected(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "import json; import re\n\n\ndef run():\n return json, re\n", + }) + mustContain(t, msg, "own line") +} + +func TestImportSharingALineWithCodeIsRejected(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "import json; x = 1\n\n\ndef run():\n return x\n", + }) + mustContain(t, msg, "own line") +} + +// Semicolons elsewhere are fine; only imports constrain the rewriter. +func TestSemicolonInOrdinaryCodeIsAllowed(t *testing.T) { + res, err := Bundle(mapFS(map[string]string{ + "fn/main.py": "A = 1; B = 2\n\n\ndef run():\n return A + B\n", + }), "fn/main.py", Options{}) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if !strings.Contains(res.Code, "A = 1; B = 2") { + t.Fatalf("line altered:\n%s", res.Code) + } +} + +// Collisions must be caught for names bound without an '=' too. +func TestClauseBoundNamesCollide(t *testing.T) { + for _, b := range []struct{ name, src string }{ + {"for", "for item in [1]:\n pass\n"}, + {"with", "with open(\"f\") as item:\n pass\n"}, + {"walrus", "if (item := 1):\n pass\n"}, + } { + t.Run(b.name, func(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from .a import a\nfrom .b import b\n\n\ndef run():\n return a() + b()\n", + "fn/a.py": b.src + "\n\ndef a():\n return 1\n", + "fn/b.py": "item = 2\n\n\ndef b():\n return item\n", + }) + mustContain(t, msg, "item") + }) + } +} + +func TestPackageImportIsRejectedWithAnExplanation(t *testing.T) { + msg := wantErr(t, map[string]string{ + "fn/main.py": "from .pkg import sub as alias\n\n\ndef run():\n return alias\n", + "fn/pkg/__init__.py": "", + "fn/pkg/sub.py": "def helper():\n return 1\n", + }) + mustContain(t, msg, "package", "from .pkg. import") +} diff --git a/internal/bundle/golden_test.go b/internal/bundle/golden_test.go new file mode 100644 index 0000000..1463abd --- /dev/null +++ b/internal/bundle/golden_test.go @@ -0,0 +1,190 @@ +package bundle + +import ( + "flag" + "os" + "path/filepath" + "strings" + "testing" +) + +// -update rewrites the golden files. Review the diff it produces: these files +// are the specification of what the bundler emits. +var update = flag.Bool("update", false, "rewrite golden files") + +// goldenCases are the fixtures under testdata//in, bundled from +// fn/main.py and compared against testdata//want.py. +var goldenCases = []string{ + "single-file", + "alias-preserved", + "topo-order", + "diamond", + "import-hoist-dedup", + "future-annotations", + "shared-parent", + "docstring-not-an-import", +} + +func bundleCase(t *testing.T, name string) *Result { + t.Helper() + res, err := Bundle(os.DirFS(filepath.Join("testdata", name, "in")), "fn/main.py", Options{}) + if err != nil { + t.Fatalf("%s: %v", name, err) + } + return res +} + +func TestGolden(t *testing.T) { + for _, name := range goldenCases { + t.Run(name, func(t *testing.T) { + res := bundleCase(t, name) + goldenPath := filepath.Join("testdata", name, "want.py") + + if *update { + if err := os.WriteFile(goldenPath, []byte(res.Code), 0o644); err != nil { + t.Fatal(err) + } + return + } + want, err := os.ReadFile(goldenPath) + if err != nil { + t.Fatalf("missing golden file (run: go test ./internal/bundle -update): %v", err) + } + if res.Code != string(want) { + t.Errorf("artifact differs from golden\n--- got ---\n%s\n--- want ---\n%s", res.Code, want) + } + }) + } +} + +// The artifact hash drives the remote diff, so identical inputs must produce +// identical bytes. Map iteration order is the obvious way for that to break. +func TestBundleIsDeterministic(t *testing.T) { + for _, name := range goldenCases { + t.Run(name, func(t *testing.T) { + first := bundleCase(t, name) + for i := 0; i < 8; i++ { + again := bundleCase(t, name) + if again.Code != first.Code { + t.Fatalf("run %d differs from run 0", i+1) + } + if again.ArtifactSHA256 != first.ArtifactSHA256 { + t.Fatalf("ArtifactSHA256 unstable: %s vs %s", again.ArtifactSHA256, first.ArtifactSHA256) + } + if again.SourceSHA256 != first.SourceSHA256 { + t.Fatalf("SourceSHA256 unstable") + } + } + }) + } +} + +// The whole point of the alias rule: the artifact must still bind pr. +func TestAliasBindingSurvivesInArtifact(t *testing.T) { + res := bundleCase(t, "alias-preserved") + if !strings.Contains(res.Code, "pr = parse_rows") { + t.Fatalf("alias assignment missing:\n%s", res.Code) + } + if strings.Contains(res.Code, "from .parse import") { + t.Fatalf("relative import survived into the artifact:\n%s", res.Code) + } + // clean is unaliased, so it needs no assignment — the definition carries it. + if strings.Contains(res.Code, "clean = clean") { + t.Fatalf("emitted a redundant self-assignment:\n%s", res.Code) + } +} + +func TestTopologicalOrder(t *testing.T) { + res := bundleCase(t, "topo-order") + if want := []string{"fn/base.py", "fn/mid.py", "fn/main.py"}; !equal(res.Sources, want) { + t.Fatalf("sources = %v, want %v", res.Sources, want) + } + base := strings.Index(res.Code, "def base()") + middle := strings.Index(res.Code, "def middle()") + run := strings.Index(res.Code, "def run()") + if base >= middle || middle >= run { + t.Fatalf("definitions out of dependency order: base=%d middle=%d run=%d", base, middle, run) + } +} + +// A module reached by two paths must appear once; twice would be a redefinition +// and, for a class, a different object than the one already captured. +func TestDiamondEmitsSharedModuleOnce(t *testing.T) { + res := bundleCase(t, "diamond") + if n := strings.Count(res.Code, "def shared()"); n != 1 { + t.Fatalf("shared emitted %d times, want 1:\n%s", n, res.Code) + } + if n := countString(res.Sources, "fn/shared.py"); n != 1 { + t.Fatalf("shared listed %d times in Sources", n) + } +} + +func TestHoistedImportsAreDeduplicated(t *testing.T) { + res := bundleCase(t, "import-hoist-dedup") + if n := strings.Count(res.Code, "import requests"); n != 1 { + t.Fatalf("`import requests` appears %d times, want 1:\n%s", n, res.Code) + } + if n := strings.Count(res.Code, "from pydantic import BaseModel"); n != 1 { + t.Fatalf("pydantic import appears %d times, want 1:\n%s", n, res.Code) + } +} + +// Only one __future__ import, and it has to be the first statement or Python +// raises SyntaxError. +func TestFutureAnnotationsEmittedOnceAndFirst(t *testing.T) { + res := bundleCase(t, "future-annotations") + if n := strings.Count(res.Code, "from __future__ import annotations"); n != 1 { + t.Fatalf("appears %d times, want 1:\n%s", n, res.Code) + } + for _, line := range strings.Split(res.Code, "\n") { + if strings.TrimSpace(line) == "" || strings.HasPrefix(line, "#") { + continue + } + if line != "from __future__ import annotations" { + t.Fatalf("first real statement is %q, want the __future__ import", line) + } + break + } +} + +func TestParentPackageImportResolves(t *testing.T) { + res := bundleCase(t, "shared-parent") + if want := []string{"_shared/http.py", "fn/main.py"}; !equal(res.Sources, want) { + t.Fatalf("sources = %v, want %v", res.Sources, want) + } + if !strings.Contains(res.Code, "def fetch_json(q):") { + t.Fatalf("shared helper not inlined:\n%s", res.Code) + } +} + +// A single-file function has nothing to flatten; it must survive intact. +func TestSingleFileFunctionIsUnchangedApartFromHoisting(t *testing.T) { + res := bundleCase(t, "single-file") + if len(res.Sources) != 1 { + t.Fatalf("sources = %v", res.Sources) + } + if !strings.Contains(res.Code, `return requests.get("https://x.test").text`) { + t.Fatalf("body altered:\n%s", res.Code) + } +} + +// Import-looking text inside a docstring must not be followed. +func TestDocstringImportsAreNotResolved(t *testing.T) { + res := bundleCase(t, "docstring-not-an-import") + if want := []string{"fn/real.py", "fn/main.py"}; !equal(res.Sources, want) { + t.Fatalf("sources = %v, want %v — a docstring import was followed", res.Sources, want) + } + if !strings.Contains(res.Code, "from .ghost import missing") { + t.Fatalf("docstring content was stripped; it should be copied verbatim:\n%s", res.Code) + } +} + +func countString(xs []string, want string) int { + n := 0 + for _, x := range xs { + if x == want { + n++ + } + } + return n +} diff --git a/internal/bundle/imports.go b/internal/bundle/imports.go new file mode 100644 index 0000000..f6bf885 --- /dev/null +++ b/internal/bundle/imports.go @@ -0,0 +1,366 @@ +package bundle + +import ( + "sort" + "strings" +) + +// ImportKind classifies a top-level import statement. The distinction that +// matters is Relative versus everything else: relative imports are resolved +// and inlined, absolute ones are hoisted verbatim and checked against the +// runtime allowlist. +type ImportKind int + +const ( + NotImport ImportKind = iota + ImportAbsolute + ImportFrom + ImportRelative + ImportFuture +) + +// Name is one imported name, with the alias it was bound under if any. +type Name struct { + Name string + Alias string +} + +// Binding is the module-level name this import actually defines. +// +// `import a.b` binds a, not a.b — the submodule is reached through the parent. +// Aliasing changes that to the alias in every form. +func (n Name) Binding() string { + if n.Alias != "" { + return n.Alias + } + if i := strings.IndexByte(n.Name, '.'); i >= 0 { + return n.Name[:i] + } + return n.Name +} + +// Import is a parsed top-level import statement. +type Import struct { + Kind ImportKind + Level int // leading dots; relative imports only + Module string // "" for `import x` and for `from . import x` + Names []Name + Star bool + Stmt Stmt +} + +// Bindings are the module-level names this statement introduces. +func (im Import) Bindings() []string { + out := make([]string, 0, len(im.Names)) + for _, n := range im.Names { + out = append(out, n.Binding()) + } + return out +} + +// ParseImport reads an import statement, or reports false if the statement is +// not one. Input is Stmt.Text, so continuations are already joined. +func ParseImport(s Stmt) (Import, bool) { + text := s.Text + switch { + case strings.HasPrefix(text, "import "): + names, star := parseNameList(text[len("import "):]) + return Import{Kind: ImportAbsolute, Names: names, Star: star, Stmt: s}, true + case strings.HasPrefix(text, "from "): + rest := text[len("from "):] + idx := strings.Index(rest, " import ") + if idx < 0 { + return Import{}, false + } + spec := strings.TrimSpace(rest[:idx]) + names, star := parseNameList(rest[idx+len(" import "):]) + + level := 0 + for level < len(spec) && spec[level] == '.' { + level++ + } + module := strings.TrimSpace(spec[level:]) + + im := Import{Level: level, Module: module, Names: names, Star: star, Stmt: s} + switch { + case module == "__future__" && level == 0: + im.Kind = ImportFuture + case level > 0: + im.Kind = ImportRelative + default: + im.Kind = ImportFrom + } + return im, true + } + return Import{}, false +} + +// parseNameList reads the comma-separated tail of an import statement, +// tolerating the parenthesised multi-line form the scanner has already joined. +func parseNameList(s string) ([]Name, bool) { + s = strings.TrimSpace(s) + s = strings.TrimPrefix(s, "(") + s = strings.TrimSuffix(strings.TrimSpace(s), ")") + + var names []Name + star := false + for _, part := range strings.Split(s, ",") { + fields := strings.Fields(part) + switch { + case len(fields) == 0: + continue + case fields[0] == "*": + star = true + case len(fields) >= 3 && fields[1] == "as": + names = append(names, Name{Name: fields[0], Alias: fields[2]}) + default: + names = append(names, Name{Name: fields[0]}) + } + } + return names, star +} + +// TopLevelBindings are the module-level names a statement defines, for +// collision detection across concatenated modules. +// +// Attribute and subscript targets (a.b = ..., a[0] = ...) mutate an existing +// object rather than binding a module name, so they are deliberately absent. +func TopLevelBindings(s Stmt) []string { + text := s.Text + switch { + case strings.HasPrefix(text, "def "): + return []string{identAfter(text, "def ")} + case strings.HasPrefix(text, "async def "): + return []string{identAfter(text, "async def ")} + case strings.HasPrefix(text, "class "): + return []string{identAfter(text, "class ")} + } + if im, ok := ParseImport(s); ok { + if im.Kind == ImportFuture { + return nil + } + return im.Bindings() + } + if names := clauseTargets(text); len(names) > 0 { + return names + } + if names := walrusTargets(text); len(names) > 0 { + return names + } + return assignTargets(text) +} + +// clauseTargets covers the top-level statements that bind a name without an +// '=': a for loop's variable, a with block's alias, and an except clause's +// exception. They are rare at module level but bind exactly as a def does, so +// omitting them means a real collision goes unreported. +func clauseTargets(text string) []string { + switch { + case strings.HasPrefix(text, "for "): + rest := text[len("for "):] + idx := strings.Index(rest, " in ") + if idx < 0 { + return nil + } + return splitTargets(rest[:idx]) + case strings.HasPrefix(text, "with "), strings.HasPrefix(text, "async with "), + strings.HasPrefix(text, "except "), strings.HasPrefix(text, "except* "): + // Every `as NAME` in the clause; `with` may carry several. + var out []string + for _, part := range strings.Split(text, " as ")[1:] { + name := strings.TrimSpace(part) + end := 0 + for end < len(name) && isIdentByte(name[end]) { + end++ + } + if n := name[:end]; isIdentifier(n) { + out = append(out, n) + } + } + return out + } + return nil +} + +// walrusTargets covers `if (found := f()):` and friends at module level. +func walrusTargets(text string) []string { + var out []string + for i := 0; i+1 < len(text); i++ { + if text[i] != ':' || text[i+1] != '=' { + continue + } + end := i + for end > 0 && text[end-1] == ' ' { + end-- + } + start := end + for start > 0 && isIdentByte(text[start-1]) { + start-- + } + if n := text[start:end]; isIdentifier(n) { + out = append(out, n) + } + } + return out +} + +// splitTargets pulls plain identifiers out of a possibly-tupled target list. +func splitTargets(s string) []string { + var out []string + for _, part := range strings.Split(s, ",") { + name := strings.TrimSpace(strings.Trim(strings.TrimSpace(part), "()[]")) + if isIdentifier(name) { + out = append(out, name) + } + } + return out +} + +// identAfter reads the identifier following a keyword. +func identAfter(text, keyword string) string { + rest := strings.TrimSpace(text[len(keyword):]) + end := 0 + for end < len(rest) && isIdentByte(rest[end]) { + end++ + } + return rest[:end] +} + +// assignTargets extracts names bound by a top-level assignment, covering the +// plain, annotated, tuple and chained forms. +func assignTargets(text string) []string { + eq := topLevelAssign(text) + if eq < 0 { + return nil + } + lhs := text[:eq] + + // Chained assignment: every segment left of the final value is a target. + var out []string + for _, segment := range strings.Split(lhs, "=") { + // Annotated form: the type follows a colon and binds nothing. + if colon := strings.IndexByte(segment, ':'); colon >= 0 { + segment = segment[:colon] + } + for _, target := range strings.Split(segment, ",") { + name := strings.TrimSpace(strings.Trim(strings.TrimSpace(target), "()[]")) + if name == "" || !isIdentifier(name) { + continue + } + out = append(out, name) + } + } + return out +} + +// topLevelAssign returns the index of the assignment '=' that separates +// targets from the value, or -1 if the statement assigns nothing. +// +// It returns the *last* such '=' so that chained assignment (A = B = 1) keeps +// every target on the left; taking the first silently drops all but one. +// Comparisons are stepped over rather than rejected, because `A = B == C` is a +// perfectly good binding, while an augmented operator rebinds an existing name +// and introduces none. +func topLevelAssign(text string) int { + depth := 0 + last := -1 + var quote byte + for i := 0; i < len(text); i++ { + c := text[i] + if quote != 0 { + switch c { + case '\\': + i++ + case quote: + quote = 0 + } + continue + } + switch c { + case '"', '\'': + quote = c + case '(', '[', '{': + depth++ + case ')', ']', '}': + depth-- + case '=': + if depth != 0 { + continue + } + if i+1 < len(text) && text[i+1] == '=' { + i++ // comparison; step over both characters + continue + } + if i > 0 && strings.IndexByte("=!<>+-*/%&|^@", text[i-1]) >= 0 { + return -1 // augmented assignment rebinds, it does not bind + } + last = i + } + } + return last +} + +func isIdentifier(s string) bool { + if s == "" || (s[0] >= '0' && s[0] <= '9') { + return false + } + for i := 0; i < len(s); i++ { + if !isIdentByte(s[i]) { + return false + } + } + return true +} + +func isIdentByte(c byte) bool { + return c == '_' || + (c >= 'a' && c <= 'z') || + (c >= 'A' && c <= 'Z') || + (c >= '0' && c <= '9') +} + +// ExternalImports are the non-relative module names a bundled artifact +// imports, deduplicated and sorted by root. +// +// After flattening there are no relative imports left, so everything returned +// is a real module the runtime will have to resolve. Roots only: `os.path` is +// reported as `os`, because that is the granularity the allowlist works at. +func ExternalImports(code string) []string { + seen := map[string]bool{} + var out []string + for _, stmt := range Scan(code) { + if !stmt.TopLevel() { + continue + } + im, ok := ParseImport(stmt) + if !ok || (im.Kind != ImportAbsolute && im.Kind != ImportFrom) { + continue + } + for _, module := range importedModules(im) { + root := module + if i := strings.IndexByte(root, '.'); i >= 0 { + root = root[:i] + } + if root == "" || seen[root] { + continue + } + seen[root] = true + out = append(out, root) + } + } + sort.Strings(out) + return out +} + +// importedModules is the module names a statement loads. `import a.b` loads +// a.b even though it binds a, and `from a.b import c` loads a.b. +func importedModules(im Import) []string { + if im.Kind == ImportFrom { + return []string{im.Module} + } + out := make([]string, 0, len(im.Names)) + for _, n := range im.Names { + out = append(out, n.Name) + } + return out +} diff --git a/internal/bundle/imports_test.go b/internal/bundle/imports_test.go new file mode 100644 index 0000000..5049d6e --- /dev/null +++ b/internal/bundle/imports_test.go @@ -0,0 +1,236 @@ +package bundle + +import "testing" + +// parseOne scans a single statement and parses it as an import. +func parseOne(t *testing.T, src string) Import { + t.Helper() + stmts := Scan(src) + if len(stmts) != 1 { + t.Fatalf("expected 1 statement from %q, got %d", src, len(stmts)) + } + im, ok := ParseImport(stmts[0]) + if !ok { + t.Fatalf("%q did not parse as an import", src) + } + return im +} + +func TestParseImportAbsolute(t *testing.T) { + im := parseOne(t, "import requests\n") + if im.Kind != ImportAbsolute { + t.Fatalf("kind = %v", im.Kind) + } + if len(im.Names) != 1 || im.Names[0].Name != "requests" { + t.Fatalf("names = %+v", im.Names) + } +} + +func TestParseImportDottedBindsFirstSegment(t *testing.T) { + im := parseOne(t, "import os.path\n") + if got := im.Names[0].Binding(); got != "os" { + t.Fatalf("binding = %q, want %q", got, "os") + } +} + +func TestParseImportAliasBindsAlias(t *testing.T) { + im := parseOne(t, "import numpy.linalg as la\n") + if got := im.Names[0].Binding(); got != "la" { + t.Fatalf("binding = %q, want %q", got, "la") + } +} + +func TestParseImportMultipleNames(t *testing.T) { + im := parseOne(t, "import json, re as regex\n") + if len(im.Names) != 2 { + t.Fatalf("names = %+v", im.Names) + } + if im.Names[0].Binding() != "json" || im.Names[1].Binding() != "regex" { + t.Fatalf("bindings = %v", im.Bindings()) + } +} + +func TestParseFromImport(t *testing.T) { + im := parseOne(t, "from pydantic import BaseModel\n") + if im.Kind != ImportFrom { + t.Fatalf("kind = %v", im.Kind) + } + if im.Module != "pydantic" || im.Level != 0 { + t.Fatalf("module = %q level = %d", im.Module, im.Level) + } +} + +func TestParseRelativeImportLevels(t *testing.T) { + one := parseOne(t, "from .parse import f\n") + if one.Kind != ImportRelative || one.Level != 1 || one.Module != "parse" { + t.Fatalf("got kind=%v level=%d module=%q", one.Kind, one.Level, one.Module) + } + + two := parseOne(t, "from .._shared.http import fetch\n") + if two.Level != 2 || two.Module != "_shared.http" { + t.Fatalf("level = %d module = %q", two.Level, two.Module) + } +} + +// `from . import mod` has no module part; the bundler rejects it, but the +// parser still has to represent it so the rejection can name it. +func TestParseFromDotImportHasEmptyModule(t *testing.T) { + im := parseOne(t, "from . import parse\n") + if im.Kind != ImportRelative || im.Level != 1 || im.Module != "" { + t.Fatalf("got kind=%v level=%d module=%q", im.Kind, im.Level, im.Module) + } +} + +func TestParseStarImport(t *testing.T) { + im := parseOne(t, "from .parse import *\n") + if !im.Star { + t.Fatal("star not detected") + } +} + +func TestParseFutureImport(t *testing.T) { + im := parseOne(t, "from __future__ import annotations\n") + if im.Kind != ImportFuture { + t.Fatalf("kind = %v", im.Kind) + } +} + +func TestParseParenthesisedRelativeImport(t *testing.T) { + im := parseOne(t, "from .parse import (\n parse_rows,\n clean as scrub,\n)\n") + if len(im.Names) != 2 { + t.Fatalf("names = %+v", im.Names) + } + if im.Names[0].Binding() != "parse_rows" || im.Names[1].Binding() != "scrub" { + t.Fatalf("bindings = %v", im.Bindings()) + } + if im.Names[1].Name != "clean" { + t.Fatalf("aliased name = %q, want clean", im.Names[1].Name) + } +} + +func TestParseImportRejectsNonImports(t *testing.T) { + for _, src := range []string{"x = 1\n", "def f():\n pass\n", "important = 1\n"} { + stmts := Scan(src) + if _, ok := ParseImport(stmts[0]); ok { + t.Fatalf("%q parsed as an import", src) + } + } +} + +func bindingsOf(t *testing.T, src string) []string { + t.Helper() + stmts := Scan(src) + if len(stmts) == 0 { + t.Fatalf("no statements in %q", src) + } + return TopLevelBindings(stmts[0]) +} + +func TestTopLevelBindingsDefClass(t *testing.T) { + if got := bindingsOf(t, "def clean(x):\n return x\n"); !equal(got, []string{"clean"}) { + t.Fatalf("got %q", got) + } + if got := bindingsOf(t, "async def fetch(x):\n return x\n"); !equal(got, []string{"fetch"}) { + t.Fatalf("got %q", got) + } + if got := bindingsOf(t, "class Response(BaseModel):\n pass\n"); !equal(got, []string{"Response"}) { + t.Fatalf("got %q", got) + } + if got := bindingsOf(t, "class Bare:\n pass\n"); !equal(got, []string{"Bare"}) { + t.Fatalf("got %q", got) + } +} + +func TestTopLevelBindingsAssignments(t *testing.T) { + cases := []struct { + src string + want []string + }{ + {"TARGET = 1\n", []string{"TARGET"}}, + {"TARGET: str = \"x\"\n", []string{"TARGET"}}, + {"A, B = 1, 2\n", []string{"A", "B"}}, + {"A = B = 1\n", []string{"A", "B"}}, + } + for _, tc := range cases { + if got := bindingsOf(t, tc.src); !equal(got, tc.want) { + t.Fatalf("%q: got %q, want %q", tc.src, got, tc.want) + } + } +} + +// These bind nothing at module level. Counting them would produce phantom +// collisions between modules that merely mutate the same kind of object. +func TestTopLevelBindingsIgnoresNonBindingForms(t *testing.T) { + for _, src := range []string{ + "obj.attr = 1\n", + "items[0] = 1\n", + "COUNT += 1\n", + "if a == b:\n pass\n", + "print(\"x = 1\")\n", + } { + if got := bindingsOf(t, src); len(got) != 0 { + t.Fatalf("%q bound %q, want nothing", src, got) + } + } +} + +// A default argument containing '=' must not be read as an assignment target. +func TestTopLevelBindingsIgnoresEqualsInsideBrackets(t *testing.T) { + if got := bindingsOf(t, "CONFIG = dict(a=1, b=2)\n"); !equal(got, []string{"CONFIG"}) { + t.Fatalf("got %q", got) + } +} + +func TestTopLevelBindingsFutureImportBindsNothing(t *testing.T) { + if got := bindingsOf(t, "from __future__ import annotations\n"); len(got) != 0 { + t.Fatalf("got %q", got) + } +} + +func TestExternalImportsCollectsRootsOnly(t *testing.T) { + code := `import requests +import os.path +from pydantic import BaseModel +from notte_sdk.types import os as notte_os +import json, re + + +def run(): + return 1 +` + got := ExternalImports(code) + want := []string{"json", "notte_sdk", "os", "pydantic", "re", "requests"} + if !equal(got, want) { + t.Fatalf("got %q, want %q", got, want) + } +} + +// After flattening there are no relative imports left, but a stray one must +// not be reported as a module the runtime has to resolve. +func TestExternalImportsSkipsRelativeAndIndented(t *testing.T) { + code := `from .parse import clean +import requests + + +def run(): + import json + return clean(requests, json) +` + if got := ExternalImports(code); !equal(got, []string{"requests"}) { + t.Fatalf("got %q, want [requests]", got) + } +} + +func TestExternalImportsIgnoresDocstrings(t *testing.T) { + code := "\"\"\"Doc.\n\nimport ghost\n\"\"\"\nimport requests\n\n\ndef run():\n return 1\n" + if got := ExternalImports(code); !equal(got, []string{"requests"}) { + t.Fatalf("got %q, want [requests]", got) + } +} + +func TestExternalImportsDeduplicates(t *testing.T) { + code := "import requests\nimport requests\nfrom requests import get\n\n\ndef run():\n return 1\n" + if got := ExternalImports(code); !equal(got, []string{"requests"}) { + t.Fatalf("got %q", got) + } +} diff --git a/internal/bundle/python_test.go b/internal/bundle/python_test.go new file mode 100644 index 0000000..52aecb3 --- /dev/null +++ b/internal/bundle/python_test.go @@ -0,0 +1,118 @@ +package bundle + +import ( + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "testing" +) + +// python locates an interpreter, or skips. These tests assert that the artifact +// is real Python rather than merely plausible-looking text, which no amount of +// string matching in Go can establish. +func python(t *testing.T) string { + t.Helper() + p, err := exec.LookPath("python3") + if err != nil { + t.Skip("python3 not available") + } + return p +} + +// writeTemp puts code in a file named so tracebacks are legible. +func writeTemp(t *testing.T, code string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "artifact.py") + if err := os.WriteFile(path, []byte(code), 0o644); err != nil { + t.Fatal(err) + } + return path +} + +// Every golden artifact must compile. A bundler that emits a syntax error is +// worse than one that refuses, because the failure surfaces after upload. +func TestGoldenArtifactsCompile(t *testing.T) { + py := python(t) + for _, name := range goldenCases { + t.Run(name, func(t *testing.T) { + code, err := os.ReadFile(filepath.Join("testdata", name, "want.py")) + if err != nil { + t.Skipf("no golden file yet: %v", err) + } + path := writeTemp(t, string(code)) + out, err := exec.Command(py, "-m", "py_compile", path).CombinedOutput() + if err != nil { + t.Fatalf("artifact does not compile: %v\n%s\n--- code ---\n%s", err, out, code) + } + }) + } +} + +// runArtifact executes the artifact and evaluates expr against its namespace. +func runArtifact(t *testing.T, code, expr string) string { + t.Helper() + py := python(t) + path := writeTemp(t, code) + script := "import runpy; ns = runpy.run_path(" + strconv.Quote(path) + "); print(" + expr + ")" + out, err := exec.Command(py, "-c", script).CombinedOutput() + if err != nil { + t.Fatalf("executing artifact failed: %v\n%s\n--- code ---\n%s", err, out, code) + } + return strings.TrimSpace(string(out)) +} + +// The alias rule, proven rather than asserted: deleting the import without +// recreating the binding raises NameError here, which is exactly the failure +// that would otherwise reach production. +func TestAliasedImportArtifactActuallyRuns(t *testing.T) { + res := bundleCase(t, "alias-preserved") + if got := runArtifact(t, res.Code, `ns["run"](" hi ")`); got != "['hi']" { + t.Fatalf("run() returned %q, want %q", got, "['hi']") + } +} + +func TestFlattenedDependencyChainRuns(t *testing.T) { + res := bundleCase(t, "topo-order") + if got := runArtifact(t, res.Code, `ns["run"]()`); got != "2" { + t.Fatalf("run() returned %q, want 2", got) + } +} + +func TestDiamondArtifactRuns(t *testing.T) { + res := bundleCase(t, "diamond") + if got := runArtifact(t, res.Code, `ns["run"]()`); got != "1" { + t.Fatalf("run() returned %q, want 1", got) + } +} + +// __future__ must be the first statement or Python refuses the file outright, +// so this compiles only if the emitter got the ordering right. +func TestFutureAnnotationsArtifactCompiles(t *testing.T) { + res := bundleCase(t, "future-annotations") + path := writeTemp(t, res.Code) + out, err := exec.Command(python(t), "-m", "py_compile", path).CombinedOutput() + if err != nil { + t.Fatalf("misplaced __future__ import: %v\n%s\n%s", err, out, res.Code) + } +} + +// Property test: every module in the package must survive into the artifact +// with its top-level definitions intact. +func TestAllGoldenArtifactsDefineRun(t *testing.T) { + py := python(t) + for _, name := range goldenCases { + t.Run(name, func(t *testing.T) { + res := bundleCase(t, name) + if strings.Contains(res.Code, "import requests") || strings.Contains(res.Code, "pydantic") { + t.Skip("artifact needs third-party packages that are not installed for tests") + } + path := writeTemp(t, res.Code) + script := "import runpy; ns = runpy.run_path(" + strconv.Quote(path) + "); assert callable(ns.get('run')), 'run() missing'" + if out, err := exec.Command(py, "-c", script).CombinedOutput(); err != nil { + t.Fatalf("%v\n%s", err, out) + } + }) + } +} diff --git a/internal/bundle/scanner.go b/internal/bundle/scanner.go new file mode 100644 index 0000000..0a166e2 --- /dev/null +++ b/internal/bundle/scanner.go @@ -0,0 +1,174 @@ +package bundle + +import "strings" + +// Stmt is one logical Python statement: a physical line plus any lines joined +// to it by an open bracket or a trailing backslash. +// +// Text has comments stripped and continuation newlines collapsed to single +// spaces, which is what the import parser wants. It is deliberately not +// faithful to the source; emission works from StartLine/EndLine against the +// original file so that everything the bundler does not rewrite survives +// byte-for-byte. +type Stmt struct { + Text string + StartLine int // 1-based, inclusive + EndLine int // 1-based, inclusive + Indent int // leading space/tab count of the first physical line +} + +// TopLevel reports whether the statement starts at column zero. Only +// column-zero statements bind module-level names, so everything the bundler +// analyses is filtered through this. +func (s Stmt) TopLevel() bool { return s.Indent == 0 } + +// Scan splits Python source into logical statements. +// +// It is a tokenizer, not a parser: it understands strings, comments, brackets +// and continuations well enough to know where statements begin and end, and +// nothing else. That is sufficient because the bundler rejects every construct +// whose handling would need a real parse — see Bundle. +// +// Iteration is over bytes rather than runes on purpose. Every byte it reacts to +// is ASCII, and a UTF-8 continuation byte is never ASCII, so multi-byte +// characters pass through untouched. +func Scan(src string) []Stmt { + var out []Stmt + var buf strings.Builder + + line := 1 + depth := 0 + inStmt := false + stmtStart := 0 + stmtIndent := 0 + atLineStart := true + indent := 0 + + var quote byte // 0 when not inside a string literal + var triple bool + + flush := func(end int) { + if !inStmt { + return + } + if text := strings.TrimSpace(buf.String()); text != "" { + out = append(out, Stmt{ + Text: text, + StartLine: stmtStart, + EndLine: end, + Indent: stmtIndent, + }) + } + buf.Reset() + inStmt = false + } + + for i := 0; i < len(src); i++ { + c := src[i] + + if quote != 0 { + buf.WriteByte(c) + switch { + case c == '\n': + line++ + case c == '\\' && i+1 < len(src): + // Backslash consumes the next byte in raw literals too — r"\"" + // is a two-character string, not an unterminated one — so the + // prefix never changes where a string ends and is not parsed. + i++ + buf.WriteByte(src[i]) + if src[i] == '\n' { + line++ + } + case c == quote && triple: + if i+2 < len(src) && src[i+1] == quote && src[i+2] == quote { + buf.WriteByte(src[i+1]) + buf.WriteByte(src[i+2]) + i += 2 + quote = 0 + } + case c == quote: + quote = 0 + } + continue + } + + if c == '\n' { + if depth > 0 && inStmt { + buf.WriteByte(' ') + } else { + flush(line) + } + line++ + atLineStart = true + indent = 0 + continue + } + + if atLineStart { + if c == ' ' || c == '\t' { + indent++ + continue + } + atLineStart = false + if !inStmt { + inStmt = true + stmtStart = line + stmtIndent = indent + } + } + + if c == '#' { + for i < len(src) && src[i] != '\n' { + i++ + } + i-- // hand the newline back to the loop + continue + } + + if c == '\\' && i+1 < len(src) && src[i+1] == '\n' { + i++ + line++ + atLineStart = true + indent = 0 + buf.WriteByte(' ') + continue + } + + switch c { + case ';': + // A semicolon separates statements on one physical line. Without + // this the parser reads `import json; import re` as a single import + // of a module literally named "json;", drops the line as an import, + // and never hoists re — a NameError from an artifact that compiled. + if depth == 0 { + end := line + flush(end) + inStmt = true + stmtStart = end + stmtIndent = indent + continue + } + case '(', '[', '{': + depth++ + case ')', ']', '}': + if depth > 0 { + depth-- + } + case '"', '\'': + quote = c + triple = i+2 < len(src) && src[i+1] == c && src[i+2] == c + buf.WriteByte(c) + if triple { + buf.WriteByte(src[i+1]) + buf.WriteByte(src[i+2]) + i += 2 + } + continue + } + + buf.WriteByte(c) + } + flush(line) + return out +} diff --git a/internal/bundle/scanner_test.go b/internal/bundle/scanner_test.go new file mode 100644 index 0000000..4135116 --- /dev/null +++ b/internal/bundle/scanner_test.go @@ -0,0 +1,203 @@ +package bundle + +import ( + "strings" + "testing" +) + +// texts is the statement text of every scanned statement, for terse assertions. +func texts(stmts []Stmt) []string { + out := make([]string, len(stmts)) + for i, s := range stmts { + out[i] = s.Text + } + return out +} + +func equal(a, b []string) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} + +func TestScanSplitsSimpleStatements(t *testing.T) { + got := texts(Scan("import os\nx = 1\n")) + want := []string{"import os", "x = 1"} + if !equal(got, want) { + t.Fatalf("got %q, want %q", got, want) + } +} + +func TestScanIgnoresBlankAndCommentOnlyLines(t *testing.T) { + got := texts(Scan("\n# a comment\n\nx = 1\n # indented comment\n")) + if want := []string{"x = 1"}; !equal(got, want) { + t.Fatalf("got %q, want %q", got, want) + } +} + +func TestScanStripsTrailingComments(t *testing.T) { + got := texts(Scan("from .parse import f # keep f\n")) + if want := []string{"from .parse import f"}; !equal(got, want) { + t.Fatalf("got %q, want %q", got, want) + } +} + +// A '#' inside a string is not a comment. Getting this wrong truncates the +// statement and can make an import look like it imports fewer names. +func TestScanDoesNotTreatHashInStringAsComment(t *testing.T) { + got := texts(Scan(`url = "https://x.test/#frag"` + "\n")) + if want := []string{`url = "https://x.test/#frag"`}; !equal(got, want) { + t.Fatalf("got %q, want %q", got, want) + } +} + +// The whole reason the scanner exists rather than a line split: an import can +// be parenthesised across many lines. +func TestScanJoinsParenthesisedImport(t *testing.T) { + src := "from .parse import (\n a,\n b as c,\n)\nx = 1\n" + stmts := Scan(src) + if len(stmts) != 2 { + t.Fatalf("expected 2 statements, got %d: %q", len(stmts), texts(stmts)) + } + if !strings.Contains(stmts[0].Text, "a,") || !strings.Contains(stmts[0].Text, "b as c") { + t.Fatalf("continuation not joined: %q", stmts[0].Text) + } + if stmts[0].StartLine != 1 || stmts[0].EndLine != 4 { + t.Fatalf("span = %d..%d, want 1..4", stmts[0].StartLine, stmts[0].EndLine) + } + if stmts[1].StartLine != 5 { + t.Fatalf("second statement starts at %d, want 5", stmts[1].StartLine) + } +} + +func TestScanJoinsBackslashContinuation(t *testing.T) { + stmts := Scan("x = 1 + \\\n 2\ny = 3\n") + if len(stmts) != 2 { + t.Fatalf("expected 2 statements, got %d: %q", len(stmts), texts(stmts)) + } + if stmts[0].EndLine != 2 { + t.Fatalf("EndLine = %d, want 2", stmts[0].EndLine) + } + if stmts[1].Text != "y = 3" { + t.Fatalf("got %q", stmts[1].Text) + } +} + +// A triple-quoted docstring containing import-like text must not produce +// statements. This is the failure that would make the bundler chase imports +// that do not exist. +func TestScanSkipsTripleQuotedContent(t *testing.T) { + src := "\"\"\"Module doc.\n\nfrom .nope import ghost\nimport nothing\n\"\"\"\nimport requests\n" + got := texts(Scan(src)) + if len(got) != 2 { + t.Fatalf("expected 2 statements, got %d: %q", len(got), got) + } + if got[1] != "import requests" { + t.Fatalf("second statement = %q, want %q", got[1], "import requests") + } + if strings.Contains(got[0], "ghost") && !strings.HasPrefix(got[0], `"""`) { + t.Fatalf("docstring body leaked out of its literal: %q", got[0]) + } +} + +func TestScanHandlesSingleQuotedTripleStrings(t *testing.T) { + src := "x = '''\nimport ghost\n'''\ny = 1\n" + got := texts(Scan(src)) + if len(got) != 2 || got[1] != "y = 1" { + t.Fatalf("got %q", got) + } +} + +// An escaped quote must not close the string. If it does, everything after is +// mis-tokenized. +func TestScanHandlesEscapedQuote(t *testing.T) { + src := `x = "she said \"hi\" # not a comment"` + "\ny = 1\n" + got := texts(Scan(src)) + if len(got) != 2 { + t.Fatalf("expected 2 statements, got %d: %q", len(got), got) + } + if got[1] != "y = 1" { + t.Fatalf("got %q", got) + } +} + +// r"\" is a raw string whose backslash still pairs with the closing quote in +// the tokenizer. Treating raw strings as backslash-free ends the literal early. +func TestScanHandlesRawStringWithEscapedQuote(t *testing.T) { + src := `p = r"\""` + "\n" + "y = 1\n" + got := texts(Scan(src)) + if len(got) != 2 || got[1] != "y = 1" { + t.Fatalf("got %q", got) + } +} + +func TestScanRecordsIndent(t *testing.T) { + src := "def f():\n import inner\n return 1\nx = 2\n" + stmts := Scan(src) + if len(stmts) != 4 { + t.Fatalf("expected 4 statements, got %d: %q", len(stmts), texts(stmts)) + } + if !stmts[0].TopLevel() { + t.Fatal("def should be top level") + } + if stmts[1].TopLevel() { + t.Fatalf("indented import reported as top level (indent=%d)", stmts[1].Indent) + } + if !stmts[3].TopLevel() { + t.Fatal("x = 2 should be top level") + } +} + +func TestScanHandlesFileWithoutTrailingNewline(t *testing.T) { + got := texts(Scan("x = 1")) + if want := []string{"x = 1"}; !equal(got, want) { + t.Fatalf("got %q, want %q", got, want) + } +} + +func TestScanHandlesEmptyInput(t *testing.T) { + if got := Scan(""); len(got) != 0 { + t.Fatalf("got %d statements, want 0", len(got)) + } +} + +// Line numbers are what the source map and every error message depend on, so +// they are asserted directly rather than via a golden file. +func TestScanLineNumbersSurviveStringsAndComments(t *testing.T) { + src := "# c\n\"\"\"\ndoc\n\"\"\"\n\nimport requests\n" + stmts := Scan(src) + last := stmts[len(stmts)-1] + if last.Text != "import requests" { + t.Fatalf("last statement = %q", last.Text) + } + if last.StartLine != 6 { + t.Fatalf("StartLine = %d, want 6", last.StartLine) + } +} + +func TestScanNestedBracketsStayOpen(t *testing.T) { + src := "x = [\n (1,\n 2),\n]\ny = 1\n" + stmts := Scan(src) + if len(stmts) != 2 { + t.Fatalf("expected 2 statements, got %d: %q", len(stmts), texts(stmts)) + } + if stmts[0].EndLine != 4 { + t.Fatalf("EndLine = %d, want 4", stmts[0].EndLine) + } +} + +// f-strings may contain braces and quotes; they must not desynchronise the +// bracket depth or the string state. +func TestScanHandlesFString(t *testing.T) { + src := "msg = f\"value={x['k']} #\"\ny = 1\n" + got := texts(Scan(src)) + if len(got) != 2 || got[1] != "y = 1" { + t.Fatalf("got %q", got) + } +} diff --git a/internal/bundle/sourcemap_test.go b/internal/bundle/sourcemap_test.go new file mode 100644 index 0000000..a667251 --- /dev/null +++ b/internal/bundle/sourcemap_test.go @@ -0,0 +1,113 @@ +package bundle + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// Every line the bundler copied from a source file must map back to the exact +// text it came from. This is the invariant the whole source map exists for; if +// it drifts by even one line, a traceback points at the wrong statement, which +// is worse than pointing at nothing. +func TestSourceMapResolvesEveryCopiedLine(t *testing.T) { + for _, name := range goldenCases { + t.Run(name, func(t *testing.T) { + res := bundleCase(t, name) + root := filepath.Join("testdata", name, "in") + + artifactLines := strings.Split(res.Code, "\n") + checked := 0 + for i, text := range artifactLines { + path, srcLine, ok := res.Map.Lookup(i + 1) + if !ok { + continue + } + raw, err := os.ReadFile(filepath.Join(root, path)) + if err != nil { + t.Fatalf("mapped to unreadable file %q: %v", path, err) + } + srcLines := strings.Split(string(raw), "\n") + if srcLine < 1 || srcLine > len(srcLines) { + t.Fatalf("artifact line %d maps to %s:%d, out of range", i+1, path, srcLine) + } + if got := srcLines[srcLine-1]; got != text { + t.Fatalf("artifact line %d = %q but %s:%d = %q", i+1, text, path, srcLine, got) + } + checked++ + } + if checked == 0 { + t.Fatal("no artifact line mapped to a source; the map is empty") + } + }) + } +} + +// Generated lines — the header, hoisted imports, alias assignments — have no +// source. Reporting a location for them would be a fabricated traceback. +func TestSourceMapReturnsNotOkForGeneratedLines(t *testing.T) { + res := bundleCase(t, "alias-preserved") + lines := strings.Split(res.Code, "\n") + + var aliasLine int + for i, l := range lines { + if strings.TrimSpace(l) == "pr = parse_rows" { + aliasLine = i + 1 + break + } + } + if aliasLine == 0 { + t.Fatalf("alias assignment not found:\n%s", res.Code) + } + if path, line, ok := res.Map.Lookup(aliasLine); ok { + t.Fatalf("generated alias line mapped to %s:%d, want no mapping", path, line) + } +} + +func TestSourceMapHeaderCommentIsNotMapped(t *testing.T) { + res := bundleCase(t, "topo-order") + for i, l := range strings.Split(res.Code, "\n") { + if strings.HasPrefix(l, "# ── ") { + if _, _, ok := res.Map.Lookup(i + 1); ok { + t.Fatalf("module header at line %d claims a source location", i+1) + } + } + } +} + +func TestSourceMapLookupOutOfRange(t *testing.T) { + res := bundleCase(t, "single-file") + if _, _, ok := res.Map.Lookup(0); ok { + t.Fatal("line 0 should not resolve") + } + if _, _, ok := res.Map.Lookup(1 << 20); ok { + t.Fatal("a line past the artifact should not resolve") + } +} + +// Contiguous runs are coalesced, so the map stays small on a large bundle. +func TestSourceMapCoalescesContiguousRuns(t *testing.T) { + res := bundleCase(t, "topo-order") + if len(res.Map.Entries) > len(res.Sources)*2 { + t.Fatalf("map has %d entries for %d sources; runs are not being merged", + len(res.Map.Entries), len(res.Sources)) + } +} + +// A run's Count must not overstate its extent, or lines belonging to the next +// module resolve to the previous one. +func TestSourceMapRunsDoNotOverlap(t *testing.T) { + for _, name := range goldenCases { + t.Run(name, func(t *testing.T) { + res := bundleCase(t, name) + prevEnd := 0 + for _, e := range res.Map.Entries { + if e.ArtifactLine <= prevEnd { + t.Fatalf("entry at artifact line %d overlaps a run ending at %d", e.ArtifactLine, prevEnd) + } + prevEnd = e.ArtifactLine + e.Count - 1 + } + }) + } +} diff --git a/internal/bundle/testdata/alias-preserved/in/fn/main.py b/internal/bundle/testdata/alias-preserved/in/fn/main.py new file mode 100644 index 0000000..4977784 --- /dev/null +++ b/internal/bundle/testdata/alias-preserved/in/fn/main.py @@ -0,0 +1,5 @@ +from .parse import parse_rows as pr, clean + + +def run(q: str = "x"): + return pr(clean(q)) diff --git a/internal/bundle/testdata/alias-preserved/in/fn/parse.py b/internal/bundle/testdata/alias-preserved/in/fn/parse.py new file mode 100644 index 0000000..05415ea --- /dev/null +++ b/internal/bundle/testdata/alias-preserved/in/fn/parse.py @@ -0,0 +1,6 @@ +def clean(s): + return s.strip() + + +def parse_rows(s): + return [s] diff --git a/internal/bundle/testdata/alias-preserved/want.py b/internal/bundle/testdata/alias-preserved/want.py new file mode 100644 index 0000000..da00651 --- /dev/null +++ b/internal/bundle/testdata/alias-preserved/want.py @@ -0,0 +1,14 @@ +# ── fn/parse.py ── +def clean(s): + return s.strip() + + +def parse_rows(s): + return [s] + +# ── fn/main.py ── +pr = parse_rows + + +def run(q: str = "x"): + return pr(clean(q)) diff --git a/internal/bundle/testdata/diamond/in/fn/left.py b/internal/bundle/testdata/diamond/in/fn/left.py new file mode 100644 index 0000000..6fd2567 --- /dev/null +++ b/internal/bundle/testdata/diamond/in/fn/left.py @@ -0,0 +1,5 @@ +from .shared import shared + + +def left(): + return shared() diff --git a/internal/bundle/testdata/diamond/in/fn/main.py b/internal/bundle/testdata/diamond/in/fn/main.py new file mode 100644 index 0000000..ad848b5 --- /dev/null +++ b/internal/bundle/testdata/diamond/in/fn/main.py @@ -0,0 +1,6 @@ +from .left import left +from .right import right + + +def run(): + return left() + right() diff --git a/internal/bundle/testdata/diamond/in/fn/right.py b/internal/bundle/testdata/diamond/in/fn/right.py new file mode 100644 index 0000000..63a796f --- /dev/null +++ b/internal/bundle/testdata/diamond/in/fn/right.py @@ -0,0 +1,5 @@ +from .shared import shared + + +def right(): + return shared() + 1 diff --git a/internal/bundle/testdata/diamond/in/fn/shared.py b/internal/bundle/testdata/diamond/in/fn/shared.py new file mode 100644 index 0000000..8dff7cb --- /dev/null +++ b/internal/bundle/testdata/diamond/in/fn/shared.py @@ -0,0 +1,2 @@ +def shared(): + return 0 diff --git a/internal/bundle/testdata/diamond/want.py b/internal/bundle/testdata/diamond/want.py new file mode 100644 index 0000000..a19e1fd --- /dev/null +++ b/internal/bundle/testdata/diamond/want.py @@ -0,0 +1,15 @@ +# ── fn/shared.py ── +def shared(): + return 0 + +# ── fn/left.py ── +def left(): + return shared() + +# ── fn/right.py ── +def right(): + return shared() + 1 + +# ── fn/main.py ── +def run(): + return left() + right() diff --git a/internal/bundle/testdata/docstring-not-an-import/in/fn/main.py b/internal/bundle/testdata/docstring-not-an-import/in/fn/main.py new file mode 100644 index 0000000..f020f4e --- /dev/null +++ b/internal/bundle/testdata/docstring-not-an-import/in/fn/main.py @@ -0,0 +1,10 @@ +"""Doc. + +from .ghost import missing +""" + +from .real import real + + +def run(): + return real() diff --git a/internal/bundle/testdata/docstring-not-an-import/in/fn/real.py b/internal/bundle/testdata/docstring-not-an-import/in/fn/real.py new file mode 100644 index 0000000..59c1402 --- /dev/null +++ b/internal/bundle/testdata/docstring-not-an-import/in/fn/real.py @@ -0,0 +1,2 @@ +def real(): + return 1 diff --git a/internal/bundle/testdata/docstring-not-an-import/want.py b/internal/bundle/testdata/docstring-not-an-import/want.py new file mode 100644 index 0000000..d55e1eb --- /dev/null +++ b/internal/bundle/testdata/docstring-not-an-import/want.py @@ -0,0 +1,14 @@ +# ── fn/real.py ── +def real(): + return 1 + +# ── fn/main.py ── +"""Doc. + +from .ghost import missing +""" + + + +def run(): + return real() diff --git a/internal/bundle/testdata/future-annotations/in/fn/helper.py b/internal/bundle/testdata/future-annotations/in/fn/helper.py new file mode 100644 index 0000000..b83d912 --- /dev/null +++ b/internal/bundle/testdata/future-annotations/in/fn/helper.py @@ -0,0 +1,5 @@ +from __future__ import annotations + + +def helper() -> dict: + return {} diff --git a/internal/bundle/testdata/future-annotations/in/fn/main.py b/internal/bundle/testdata/future-annotations/in/fn/main.py new file mode 100644 index 0000000..e50dec5 --- /dev/null +++ b/internal/bundle/testdata/future-annotations/in/fn/main.py @@ -0,0 +1,7 @@ +from __future__ import annotations + +from .helper import helper + + +def run() -> dict: + return helper() diff --git a/internal/bundle/testdata/future-annotations/want.py b/internal/bundle/testdata/future-annotations/want.py new file mode 100644 index 0000000..7afd80e --- /dev/null +++ b/internal/bundle/testdata/future-annotations/want.py @@ -0,0 +1,9 @@ +from __future__ import annotations + +# ── fn/helper.py ── +def helper() -> dict: + return {} + +# ── fn/main.py ── +def run() -> dict: + return helper() diff --git a/internal/bundle/testdata/import-hoist-dedup/in/fn/a.py b/internal/bundle/testdata/import-hoist-dedup/in/fn/a.py new file mode 100644 index 0000000..00f6677 --- /dev/null +++ b/internal/bundle/testdata/import-hoist-dedup/in/fn/a.py @@ -0,0 +1,5 @@ +import requests + + +def a(): + return requests is not None diff --git a/internal/bundle/testdata/import-hoist-dedup/in/fn/b.py b/internal/bundle/testdata/import-hoist-dedup/in/fn/b.py new file mode 100644 index 0000000..2f5b59d --- /dev/null +++ b/internal/bundle/testdata/import-hoist-dedup/in/fn/b.py @@ -0,0 +1,6 @@ +import requests +from pydantic import BaseModel + + +def b(): + return BaseModel is not None and requests is not None diff --git a/internal/bundle/testdata/import-hoist-dedup/in/fn/main.py b/internal/bundle/testdata/import-hoist-dedup/in/fn/main.py new file mode 100644 index 0000000..b6d9deb --- /dev/null +++ b/internal/bundle/testdata/import-hoist-dedup/in/fn/main.py @@ -0,0 +1,13 @@ +import requests +from pydantic import BaseModel + +from .a import a +from .b import b + + +class Response(BaseModel): + ok: bool + + +def run(): + return Response(ok=bool(requests) and a() and b()) diff --git a/internal/bundle/testdata/import-hoist-dedup/want.py b/internal/bundle/testdata/import-hoist-dedup/want.py new file mode 100644 index 0000000..e2caea1 --- /dev/null +++ b/internal/bundle/testdata/import-hoist-dedup/want.py @@ -0,0 +1,18 @@ +import requests +from pydantic import BaseModel + +# ── fn/a.py ── +def a(): + return requests is not None + +# ── fn/b.py ── +def b(): + return BaseModel is not None and requests is not None + +# ── fn/main.py ── +class Response(BaseModel): + ok: bool + + +def run(): + return Response(ok=bool(requests) and a() and b()) diff --git a/internal/bundle/testdata/shared-parent/in/_shared/http.py b/internal/bundle/testdata/shared-parent/in/_shared/http.py new file mode 100644 index 0000000..4bb461a --- /dev/null +++ b/internal/bundle/testdata/shared-parent/in/_shared/http.py @@ -0,0 +1,5 @@ +import requests + + +def fetch_json(q): + return requests.get(q).json() diff --git a/internal/bundle/testdata/shared-parent/in/fn/main.py b/internal/bundle/testdata/shared-parent/in/fn/main.py new file mode 100644 index 0000000..d7d4e25 --- /dev/null +++ b/internal/bundle/testdata/shared-parent/in/fn/main.py @@ -0,0 +1,5 @@ +from .._shared.http import fetch_json + + +def run(q: str = "x"): + return fetch_json(q) diff --git a/internal/bundle/testdata/shared-parent/want.py b/internal/bundle/testdata/shared-parent/want.py new file mode 100644 index 0000000..83e5249 --- /dev/null +++ b/internal/bundle/testdata/shared-parent/want.py @@ -0,0 +1,9 @@ +import requests + +# ── _shared/http.py ── +def fetch_json(q): + return requests.get(q).json() + +# ── fn/main.py ── +def run(q: str = "x"): + return fetch_json(q) diff --git a/internal/bundle/testdata/single-file/in/fn/main.py b/internal/bundle/testdata/single-file/in/fn/main.py new file mode 100644 index 0000000..ac260ff --- /dev/null +++ b/internal/bundle/testdata/single-file/in/fn/main.py @@ -0,0 +1,5 @@ +import requests + + +def run(): + return requests.get("https://x.test").text diff --git a/internal/bundle/testdata/single-file/want.py b/internal/bundle/testdata/single-file/want.py new file mode 100644 index 0000000..2eb3073 --- /dev/null +++ b/internal/bundle/testdata/single-file/want.py @@ -0,0 +1,5 @@ +import requests + +# ── fn/main.py ── +def run(): + return requests.get("https://x.test").text diff --git a/internal/bundle/testdata/topo-order/in/fn/base.py b/internal/bundle/testdata/topo-order/in/fn/base.py new file mode 100644 index 0000000..322f737 --- /dev/null +++ b/internal/bundle/testdata/topo-order/in/fn/base.py @@ -0,0 +1,2 @@ +def base(): + return 1 diff --git a/internal/bundle/testdata/topo-order/in/fn/main.py b/internal/bundle/testdata/topo-order/in/fn/main.py new file mode 100644 index 0000000..1a13d28 --- /dev/null +++ b/internal/bundle/testdata/topo-order/in/fn/main.py @@ -0,0 +1,5 @@ +from .mid import middle + + +def run(): + return middle() diff --git a/internal/bundle/testdata/topo-order/in/fn/mid.py b/internal/bundle/testdata/topo-order/in/fn/mid.py new file mode 100644 index 0000000..1aa2c10 --- /dev/null +++ b/internal/bundle/testdata/topo-order/in/fn/mid.py @@ -0,0 +1,5 @@ +from .base import base + + +def middle(): + return base() + 1 diff --git a/internal/bundle/testdata/topo-order/want.py b/internal/bundle/testdata/topo-order/want.py new file mode 100644 index 0000000..47f90df --- /dev/null +++ b/internal/bundle/testdata/topo-order/want.py @@ -0,0 +1,11 @@ +# ── fn/base.py ── +def base(): + return 1 + +# ── fn/mid.py ── +def middle(): + return base() + 1 + +# ── fn/main.py ── +def run(): + return middle() diff --git a/internal/cmd/stack.go b/internal/cmd/stack.go new file mode 100644 index 0000000..03c4ebf --- /dev/null +++ b/internal/cmd/stack.go @@ -0,0 +1,218 @@ +package cmd + +import ( + "embed" + "fmt" + "os" + "path/filepath" + "strings" + "text/template" + + "github.com/spf13/cobra" + + "github.com/nottelabs/notte-cli/internal/project" +) + +//go:embed stacktmpl/*.tmpl +var stackTemplates embed.FS + +// stackEnv is the --env flag. It defaults to prod because almost every project +// has exactly one environment; multi-environment support exists for internal +// catalogs and should not be visible to anyone who does not need it. +var stackEnv string + +var stackCmd = &cobra.Command{ + Use: "stack", + Short: "Manage a git-versioned project of Notte functions", + Long: `Manage a directory of Notte functions as one project. + +A stack is a folder of Python functions, a notte.toml describing them, and a +lockfile mapping each to the function it became in every environment. Shared +code is imported normally and flattened into each artifact at deploy time. + +These commands require uv, which supplies the Python the runtime uses: + curl -LsSf https://astral.sh/uv/install.sh | sh`, +} + +func init() { + rootCmd.AddCommand(stackCmd) + stackCmd.PersistentFlags().StringVar(&stackEnv, "env", "", + "Environment to target (default: prod)") +} + +// loadStack finds and loads the project containing the working directory. +func loadStack() (*project.Config, error) { + wd, err := os.Getwd() + if err != nil { + return nil, err + } + root, err := project.Find(wd) + if err != nil { + return nil, fmt.Errorf("%w\n run `notte stack init` to create one", err) + } + return project.Load(root) +} + +// ---------------------------------------------------------------- init + +var ( + stackInitName string + stackInitForce bool +) + +var stackInitCmd = &cobra.Command{ + Use: "init [dir]", + Short: "Scaffold a new stack", + Long: `Create notte.toml, a functions package, and the editor configuration +that makes an editor resolve notte_sdk correctly. + +Nothing is written that already exists unless --force is passed.`, + Args: cobra.MaximumNArgs(1), + RunE: runStackInit, +} + +func init() { + stackCmd.AddCommand(stackInitCmd) + stackInitCmd.Flags().StringVar(&stackInitName, "name", "", "Project name (default: directory name)") + stackInitCmd.Flags().BoolVar(&stackInitForce, "force", false, "Overwrite existing files") +} + +func runStackInit(cmd *cobra.Command, args []string) error { + dir := "." + if len(args) == 1 { + dir = args[0] + } + abs, err := filepath.Abs(dir) + if err != nil { + return err + } + if err := os.MkdirAll(abs, 0o755); err != nil { + return err + } + + name := stackInitName + if name == "" { + name = filepath.Base(abs) + } + data := map[string]string{"Name": name, "Example": "hello"} + + // A function is scaffolded alongside the config so that `notte stack check` + // has something to check immediately, rather than reporting an empty stack. + files := []struct{ path, tmpl string }{ + {project.ConfigName, "notte.toml.tmpl"}, + {".gitignore", "gitignore.tmpl"}, + {"AGENTS.md", "AGENTS.md.tmpl"}, + {"pyrightconfig.json", "pyrightconfig.json.tmpl"}, + {filepath.Join(project.DefaultFunctionsDir, "__init__.py"), ""}, + {filepath.Join(project.DefaultFunctionsDir, "_shared", "__init__.py"), ""}, + {filepath.Join(project.DefaultFunctionsDir, "hello", "__init__.py"), ""}, + {filepath.Join(project.DefaultFunctionsDir, "hello", project.EntrypointName), "main.py.tmpl"}, + } + + var written, skipped []string + for _, f := range files { + target := filepath.Join(abs, f.path) + if _, err := os.Stat(target); err == nil && !stackInitForce { + skipped = append(skipped, f.path) + continue + } + body, err := renderTemplate(f.tmpl, data) + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil { + return err + } + if err := os.WriteFile(target, body, 0o644); err != nil { + return err + } + written = append(written, f.path) + } + + for _, p := range written { + PrintInfo(" created " + p) + } + for _, p := range skipped { + PrintInfo(" exists " + p + " (use --force to overwrite)") + } + + // The editor will report unresolved imports until .notte/venv exists, and + // init deliberately does not build it: scaffolding should work offline and + // without credentials. Saying so here costs a line and saves someone + // debugging a project that is not actually broken. + return PrintResult( + fmt.Sprintf("\nStack ready in %s.\n\n"+ + " next: notte stack sync builds .notte/venv so your editor resolves imports\n"+ + " notte stack check bundles and validates every function\n\n"+ + " Until sync runs, an editor will report pydantic and notte_sdk as unresolved:\n"+ + " there is no environment for it to resolve against yet.", + abs), + map[string]any{"root": abs, "created": written, "skipped": skipped}, + ) +} + +// renderTemplate expands a scaffold template. An empty name means an empty +// file, which is what __init__.py is. +func renderTemplate(name string, data map[string]string) ([]byte, error) { + if name == "" { + return nil, nil + } + raw, err := stackTemplates.ReadFile("stacktmpl/" + name) + if err != nil { + return nil, err + } + tmpl, err := template.New(name).Parse(string(raw)) + if err != nil { + return nil, err + } + var out strings.Builder + if err := tmpl.Execute(&out, data); err != nil { + return nil, err + } + return []byte(out.String()), nil +} + +// ---------------------------------------------------------------- new + +var stackNewCmd = &cobra.Command{ + Use: "new ", + Short: "Scaffold one function", + Args: cobra.ExactArgs(1), + RunE: runStackNew, +} + +func init() { stackCmd.AddCommand(stackNewCmd) } + +func runStackNew(cmd *cobra.Command, args []string) error { + cfg, err := loadStack() + if err != nil { + return err + } + name := args[0] + dir := filepath.Join(cfg.FunctionsPath(), name) + if _, err := os.Stat(dir); err == nil { + return fmt.Errorf("%s already exists", filepath.Join(cfg.Project.FunctionsDir, name)) + } + if err := os.MkdirAll(dir, 0o755); err != nil { + return err + } + + body, err := renderTemplate("main.py.tmpl", map[string]string{"Name": name}) + if err != nil { + return err + } + for path, content := range map[string][]byte{ + filepath.Join(dir, "__init__.py"): nil, + filepath.Join(dir, project.EntrypointName): body, + } { + if err := os.WriteFile(path, content, 0o644); err != nil { + return err + } + } + + entry := filepath.Join(cfg.Project.FunctionsDir, name, project.EntrypointName) + return PrintResult( + fmt.Sprintf("created %s", entry), + map[string]any{"name": name, "entrypoint": entry}, + ) +} diff --git a/internal/cmd/stack_check.go b/internal/cmd/stack_check.go new file mode 100644 index 0000000..8f57c0d --- /dev/null +++ b/internal/cmd/stack_check.go @@ -0,0 +1,356 @@ +package cmd + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/spf13/cobra" + + "github.com/nottelabs/notte-cli/internal/auth" + "github.com/nottelabs/notte-cli/internal/bundle" + "github.com/nottelabs/notte-cli/internal/project" + "github.com/nottelabs/notte-cli/internal/pyenv" +) + +var stackCheckCmd = &cobra.Command{ + Use: "check [target]", + Short: "Build and validate every function, writing nothing remote", + Long: `Bundle each function, then check the result against the runtime's own +rules: its import allowlist, the script contract, and a type check. + +Writes nothing to the API, so it is safe as a CI gate. Target may be a name, a +glob, a path, or "all".`, + Args: cobra.MaximumNArgs(1), + RunE: runStackCheck, +} + +func init() { stackCmd.AddCommand(stackCheckCmd) } + +// checked is one function's outcome. +type checked struct { + Name string `json:"name"` + Entrypoint string `json:"entrypoint"` + Sources []string `json:"sources"` + SourceSHA256 string `json:"source_sha256"` + ArtifactSHA256 string `json:"artifact_sha256"` + Problems []string `json:"problems,omitempty"` +} + +func runStackCheck(cmd *cobra.Command, args []string) error { + target := "" + if len(args) == 1 { + target = args[0] + } + prep, err := prepareStack(cmd, target) + if err != nil { + return err + } + reportChecked(prep.results, prep.failed) + if prep.failed > 0 { + return fmt.Errorf("%d of %d function(s) failed", prep.failed, len(prep.results)) + } + return nil +} + +// prepared is the outcome of bundling and validating a stack. deploy runs the +// same pipeline before it uploads anything, so the two can never disagree +// about whether a function is deployable. +type prepared struct { + cfg *project.Config + target *stackTarget + selected []project.Function + // config is each unit's resolved configuration, central or sidecar. Held + // once so no caller reaches back into cfg.Functions and silently ignores a + // sidecar. + config map[string]project.FunctionConfig + artifacts map[string]*bundle.Result + results []checked + failed int +} + +func prepareStack(cmd *cobra.Command, target string) (*prepared, error) { + cfg, err := loadStack() + if err != nil { + return nil, err + } + + functions, err := project.Discover(cfg) + if err != nil { + return nil, err + } + selected, err := project.Select(functions, target) + if err != nil { + return nil, err + } + + // Bundling comes first because it needs nothing external. A syntax or + // layout problem is reported without ever touching the network. + unitConfig := make(map[string]project.FunctionConfig, len(selected)) + for _, fn := range selected { + fc, err := cfg.FunctionConfigFor(fn) + if err != nil { + return nil, err + } + unitConfig[fn.Name] = fc + } + + fsys := os.DirFS(cfg.FunctionsPath()) + results := make([]checked, 0, len(selected)) + artifacts := map[string]*bundle.Result{} + failed := 0 + + for _, fn := range selected { + res, err := bundle.Bundle(fsys, fn.Entrypoint, bundle.Options{ + Header: bundleHeader(fn.Name), + }) + if err != nil { + results = append(results, checked{Name: fn.Name, Entrypoint: fn.Entrypoint, Problems: []string{err.Error()}}) + failed++ + continue + } + artifacts[fn.Name] = res + results = append(results, checked{ + Name: fn.Name, Entrypoint: fn.Entrypoint, Sources: res.Sources, + SourceSHA256: res.SourceSHA256, ArtifactSHA256: res.ArtifactSHA256, + }) + } + + // Everything below needs the runtime's description of itself, and there is + // no local copy of it to fall back on — that is the point. + dest, err := resolveStackTarget(cfg) + if err != nil { + return nil, err + } + health, tc, err := stackRuntime(cmd, dest) + if err != nil { + return nil, err + } + + // Imports come from the sources rather than the artifacts: a function that + // failed to bundle above still needs its dependencies present, or every + // later diagnostic is a spurious unresolved-import. + imports, err := sourceImports(cfg) + if err != nil { + return nil, err + } + venv := cfg.StatePath("venv") + sync, err := pyenv.Sync(cmd.Context(), tc, pyenv.SyncRequest{ + VenvDir: venv, Health: health, Imports: imports, + }) + if err != nil { + return nil, err + } + reportEnvironment(sync) + + buildDir := cfg.StatePath("build", dest.Env) + if err := os.MkdirAll(buildDir, 0o755); err != nil { + return nil, err + } + + // The sources are checked as well as the artifacts, and not only for + // tidiness: a module under _shared/ that no function imports appears in no + // artifact, so checking artifacts alone would never look at it. Source + // diagnostics also land on the real file directly, with no map in between. + srcRes, err := pyenv.TypeCheck(cmd.Context(), tc, cfg.Root, venv, []string{cfg.Project.FunctionsDir}) + if err != nil { + return nil, err + } + if broken := srcRes.Misconfigured(health); len(broken) > 0 { + return nil, environmentBrokenError(venv, broken) + } + sourceProblems := map[string][]string{} + for _, d := range srcRes.Diagnostics { + owner := functionOwning(d.Path, selected, cfg.Project.FunctionsDir) + sourceProblems[owner] = append(sourceProblems[owner], + fmt.Sprintf("%s:%d: %s [%s]", d.Path, d.Line, d.Message, d.Rule)) + } + + for i := range results { + res, ok := artifacts[results[i].Name] + if !ok { + continue + } + artifactPath := filepath.Join(buildDir, results[i].Name+".py") + if err := os.WriteFile(artifactPath, []byte(res.Code), 0o644); err != nil { + return nil, err + } + + verdict, err := pyenv.Validate(cmd.Context(), venv, health, res.Code) + if err != nil { + return nil, err + } + results[i].Problems = append(results[i].Problems, verdict.Errors...) + + // run()'s parameters are known now, so a cron_variables typo is caught + // here rather than at 09:00 on a Sunday when the schedule fires. + params := make([]project.Param, 0, len(verdict.Variables)) + for _, v := range verdict.Variables { + params = append(params, project.Param{Name: v.Name, HasDefault: v.Default != nil}) + } + results[i].Problems = append(results[i].Problems, + unitConfig[results[i].Name].ScheduleProblems(results[i].Name, params)...) + + rel, err := filepath.Rel(cfg.Root, artifactPath) + if err != nil { + rel = artifactPath + } + tyRes, err := pyenv.TypeCheck(cmd.Context(), tc, cfg.Root, venv, []string{rel}) + if err != nil { + return nil, err + } + // An unresolved import of something the runtime ships means the venv + // is wrong, not the code. Blaming the file would send someone to fix + // something that is fine. + if broken := tyRes.Misconfigured(health); len(broken) > 0 { + return nil, environmentBrokenError(venv, broken) + } + for _, d := range tyRes.Diagnostics { + results[i].Problems = append(results[i].Problems, mapDiagnostic(res, d)) + } + results[i].Problems = append(results[i].Problems, sourceProblems[results[i].Name]...) + if len(results[i].Problems) > 0 { + failed++ + } + } + + // Diagnostics in shared code belong to no single function. Reporting them + // under whichever function happened to import it would be arbitrary, and + // dropping them would hide the case this whole pass exists for. + if shared := sourceProblems[""]; len(shared) > 0 { + failed++ + results = append(results, checked{Name: "(shared)", Problems: shared}) + } + + return &prepared{cfg: cfg, target: dest, selected: selected, config: unitConfig, artifacts: artifacts, results: results, failed: failed}, nil +} + +// mapDiagnostic rewrites an artifact location back to the source it came from. +// +// Without this a report points into a concatenated file, and the reader has to +// work out which module line 612 belonged to. +func mapDiagnostic(res *bundle.Result, d pyenv.Diagnostic) string { + if path, line, ok := res.Map.Lookup(d.Line); ok { + return fmt.Sprintf("%s:%d: %s [%s]", path, line, d.Message, d.Rule) + } + return fmt.Sprintf("(generated):%d: %s [%s]", d.Line, d.Message, d.Rule) +} + +// functionOwning maps a source path to the function whose directory contains +// it, or "" for shared code that belongs to none. +func functionOwning(path string, functions []project.Function, functionsDir string) string { + rel := strings.TrimPrefix(filepath.ToSlash(path), functionsDir+"/") + for _, f := range functions { + if f.Dir && strings.HasPrefix(rel, f.Name+"/") { + return f.Name + } + if !f.Dir && rel == f.Entrypoint { + return f.Name + } + } + return "" +} + +// environmentBrokenError covers the one check failure that is about the +// environment rather than the code: ty could not resolve a package the runtime +// reports as installed, so the venv is wrong, not the function. +// +// It names --force specifically. A plain sync would reuse the environment, +// because the stamp records what it was built from and still matches — so the +// obvious advice is the advice that does nothing. +func environmentBrokenError(venv string, broken []string) error { + return fmt.Errorf( + "the environment in %s cannot resolve %s, which the runtime reports as installed.\n"+ + " the environment is wrong, not your code — rebuild it with:\n"+ + " notte stack sync --force", + venv, strings.Join(broken, ", ")) +} + +func bundleHeader(name string) string { + return fmt.Sprintf("# generated by notte from the %q stack function — do not edit", name) +} + +func reportEnvironment(sync *pyenv.SyncResult) { + verb := "built" + if sync.Reused { + verb = "reused" + } + PrintInfo(fmt.Sprintf("environment %s (Python %s, %d package(s))", verb, sync.Python, len(sync.Installed))) + + // Allowed but absent from the image: passes upload validation, then dies + // on ModuleNotFoundError mid-run. Nothing else makes this visible. + if len(sync.AllowedButMissing) > 0 { + PrintInfo(" warning: the runtime allows but does not ship: " + + strings.Join(sync.AllowedButMissing, ", ")) + } + if len(sync.NotAllowed) > 0 { + PrintInfo(" warning: not available at run time: " + strings.Join(sync.NotAllowed, ", ")) + } +} + +func reportChecked(results []checked, failed int) { + if IsJSONOutput() { + _ = GetFormatter().Print(map[string]any{"functions": results, "failed": failed}) + return + } + for _, r := range results { + if len(r.Problems) == 0 { + PrintInfo(fmt.Sprintf(" ok %-24s %d source(s) %s", r.Name, len(r.Sources), short(r.ArtifactSHA256))) + continue + } + PrintInfo(fmt.Sprintf(" FAIL %s", r.Name)) + for _, p := range r.Problems { + PrintInfo(" " + p) + } + } +} + +func short(sha string) string { + if len(sha) > 12 { + return sha[:12] + } + return sha +} + +// envName is the label used before a target has been resolved. +// +// It must describe the endpoint actually being written to. Defaulting to +// "prod" while NOTTE_API_URL points at staging would file staging function ids +// under the prod key, and the next real prod deploy would then update whatever +// id happened to be there — the exact confusion the per-environment lock +// exists to prevent. +// +// An explicit --env wins, since a project that declares environments has said +// what it means by them. Otherwise the label is derived from the resolved API +// URL using the same mapping the keyring already uses, so the two agree. +func envName() string { + if stackEnv != "" { + return stackEnv + } + return auth.ResolveEnvLabel(auth.GetCurrentAPIURL()) +} + +// stackRuntime fetches the runtime's report from the environment being +// targeted, not from whatever endpoint is ambient. +func stackRuntime(cmd *cobra.Command, target *stackTarget) (*pyenv.Health, *pyenv.Toolchain, error) { + tc, err := pyenv.FindToolchain() + if err != nil { + return nil, nil, err + } + client := target.client + ctx, cancel := GetContextWithTimeout(cmd.Context()) + defer cancel() + + health, err := pyenv.FetchHealth(ctx, client.HTTPClient(), client.BaseURL(), client.APIKey()) + if err != nil { + return nil, nil, err + } + if !health.Complete() { + return nil, nil, fmt.Errorf( + "the function runtime reported %q, so its package list is unavailable: %s\n"+ + " this is normal between an API deploy and a runner rebuild; try again shortly", + health.Status, health.Error) + } + return health, tc, nil +} diff --git a/internal/cmd/stack_client.go b/internal/cmd/stack_client.go new file mode 100644 index 0000000..e0a783d --- /dev/null +++ b/internal/cmd/stack_client.go @@ -0,0 +1,126 @@ +package cmd + +import ( + "fmt" + "os" + + "github.com/nottelabs/notte-cli/internal/api" + "github.com/nottelabs/notte-cli/internal/auth" + "github.com/nottelabs/notte-cli/internal/config" + "github.com/nottelabs/notte-cli/internal/project" +) + +// stackTarget is the endpoint a stack command writes to, and the label its +// results are recorded under. The two are resolved together and can never +// disagree. +type stackTarget struct { + Env string + APIURL string + client *api.NotteClient +} + +// resolveStackTarget couples --env to the endpoint it names. +// +// The previous version resolved these separately: --env chose the lockfile key +// while the client came from the ambient NOTTE_API_URL and credential. So +// `deploy --env staging` with a prod default wrote functions to prod and filed +// their ids under staging — the destination and the record disagreeing is the +// one failure a per-environment lockfile exists to prevent, and it is silent. +// +// Now naming an environment either resolves to that environment or fails. +func resolveStackTarget(cfg *project.Config) (*stackTarget, error) { + env := stackEnv + _, hasBlock := cfg.Envs[env] + if env == "" { + // No --env: the label follows whatever endpoint is already configured, + // so a single-environment project needs no configuration at all. + url := auth.GetCurrentAPIURL() + client, err := GetClient() + if err != nil { + return nil, err + } + return &stackTarget{Env: auth.ResolveEnvLabel(url), APIURL: url, client: client}, nil + } + + // An environment named but not declared is only safe when the ambient + // endpoint already is that environment. Checked before ResolveEnv so the + // error can name the endpoint the caller would otherwise have hit. + ambient := auth.GetCurrentAPIURL() + ambientLabel := auth.ResolveEnvLabel(ambient) + + var resolved project.EnvConfig + if hasBlock { + var err error + if resolved, err = cfg.ResolveEnv(env); err != nil { + return nil, err + } + } else if ambientLabel != env { + return nil, fmt.Errorf( + "--env %s is not declared in %s, and the configured endpoint is %s (%s).\n"+ + " add an [env.%s] block with its api_url, or drop --env to target %s", + env, project.ConfigName, ambient, ambientLabel, env, ambientLabel) + } + + // A section whose URL is definitively another environment is refused. Both + // the lockfile key and every report use the section name, so + // `[env.staging] api_url = "https://api.notte.cc"` would file production + // deployments under staging — and if a [env.prod] block names the same + // URL, two lock keys track one set of remote functions and each makes the + // other look permanently out of date. + // + // Only *known* hosts can contradict. A self-hosted or preview endpoint has + // no canonical label, so there is nothing for the section name to disagree + // with and it stands unchallenged. + if resolved.APIURL != "" && auth.IsKnownEnvHost(resolved.APIURL) { + if label := auth.ResolveEnvLabel(resolved.APIURL); label != env { + return nil, fmt.Errorf( + "[env.%s] names %s, which is the %s endpoint.\n"+ + " rename the section to [env.%s], or point it at the %s endpoint", + env, resolved.APIURL, label, label, env) + } + } + + if resolved.APIURL == "" { + if ambientLabel != env { + return nil, fmt.Errorf( + "[env.%s] in %s sets no api_url, and the configured endpoint is %s (%s).\n"+ + " add api_url to [env.%s], or drop --env to target %s", + env, project.ConfigName, ambient, ambientLabel, env, ambientLabel) + } + resolved.APIURL = ambient + } + + apiKey := resolved.APIKey + if apiKey == "" { + // Fall back to the keyring entry for *this* environment, which + // internal/auth already namespaces by the same label. The global + // NOTTE_API_KEY is deliberately not consulted: it is not tied to an + // endpoint, so using it here is how a prod key reaches staging. + // Chosen by endpoint, never by the section name. SetKeyringAPIKey + // files entries under ResolveEnvLabel(url), so "api_key:staging" means + // the credential for the staging *endpoint* — and a project is free to + // call any endpoint whatever it likes. An earlier version preferred the + // declared name, which meant `[env.staging] api_url = api.notte.cc` + // sent a staging credential to production. To bind a specific + // credential to a section, set api_key in the block. + endpointLabel := auth.ResolveEnvLabel(resolved.APIURL) + key, err := auth.GetKeyringAPIKeyForEnv(endpointLabel) + if err != nil { + return nil, fmt.Errorf( + "no credential for the endpoint %s (%s) that [env.%s] names: %w\n"+ + " set api_key in [env.%s], or run `notte auth login` against that endpoint", + resolved.APIURL, endpointLabel, env, err, env) + } + apiKey = key + } + + var opts []api.NotteClientOption + if origin := os.Getenv(config.EnvRequestOrigin); origin != "" { + opts = append(opts, api.WithRequestOrigin(origin)) + } + client, err := api.NewClientWithURL(apiKey, resolved.APIURL, Version, opts...) + if err != nil { + return nil, err + } + return &stackTarget{Env: env, APIURL: resolved.APIURL, client: client}, nil +} diff --git a/internal/cmd/stack_deploy.go b/internal/cmd/stack_deploy.go new file mode 100644 index 0000000..a927563 --- /dev/null +++ b/internal/cmd/stack_deploy.go @@ -0,0 +1,462 @@ +package cmd + +import ( + "bufio" + "bytes" + "context" + "fmt" + "io" + "mime/multipart" + "os" + "sort" + "strings" + + "github.com/spf13/cobra" + + "github.com/nottelabs/notte-cli/internal/api" + "github.com/nottelabs/notte-cli/internal/bundle" + "github.com/nottelabs/notte-cli/internal/project" +) + +var ( + stackDeployYes bool + stackDeployForceCreate bool + stackDeployAllowSecret bool +) + +var stackDeployCmd = &cobra.Command{ + Use: "deploy [target]", + Short: "Build, validate and upload every function", + Long: `Bundle each function, validate it against the runtime, upload what +changed, and apply any schedules. + +A function is uploaded only when its sources have changed since the last +deploy to this environment, tracked per environment in notte.lock.json.`, + Args: cobra.MaximumNArgs(1), + RunE: runStackDeploy, +} + +func init() { + stackCmd.AddCommand(stackDeployCmd) + stackDeployCmd.Flags().BoolVar(&stackDeployYes, "yes", false, "Do not ask before writing") + stackDeployCmd.Flags().BoolVar(&stackDeployForceCreate, "force-create", false, + "Create even when a function of the same name already exists remotely") + stackDeployCmd.Flags().BoolVar(&stackDeployAllowSecret, "allow-missing-secrets", false, + "Schedule even when required secrets are not configured") +} + +// plannedWrite is one function about to be created or updated. +type plannedWrite struct { + fn project.Function + artifact *bundle.Result + // existingID is empty for a create. + existingID string +} + +func runStackDeploy(cmd *cobra.Command, args []string) error { + target := "" + if len(args) == 1 { + target = args[0] + } + // Validation runs first and in full. Uploading code that check would have + // rejected just moves the failure somewhere more expensive. + prep, err := prepareStack(cmd, target) + if err != nil { + return err + } + if prep.failed > 0 { + reportChecked(prep.results, prep.failed) + return fmt.Errorf("%d function(s) failed validation; nothing was uploaded", prep.failed) + } + + lock, err := project.LoadLock(prep.cfg.Root) + if err != nil { + return err + } + + // The same client the runtime report came from, so the endpoint that was + // validated against is the endpoint written to. + env := prep.target.Env + client := prep.target.client + ctx, cancel := GetContextWithTimeout(cmd.Context()) + defer cancel() + + remote, err := remoteFunctionsByName(ctx, client) + if err != nil { + return err + } + + var writes []plannedWrite + var unchanged []string + for _, fn := range prep.selected { + art := prep.artifacts[fn.Name] + state, known := lock.State(fn.Entrypoint, env) + if known && state.SourceSHA256 == art.SourceSHA256 { + unchanged = append(unchanged, fn.Name) + continue + } + + w := plannedWrite{fn: fn, artifact: art, existingID: state.FunctionID} + if w.existingID == "" { + // Nothing in the lock, but a function of this name upstream. The + // API has no unique constraint on name, so creating would silently + // make a second one while callers keep the id of the first. + if id, clash := remote[deployName(prep.config[fn.Name], fn.Name)]; clash && !stackDeployForceCreate { + return fmt.Errorf( + "%q already exists in %s but is not in %s, so deploying would create a duplicate.\n"+ + " adopt the existing one: notte stack pull --env %s\n"+ + " or create a second: --force-create (existing id %s)", + fn.Name, env, project.LockName, env, id) + } + } + writes = append(writes, w) + } + + for _, name := range unchanged { + PrintInfo(fmt.Sprintf(" unchanged %s", name)) + } + if len(writes) == 0 { + return PrintResult("nothing to deploy", map[string]any{"unchanged": unchanged, "deployed": []any{}}) + } + + if err := confirmWrites(writes, env); err != nil { + return err + } + + configured, err := configuredSecretNames(ctx, client) + if err != nil { + return err + } + + deployed := make([]map[string]any, 0, len(writes)) + var refused int + for _, w := range writes { + fc := prep.config[w.fn.Name] + result, err := uploadFunction(ctx, client, fc, w) + if err != nil { + // Record nothing: the write did not land. + return fmt.Errorf("%s: %w", w.fn.Name, err) + } + + // The hash advances to what was pushed even though the version may not + // have been read back. Tying it to the read-back means a transient + // error on the confirmation mints a duplicate version on the next run. + lock.Record(w.fn.Entrypoint, env, project.EnvState{ + FunctionID: result.id, + Version: result.version, + SourceSHA256: w.artifact.SourceSHA256, + ArtifactSHA256: w.artifact.ArtifactSHA256, + }) + if err := lock.Save(prep.cfg.Root); err != nil { + return err + } + + verb := "updated" + if w.existingID == "" { + verb = "created" + } + PrintInfo(fmt.Sprintf(" %s %-24s %s", verb, w.fn.Name, result.version)) + + missing := missingSecrets(result.requiredSecrets, configured) + entry := map[string]any{ + "name": w.fn.Name, "function_id": result.id, "version": result.version, + "action": verb, "missing_secrets": missing, + } + // Metadata is applied on every deploy, not only at create. Sending it + // with the multipart upload sets it once and then never again, so an + // edit to name or description in notte.toml would silently never + // reach the deployed function — the gap marketplace still has, where + // copy is only editable upstream. + // A metadata failure never fails the deploy. The upload has landed, and + // refusing the whole command over a catalog field would report a + // success as a failure — the same mistake the secrets path avoids. + // self_healing in particular is refused outright for anything the CLI + // created: it resumes the thread that built the function, and a + // CLI-deployed one has none. + changed, err := applyMetadata(ctx, client, fc, deployName(fc, w.fn.Name), result.id) + if err != nil { + PrintInfo(" metadata not applied: " + err.Error()) + entry["metadata_error"] = err.Error() + } else if len(changed) > 0 { + PrintInfo(" configured " + strings.Join(changed, ", ")) + entry["configured"] = changed + } + + if len(missing) > 0 { + PrintInfo(fmt.Sprintf(" missing secrets: %s — it will fail when invoked", strings.Join(missing, ", "))) + for _, name := range missing { + // The name is positional; `--name` is not a flag this command + // takes. A suggestion that fails when pasted is worse than none. + PrintInfo(fmt.Sprintf(" notte functions secrets set %s ", name)) + } + } + + // Only a function that asked for a schedule can have one refused. + cron := fc.Cron + switch { + case cron == "": + // nothing to schedule + case len(missing) > 0 && !stackDeployAllowSecret: + refused++ + entry["scheduled"] = false + PrintInfo(" NOT scheduled — a scheduled run would fail preflight (--allow-missing-secrets to override)") + default: + if err := applySchedule(ctx, client, result.id, cron, fc.CronVariables); err != nil { + return fmt.Errorf("%s: schedule: %w", w.fn.Name, err) + } + entry["scheduled"] = true + PrintInfo(fmt.Sprintf(" scheduled %s", cron)) + } + deployed = append(deployed, entry) + } + + if err := PrintResult("", map[string]any{ + "env": env, "deployed": deployed, "unchanged": unchanged, "schedules_refused": refused, + }); err != nil { + return err + } + if refused > 0 { + // The uploads landed; only the schedules did not. Exit non-zero because + // something asked for was not done — but never pretend nothing happened. + return fmt.Errorf("%d schedule(s) not applied because required secrets are missing", refused) + } + return nil +} + +// confirmWrites shows what is about to be written and asks. +// +// Non-interactive callers must pass --yes rather than being assumed to have +// agreed: marketplace's push refuses without a terminal for the same reason, +// and names the ways out instead of failing silently. +func confirmWrites(writes []plannedWrite, env string) error { + PrintInfo(fmt.Sprintf("\nAbout to write %d function(s) to %s:", len(writes), env)) + for _, w := range writes { + verb := "update" + if w.existingID == "" { + verb = "create" + } + PrintInfo(fmt.Sprintf(" %-7s %-24s %s", verb, w.fn.Name, short(w.artifact.ArtifactSHA256))) + } + if stackDeployYes || skipConfirmation { + return nil + } + if !stdinIsTerminal() { + return fmt.Errorf("refusing to write %d change(s) without confirmation, and there is no terminal to ask.\n"+ + " pass --yes to proceed, or run `notte stack check` to review first", len(writes)) + } + ok, err := confirmDeploy(os.Stdin, os.Stderr, env) + if err != nil { + return err + } + if !ok { + return fmt.Errorf("cancelled") + } + return nil +} + +// stdinIsTerminal reports whether there is someone to ask. +func stdinIsTerminal() bool { + info, err := os.Stdin.Stat() + if err != nil { + return false + } + return info.Mode()&os.ModeCharDevice != 0 +} + +// confirmDeploy asks, defaulting to no on a bare Enter or EOF. +func confirmDeploy(in io.Reader, out io.Writer, env string) (bool, error) { + if _, err := fmt.Fprintf(out, "\nDeploy to %s? [y/N]: ", env); err != nil { + return false, err + } + response, err := bufio.NewReader(in).ReadString('\n') + if err != nil && err != io.EOF { + return false, err + } + response = strings.TrimSpace(strings.ToLower(response)) + return response == "y" || response == "yes", nil +} + +// deployName is the name a function carries upstream. +func deployName(fc project.FunctionConfig, name string) string { + if fc.Name != "" { + return fc.Name + } + return name +} + +func missingSecrets(required []string, configured map[string]bool) []string { + var missing []string + for _, name := range required { + if !configured[name] { + missing = append(missing, name) + } + } + sort.Strings(missing) + return missing +} + +// configuredSecretNames lists function_env secret names. +// +// Names only. Reading a value writes an audit row and bumps last_used_at, so a +// deploy that reads every secret every run is a bad neighbour to whoever has +// to read that log. +func configuredSecretNames(ctx context.Context, client *api.NotteClient) (map[string]bool, error) { + namespace := api.FunctionEnv + resp, err := client.Client().ListSecretsWithResponse(ctx, &api.ListSecretsParams{Namespace: &namespace}) + if err != nil { + return nil, fmt.Errorf("list secrets: %w", err) + } + if err := HandleAPIResponse(resp.HTTPResponse, resp.Body); err != nil { + return nil, err + } + out := map[string]bool{} + if resp.JSON200 != nil { + for _, s := range resp.JSON200.Items { + out[s.Name] = true + } + } + return out, nil +} + +// remoteFunctionsByName maps upstream names to ids, for the duplicate guard. +func remoteFunctionsByName(ctx context.Context, client *api.NotteClient) (map[string]string, error) { + functions, err := listAllFunctions(ctx, client) + if err != nil { + return nil, err + } + out := map[string]string{} + for _, f := range functions { + if f.Name != nil { + out[*f.Name] = f.FunctionId + } + } + return out, nil +} + +type uploadResult struct { + id string + version string + requiredSecrets []string +} + +func uploadFunction(ctx context.Context, client *api.NotteClient, fc project.FunctionConfig, w plannedWrite) (*uploadResult, error) { + var buf bytes.Buffer + writer := multipart.NewWriter(&buf) + + part, err := writer.CreateFormFile("file", w.fn.Name+".py") + if err != nil { + return nil, err + } + if _, err := part.Write([]byte(w.artifact.Code)); err != nil { + return nil, err + } + + if w.existingID == "" { + // Only what create needs to exist at all. Everything else is applied + // afterwards through the metadata endpoint, so the same code path runs + // whether this is a create or an update. + if err := writer.WriteField("name", deployName(fc, w.fn.Name)); err != nil { + return nil, err + } + if fc.Shared { + if err := writer.WriteField("shared", "true"); err != nil { + return nil, err + } + } + } + _ = writer.Close() + + var fn *api.FunctionResponse + if w.existingID == "" { + resp, err := client.Client().FunctionCreateWithBodyWithResponse(ctx, + &api.FunctionCreateParams{}, writer.FormDataContentType(), &buf) + if err != nil { + return nil, err + } + if err := HandleAPIResponse(resp.HTTPResponse, resp.Body); err != nil { + return nil, err + } + fn = resp.JSON200 + } else { + resp, err := client.Client().FunctionUpdateWithBodyWithResponse(ctx, w.existingID, + &api.FunctionUpdateParams{}, writer.FormDataContentType(), &buf) + if err != nil { + return nil, err + } + if err := HandleAPIResponse(resp.HTTPResponse, resp.Body); err != nil { + return nil, err + } + fn = resp.JSON200 + } + if fn == nil { + return nil, fmt.Errorf("the API accepted the upload but returned no function") + } + + res := &uploadResult{id: fn.FunctionId, version: fn.LatestVersion} + if fn.RequiredSecrets != nil { + res.requiredSecrets = *fn.RequiredSecrets + } + return res, nil +} + +// applyMetadata pushes the catalog fields notte.toml owns, and reports which +// ones it sent. +// +// It is a no-op when the config sets none, so a project that names nothing +// makes no extra call. self_healing is a pointer in the config precisely so +// that "unset" and "explicitly false" differ: turning a feature off because a +// file did not mention it would be a surprising deploy. +func applyMetadata(ctx context.Context, client *api.NotteClient, fc project.FunctionConfig, + deployedName, functionID string, +) ([]string, error) { + body := api.FunctionMetadataUpdateJSONRequestBody{} + var changed []string + + if fc.Name != "" { + body.Name = &deployedName + changed = append(changed, "name") + } + if fc.Description != "" { + body.Description = &fc.Description + changed = append(changed, "description") + } + if fc.Domain != "" { + body.Domain = &fc.Domain + changed = append(changed, "domain") + } + if fc.Instructions != "" { + body.Instructions = &fc.Instructions + changed = append(changed, "instructions") + } + if fc.SelfHealing != nil { + body.SelfHealing = fc.SelfHealing + changed = append(changed, "self_healing") + } + if len(changed) == 0 { + return nil, nil + } + + resp, err := client.Client().FunctionMetadataUpdateWithResponse(ctx, functionID, + &api.FunctionMetadataUpdateParams{}, body) + if err != nil { + return nil, fmt.Errorf("configure: %w", err) + } + if err := HandleAPIResponse(resp.HTTPResponse, resp.Body); err != nil { + return nil, fmt.Errorf("configure: %w", err) + } + return changed, nil +} + +func applySchedule(ctx context.Context, client *api.NotteClient, functionID, cron string, variables map[string]any) error { + vars := map[string]interface{}{} + for k, v := range variables { + vars[k] = v + } + resp, err := client.Client().FunctionScheduleSetWithResponse(ctx, functionID, + &api.FunctionScheduleSetParams{}, + api.FunctionScheduleSetJSONRequestBody{Cron: cron, Variables: &vars}) + if err != nil { + return err + } + return HandleAPIResponse(resp.HTTPResponse, resp.Body) +} diff --git a/internal/cmd/stack_doctor.go b/internal/cmd/stack_doctor.go new file mode 100644 index 0000000..6e68302 --- /dev/null +++ b/internal/cmd/stack_doctor.go @@ -0,0 +1,210 @@ +package cmd + +import ( + "fmt" + "os" + "sort" + "strings" + + "github.com/spf13/cobra" + + "github.com/nottelabs/notte-cli/internal/api" + "github.com/nottelabs/notte-cli/internal/auth" + "github.com/nottelabs/notte-cli/internal/project" + "github.com/nottelabs/notte-cli/internal/pyenv" +) + +var stackDoctorCmd = &cobra.Command{ + Use: "doctor", + Short: "Report the toolchain, the environment and the runtime", + Long: `Show what is installed, what the runtime reports, and whether the local +environment matches it. + +Answers the questions that are otherwise support tickets: which Python will my +code run under, what may I import, and why was my function not type checked.`, + Args: cobra.NoArgs, + RunE: runStackDoctor, +} + +func init() { stackCmd.AddCommand(stackDoctorCmd) } + +func runStackDoctor(cmd *cobra.Command, args []string) error { + report := map[string]any{} + var lines []string + ok := func(format string, a ...any) { lines = append(lines, " ✓ "+fmt.Sprintf(format, a...)) } + bad := func(format string, a ...any) { lines = append(lines, " ✗ "+fmt.Sprintf(format, a...)) } + warn := func(format string, a ...any) { lines = append(lines, " ! "+fmt.Sprintf(format, a...)) } + + // --- toolchain ------------------------------------------------------- + tc, tcErr := pyenv.FindToolchain() + if tcErr != nil { + bad("uv not found — `notte stack` needs it") + report["uv"] = nil + } else { + ok("uv %s", tc.UV) + report["uv"] = tc.UV + } + ok("ty pinned at %s", pyenv.TyVersion) + report["ty_version"] = pyenv.TyVersion + + // --- project --------------------------------------------------------- + cfg, cfgErr := loadStack() + if cfgErr != nil { + warn("no stack here (%v)", cfgErr) + } else { + ok("stack %s at %s", cfg.Project.Name, cfg.Root) + report["root"] = cfg.Root + if functions, err := project.Discover(cfg); err == nil { + ok("%d function(s) discovered", len(functions)) + report["functions"] = len(functions) + } else { + bad("discovery failed: %v", err) + } + } + + // --- runtime --------------------------------------------------------- + // + // Resolved through the same coupling every other stack command uses. An + // ambient client labelled with --env would report another environment's + // Python version and package list as if they were this one's, which is a + // diagnostic command telling a confident lie. + client, envLabel, clientErr := doctorClient(cfg, cfgErr) + if clientErr != nil { + bad("no credentials: %v", clientErr) + } else { + ctx, cancel := GetContextWithTimeout(cmd.Context()) + defer cancel() + health, err := pyenv.FetchHealth(ctx, client.HTTPClient(), client.BaseURL(), client.APIKey()) + switch { + case err != nil: + bad("runtime: %v", err) + case !health.Complete(): + // Normal between an API deploy and a runner rebuild, so it is + // reported as a state rather than as a fault. + warn("runtime %s — no package list yet; this clears on the next runner deploy", health.Status) + report["runtime_status"] = health.Status + default: + ok("runtime ok — Python %s, %d package(s), digest %s", + health.PythonVersion, len(health.Packages), short(strings.TrimPrefix(health.RuntimeDigest, "sha256:"))) + report["runtime_status"] = health.Status + report["python_version"] = health.PythonVersion + report["runtime_digest"] = health.RuntimeDigest + + // Allowed but absent from the image passes upload validation and + // then dies mid-run, so it is worth naming even when nothing here + // imports it. + var absent []string + for _, p := range health.Packages { + if !p.Installed { + absent = append(absent, p.ImportName) + } + } + sort.Strings(absent) + if len(absent) > 0 { + warn("allowed but not shipped by the image: %s", strings.Join(absent, ", ")) + report["allowed_but_missing"] = absent + } + + if cfgErr == nil { + doctorEnvironment(cfg, health, ok, bad, warn, report) + } + } + report["api_url"] = client.BaseURL() + report["env"] = envLabel + ok("api %s (env %s)", client.BaseURL(), envLabel) + } + + if IsJSONOutput() { + return GetFormatter().Print(report) + } + for _, line := range lines { + PrintInfo(line) + } + return nil +} + +// doctorClient resolves the endpoint to report on. +// +// Inside a stack it is the environment-coupled client, so --env means the same +// thing here as it does for deploy. Outside one there is no notte.toml to +// resolve against, and doctor still has to work — it is the command people run +// when nothing else does. +// +// The fallback keys on whether --env was promised, not on why the config is +// unavailable. An earlier version branched on cfgErr alone, which made a +// notte.toml that merely fails to parse indistinguishable from no project at +// all: `doctor --env staging` next to a malformed config silently reported on +// the ambient endpoint instead. A flag that cannot be honoured is refused. +func doctorClient(cfg *project.Config, cfgErr error) (*api.NotteClient, string, error) { + if cfgErr == nil { + dest, err := resolveStackTarget(cfg) + if err != nil { + return nil, "", err + } + return dest.client, dest.Env, nil + } + + // A project that exists but will not load is not the same as no project. + // If notte.toml is present, its [env.*] blocks may name an endpoint other + // than the ambient one — and being unable to read them is exactly why the + // ambient endpoint cannot stand in for the requested environment, even + // when the labels happen to agree. + if stackEnv != "" { + if _, findErr := project.Find(workingDir()); findErr == nil { + return nil, "", fmt.Errorf( + "--env %s cannot be resolved because %s could not be read: %v\n"+ + " fix it, or drop --env to report on the configured endpoint", + stackEnv, project.ConfigName, cfgErr) + } + } + + client, err := GetClient() + if err != nil { + return nil, "", err + } + label := auth.ResolveEnvLabel(client.BaseURL()) + + // With no project at all there is nothing for --env to resolve against, so + // it is honoured only when it already describes the configured endpoint. + if stackEnv != "" && stackEnv != label { + return nil, "", fmt.Errorf( + "--env %s cannot be resolved: there is no %s here, and the configured endpoint is %s (%s).\n"+ + " run `notte stack init`, or drop --env to report on %s", + stackEnv, project.ConfigName, client.BaseURL(), label, label) + } + return client, label, nil +} + +// workingDir is the directory doctor looks for a project in. +func workingDir() string { + wd, err := os.Getwd() + if err != nil { + return "." + } + return wd +} + +// doctorEnvironment compares the local venv against the runtime it should +// mirror, which is the check that explains a silently useless type check. +func doctorEnvironment(cfg *project.Config, health *pyenv.Health, + ok, bad, warn func(string, ...any), report map[string]any, +) { + venv := cfg.StatePath("venv") + if _, err := os.Stat(pyenv.PythonPath(venv)); err != nil { + warn("no environment yet — run `notte stack sync`") + report["venv"] = nil + return + } + report["venv"] = venv + + stamp, err := pyenv.ReadStamp(venv) + if err != nil { + warn("environment has no stamp; run `notte stack sync --force`") + return + } + if stamp.RuntimeDigest != health.RuntimeDigest { + bad("environment was built for a different runtime — run `notte stack sync`") + return + } + ok("environment matches the runtime (Python %s)", stamp.PythonVersion) +} diff --git a/internal/cmd/stack_pull.go b/internal/cmd/stack_pull.go new file mode 100644 index 0000000..8ff7987 --- /dev/null +++ b/internal/cmd/stack_pull.go @@ -0,0 +1,346 @@ +package cmd + +import ( + "context" + "fmt" + "io" + "net/http" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "sync" + + "github.com/spf13/cobra" + + "github.com/nottelabs/notte-cli/internal/api" + "github.com/nottelabs/notte-cli/internal/project" +) + +// pullConcurrency bounds the walk. A full pull is one page request per hundred +// functions plus two per function — the list carries no download URL — so +// marketplace's 2,049 functions cost about 4,120 requests. It runs that at 48 +// without ever being rate limited, so this is not a throttling guess. +const pullConcurrency = 8 + +var ( + stackPullWrite bool + stackPullLimit int +) + +var stackPullCmd = &cobra.Command{ + Use: "pull", + Short: "Adopt functions that already exist in an environment", + Long: `Record functions that exist remotely but not in the lockfile, and write +any whose source this tree does not have. + +Run this after cloning a stack, or when deploy reports that a function already +exists upstream. Nothing is ever deleted: functions present remotely and absent +locally are reported, not removed.`, + Args: cobra.NoArgs, + RunE: runStackPull, +} + +func init() { + stackCmd.AddCommand(stackPullCmd) + stackPullCmd.Flags().BoolVar(&stackPullWrite, "write-sources", true, + "Write the source of functions this tree does not have") + stackPullCmd.Flags().IntVar(&stackPullLimit, "limit", 0, + "Adopt at most this many functions (0 = all)") +} + +func runStackPull(cmd *cobra.Command, args []string) error { + cfg, err := loadStack() + if err != nil { + return err + } + dest, err := resolveStackTarget(cfg) + if err != nil { + return err + } + env, client := dest.Env, dest.client + + lock, err := project.LoadLock(cfg.Root) + if err != nil { + return err + } + ctx, cancel := GetContextWithTimeout(cmd.Context()) + defer cancel() + + remote, err := listAllFunctions(ctx, client) + if err != nil { + return err + } + + local, err := project.Discover(cfg) + if err != nil { + return err + } + localByName := map[string]project.Function{} + for _, f := range local { + localByName[f.Name] = f + } + + names := assignNames(remote) + + var adopted, alreadyKnown, written []string + for _, fn := range remote { + name := names[fn.FunctionId] + if name == "" { + continue + } + if stackPullLimit > 0 && len(adopted) >= stackPullLimit { + break + } + + // A function already deployed from this tree keeps its sources. The + // artifact cannot be un-flattened, so overwriting a package with its + // own concatenated output would destroy the sources to "sync" them. + if existing, ok := localByName[name]; ok { + if _, known := lock.State(existing.Entrypoint, env); known { + alreadyKnown = append(alreadyKnown, name) + continue + } + lock.Record(existing.Entrypoint, env, project.EnvState{ + FunctionID: fn.FunctionId, Version: fn.LatestVersion, + }) + adopted = append(adopted, name) + continue + } + + // Unknown to this tree: it lands as a single-file function, because + // that is genuinely what an artifact is. + entrypoint := name + ".py" + lock.Record(entrypoint, env, project.EnvState{ + FunctionID: fn.FunctionId, Version: fn.LatestVersion, + }) + adopted = append(adopted, name) + if stackPullWrite { + written = append(written, entrypoint) + } + } + + var failures map[string]string + if stackPullWrite && len(written) > 0 { + failures = downloadSources(ctx, client, cfg, remote, written) + } + + // The lock is written even when some downloads failed. Their entries are + // still correct — the function exists and has that id — and discarding the + // whole run because one function was unreadable is exactly the "authoritative + // only for what it inspected" rule read backwards. + if err := lock.Save(cfg.Root); err != nil { + return err + } + + // Local functions with no counterpart upstream are reported, never + // removed: absence from a listing is not an instruction to delete. + var notDeployed []string + remoteNames := map[string]bool{} + for _, fn := range remote { + remoteNames[names[fn.FunctionId]] = true + } + for _, f := range local { + if !remoteNames[f.Name] { + notDeployed = append(notDeployed, f.Name) + } + } + for name := range failures { + for i, w := range written { + if w == name+".py" { + written = append(written[:i], written[i+1:]...) + break + } + } + } + sort.Strings(adopted) + sort.Strings(notDeployed) + sort.Strings(written) + + for _, name := range adopted { + PrintInfo(" adopted " + name) + } + for _, name := range alreadyKnown { + PrintInfo(" already " + name) + } + for _, name := range notDeployed { + PrintInfo(" local only " + name + " (not deployed to " + env + " yet)") + } + for _, name := range sortedNames(failures) { + PrintInfo(" unreadable " + name + ": " + failures[name]) + } + + return PrintResult( + fmt.Sprintf("\n%d adopted, %d already known, %d not deployed", len(adopted), len(alreadyKnown), len(notDeployed)), + map[string]any{ + "env": env, "adopted": adopted, "already_known": alreadyKnown, + "not_deployed": notDeployed, "written": written, "unreadable": failures, + }, + ) +} + +var ( + slugUnsafe = regexp.MustCompile(`[^a-z0-9_-]+`) + slugSeparators = regexp.MustCompile(`[-_]{2,}`) +) + +// assignNames gives every remote function a unique local name. +// +// Names are not unique upstream — the API has no constraint on them, and a +// real workspace has several functions called "test". Slugging without +// resolving that silently collapses them onto one path, and the lockfile keeps +// whichever id happened to be recorded last while the rest become +// unreachable. Suffixes are assigned in function-id order so the mapping is +// stable across runs rather than depending on listing order. +func assignNames(remote []api.FunctionListItemResponse) map[string]string { + bySlug := map[string][]api.FunctionListItemResponse{} + for _, fn := range remote { + if slug := functionSlug(fn); slug != "" { + bySlug[slug] = append(bySlug[slug], fn) + } + } + + out := map[string]string{} + for slug, group := range bySlug { + sort.Slice(group, func(i, j int) bool { return group[i].FunctionId < group[j].FunctionId }) + for i, fn := range group { + if i == 0 { + out[fn.FunctionId] = slug + continue + } + out[fn.FunctionId] = fmt.Sprintf("%s_%d", slug, i+1) + } + } + return out +} + +// functionSlug turns an upstream name into a tree-safe identifier. +func functionSlug(fn api.FunctionListItemResponse) string { + if fn.Name == nil { + return "" + } + slug := slugUnsafe.ReplaceAllString(strings.ToLower(strings.TrimSpace(*fn.Name)), "_") + // Runs of separators collapse, so "managed auth - bluesky" becomes + // managed_auth_bluesky rather than managed_auth_-_bluesky. Hyphens + // otherwise survive: real functions are named hn-top-posts, and + // rewriting that would make the tree disagree with the console. + slug = slugSeparators.ReplaceAllString(slug, "_") + return strings.Trim(slug, "-_") +} + +// listAllFunctions walks every page. +// +// A partial walk is never returned. A short listing that looks complete would +// let a caller read absence as deletion, which is the one inference this +// command must never make. +func listAllFunctions(ctx context.Context, client *api.NotteClient) ([]api.FunctionListItemResponse, error) { + var out []api.FunctionListItemResponse + page, size := 1, 100 + for { + resp, err := client.Client().ListFunctionsWithResponse(ctx, &api.ListFunctionsParams{ + Page: &page, PageSize: &size, + }) + if err != nil { + return nil, fmt.Errorf("list functions: %w", err) + } + if err := HandleAPIResponse(resp.HTTPResponse, resp.Body); err != nil { + return nil, err + } + if resp.JSON200 == nil { + return out, nil + } + out = append(out, resp.JSON200.Items...) + if !resp.JSON200.HasNext { + return out, nil + } + page++ + if page > 400 { + return nil, fmt.Errorf("function listing did not terminate after %d pages", page) + } + } +} + +// downloadSources fetches code for the functions being adopted. +// downloadSources fetches what it can and reports what it could not, keyed by +// function name. A published function owned by another workspace answers 403, +// and one of those must not cost the caller the other two thousand. +func downloadSources(ctx context.Context, client *api.NotteClient, cfg *project.Config, + remote []api.FunctionListItemResponse, entrypoints []string, +) map[string]string { + wanted := map[string]bool{} + for _, e := range entrypoints { + wanted[strings.TrimSuffix(e, ".py")] = true + } + + sem := make(chan struct{}, pullConcurrency) + var wg sync.WaitGroup + var mu sync.Mutex + failures := map[string]string{} + + names := assignNames(remote) + for _, fn := range remote { + name := names[fn.FunctionId] + if !wanted[name] { + continue + } + wg.Add(1) + go func(fn api.FunctionListItemResponse, name string) { + defer wg.Done() + sem <- struct{}{} + defer func() { <-sem }() + + code, err := downloadOne(ctx, client, fn.FunctionId) + mu.Lock() + defer mu.Unlock() + if err != nil { + failures[name] = err.Error() + return + } + path := filepath.Join(cfg.FunctionsPath(), name+".py") + if err := os.WriteFile(path, []byte(code), 0o644); err != nil { + failures[name] = err.Error() + } + }(fn, name) + } + wg.Wait() + return failures +} + +// downloadOne resolves the signed URL and fetches the code. +func downloadOne(ctx context.Context, client *api.NotteClient, functionID string) (string, error) { + // No decryption key: the API returns a plain signed URL. An earlier version + // re-derived one with sha256("api_key:{k}:workflow_id:{id}:dumb")[:64], + // copied from the backend — the same duplicated rule marketplace carries. + // `notte functions download` never sent one, and a live call confirms the + // url field is an ordinary CloudFront link. + resp, err := client.Client().FunctionDownloadUrlWithResponse(ctx, functionID, + &api.FunctionDownloadUrlParams{}) + if err != nil { + return "", err + } + if err := HandleAPIResponse(resp.HTTPResponse, resp.Body); err != nil { + return "", err + } + if resp.JSON200 == nil || resp.JSON200.Url == "" { + return "", fmt.Errorf("the API returned no download URL") + } + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, resp.JSON200.Url, nil) + if err != nil { + return "", err + } + fetched, err := client.HTTPClient().Do(req) + if err != nil { + return "", err + } + defer func() { _ = fetched.Body.Close() }() + if fetched.StatusCode != http.StatusOK { + return "", fmt.Errorf("downloading source: HTTP %d", fetched.StatusCode) + } + body, err := io.ReadAll(io.LimitReader(fetched.Body, 8<<20)) + if err != nil { + return "", err + } + return string(body), nil +} diff --git a/internal/cmd/stack_secrets.go b/internal/cmd/stack_secrets.go new file mode 100644 index 0000000..7fc14e4 --- /dev/null +++ b/internal/cmd/stack_secrets.go @@ -0,0 +1,240 @@ +package cmd + +import ( + "bufio" + "context" + "fmt" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + + "github.com/spf13/cobra" + + "github.com/nottelabs/notte-cli/internal/api" +) + +var stackSecretsCmd = &cobra.Command{ + Use: "secrets", + Short: "Compare the secrets your functions need against what is configured", +} + +var stackSecretsDiffCmd = &cobra.Command{ + Use: "diff", + Short: "Report required secrets that are not configured", + Long: `List every secret the deployed functions declare and show which are +missing from this environment. + +What a function requires is computed server-side at upload, by scanning for +os.environ reads plus any NOTTE_REQUIRED_SECRETS constant — so this reflects +what the API will preflight, not a local guess.`, + Args: cobra.NoArgs, + RunE: runStackSecretsDiff, +} + +var stackSecretsPushCmd = &cobra.Command{ + Use: "push [file]", + Short: "Set missing secrets from a .env file", + Long: `Read KEY=VALUE lines and set the ones this environment is missing. + +Defaults to .env., which the scaffolded .gitignore already excludes. +Existing secrets are never touched: the API has no update, so changing one +means delete-then-create, and doing that implicitly would leave a window where +a live function has no secret at all.`, + Args: cobra.MaximumNArgs(1), + RunE: runStackSecretsPush, +} + +func init() { + stackCmd.AddCommand(stackSecretsCmd) + stackSecretsCmd.AddCommand(stackSecretsDiffCmd) + stackSecretsCmd.AddCommand(stackSecretsPushCmd) +} + +// requiredSecrets is the union of what every deployed function declares. +func requiredSecrets(ctx context.Context, client *api.NotteClient) (map[string][]string, error) { + functions, err := listAllFunctions(ctx, client) + if err != nil { + return nil, err + } + byName := map[string][]string{} + for _, fn := range functions { + if fn.RequiredSecrets == nil { + continue + } + for _, secret := range *fn.RequiredSecrets { + name := "?" + if fn.Name != nil { + name = *fn.Name + } + byName[secret] = append(byName[secret], name) + } + } + for secret := range byName { + sort.Strings(byName[secret]) + } + return byName, nil +} + +func runStackSecretsDiff(cmd *cobra.Command, args []string) error { + cfg, err := loadStack() + if err != nil { + return err + } + dest, err := resolveStackTarget(cfg) + if err != nil { + return err + } + client := dest.client + ctx, cancel := GetContextWithTimeout(cmd.Context()) + defer cancel() + + required, err := requiredSecrets(ctx, client) + if err != nil { + return err + } + configured, err := configuredSecretNames(ctx, client) + if err != nil { + return err + } + + var missing, satisfied []string + for secret := range required { + if configured[secret] { + satisfied = append(satisfied, secret) + } else { + missing = append(missing, secret) + } + } + // Configured but required by nothing. Reported, never deleted: a secret + // may exist for a function that has not been deployed yet. + var extra []string + for name := range configured { + if _, needed := required[name]; !needed { + extra = append(extra, name) + } + } + sort.Strings(missing) + sort.Strings(satisfied) + sort.Strings(extra) + + if IsJSONOutput() { + return GetFormatter().Print(map[string]any{ + "env": dest.Env, "missing": missing, "satisfied": satisfied, "unused": extra, + }) + } + for _, s := range missing { + PrintInfo(fmt.Sprintf(" ✗ %-28s required by %s", s, strings.Join(required[s], ", "))) + } + for _, s := range satisfied { + PrintInfo(fmt.Sprintf(" ✓ %-28s required by %s", s, strings.Join(required[s], ", "))) + } + for _, s := range extra { + PrintInfo(fmt.Sprintf(" · %-28s set but required by no deployed function", s)) + } + if len(missing) == 0 && len(satisfied) == 0 && len(extra) == 0 { + PrintInfo("no function declares a secret, and none are configured") + } + if len(missing) > 0 { + return fmt.Errorf("%d required secret(s) are not configured", len(missing)) + } + return nil +} + +var envLine = regexp.MustCompile(`^\s*(?:export\s+)?([A-Z_][A-Z0-9_]*)\s*=\s*(.*)$`) + +func runStackSecretsPush(cmd *cobra.Command, args []string) error { + cfg, err := loadStack() + if err != nil { + return err + } + dest, err := resolveStackTarget(cfg) + if err != nil { + return err + } + client := dest.client + + // Defaults to the file named for the environment being written to, so + // pushing to staging cannot pick up prod's values. + path := filepath.Join(cfg.Root, ".env."+dest.Env) + if len(args) == 1 { + path = args[0] + } + values, err := readEnvFile(path) + if err != nil { + return err + } + ctx, cancel := GetContextWithTimeout(cmd.Context()) + defer cancel() + + configured, err := configuredSecretNames(ctx, client) + if err != nil { + return err + } + + var created, skipped []string + for _, name := range sortedNames(values) { + if configured[name] { + skipped = append(skipped, name) + continue + } + body := api.StoreSecretJSONRequestBody{ + Name: name, Namespace: api.FunctionEnv, Value: values[name], + } + resp, err := client.Client().StoreSecretWithResponse(ctx, &api.StoreSecretParams{}, body) + if err != nil { + return fmt.Errorf("set %s: %w", name, err) + } + if err := HandleAPIResponse(resp.HTTPResponse, resp.Body); err != nil { + return fmt.Errorf("set %s: %w", name, err) + } + created = append(created, name) + } + + for _, name := range created { + PrintInfo(" set " + name) + } + for _, name := range skipped { + PrintInfo(" exists " + name + " (delete it first to change the value)") + } + return PrintResult( + fmt.Sprintf("\n%d set, %d already configured", len(created), len(skipped)), + map[string]any{"created": created, "skipped": skipped, "source": path}, + ) +} + +// readEnvFile parses KEY=VALUE lines. Values are never echoed anywhere. +func readEnvFile(path string) (map[string]string, error) { + file, err := os.Open(path) + if err != nil { + return nil, fmt.Errorf("read %s: %w\n create it, or pass a path", filepath.Base(path), err) + } + defer func() { _ = file.Close() }() + + out := map[string]string{} + scanner := bufio.NewScanner(file) + for scanner.Scan() { + line := strings.TrimSpace(scanner.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + m := envLine.FindStringSubmatch(line) + if m == nil { + continue + } + value := strings.TrimSpace(m[2]) + value = strings.Trim(value, `"'`) + out[m[1]] = value + } + return out, scanner.Err() +} + +func sortedNames(m map[string]string) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/internal/cmd/stack_status.go b/internal/cmd/stack_status.go new file mode 100644 index 0000000..e9df134 --- /dev/null +++ b/internal/cmd/stack_status.go @@ -0,0 +1,153 @@ +package cmd + +import ( + "fmt" + "os" + "sort" + "strings" + + "github.com/spf13/cobra" + + "github.com/nottelabs/notte-cli/internal/bundle" + "github.com/nottelabs/notte-cli/internal/project" +) + +var stackStatusCmd = &cobra.Command{ + Use: "status", + Short: "Show what has changed since the last deploy", + Long: `Compare each function against what the lockfile says was last deployed, +and report which functions a shared module edit would affect. + +Offline: it bundles locally and compares hashes, and does not call the API.`, + Args: cobra.NoArgs, + RunE: runStackStatus, +} + +func init() { stackCmd.AddCommand(stackStatusCmd) } + +type statusRow struct { + Name string `json:"name"` + State string `json:"state"` + Sources []string `json:"sources"` + Version string `json:"version,omitempty"` + Problem string `json:"problem,omitempty"` +} + +func runStackStatus(cmd *cobra.Command, args []string) error { + cfg, err := loadStack() + if err != nil { + return err + } + // status is offline, so it resolves the label without building a client: + // naming an endpoint it never contacts would still have to be right. + dest, err := resolveStackTarget(cfg) + if err != nil { + return err + } + env := dest.Env + lock, err := project.LoadLock(cfg.Root) + if err != nil { + return err + } + functions, err := project.Discover(cfg) + if err != nil { + return err + } + + fsys := os.DirFS(cfg.FunctionsPath()) + rows := make([]statusRow, 0, len(functions)) + // usedBy is the inverse of the import graph, which is what makes a shared + // edit legible: a change to _shared/http.py is invisible in a per-function + // diff, but it changes the artifact of everything that imports it. + usedBy := map[string][]string{} + + for _, fn := range functions { + row := statusRow{Name: fn.Name} + res, err := bundle.Bundle(fsys, fn.Entrypoint, bundle.Options{Header: bundleHeader(fn.Name)}) + if err != nil { + row.State, row.Problem = "broken", err.Error() + rows = append(rows, row) + continue + } + row.Sources = res.Sources + for _, src := range res.Sources { + usedBy[src] = append(usedBy[src], fn.Name) + } + + state, known := lock.State(fn.Entrypoint, env) + switch { + case !known: + row.State = "not deployed" + case state.SourceSHA256 != res.SourceSHA256: + row.State, row.Version = "drifted", state.Version + default: + row.State, row.Version = "up to date", state.Version + } + rows = append(rows, row) + } + + // Anything in the lock with no file is a rename or a deletion. Reported, + // never acted on: absence is not an instruction. + var orphaned []string + present := map[string]bool{} + for _, fn := range functions { + present[fn.Entrypoint] = true + } + for _, entry := range lock.Functions { + if _, deployed := entry.Envs[env]; deployed && !present[entry.Path] { + orphaned = append(orphaned, entry.Path) + } + } + sort.Strings(orphaned) + + shared := sharedSources(usedBy) + + if IsJSONOutput() { + return GetFormatter().Print(map[string]any{ + "env": env, "functions": rows, "orphaned": orphaned, "shared": shared, + }) + } + + PrintInfo(fmt.Sprintf("environment: %s\n", env)) + for _, r := range rows { + switch r.State { + case "up to date": + PrintInfo(fmt.Sprintf(" ✓ %-24s %s", r.Name, r.Version)) + case "drifted": + PrintInfo(fmt.Sprintf(" ✗ %-24s drifted from %s", r.Name, r.Version)) + case "not deployed": + PrintInfo(fmt.Sprintf(" + %-24s not deployed to %s", r.Name, env)) + default: + PrintInfo(fmt.Sprintf(" ! %-24s %s", r.Name, r.Problem)) + } + } + for _, path := range orphaned { + PrintInfo(fmt.Sprintf(" ? %-24s in the lockfile but not on disk (renamed or removed)", path)) + } + if len(shared) > 0 { + PrintInfo("\nshared modules:") + for _, s := range shared { + PrintInfo(fmt.Sprintf(" %-30s → %d function(s): %s", s.Path, len(s.Functions), strings.Join(s.Functions, ", "))) + } + } + return nil +} + +type sharedSource struct { + Path string `json:"path"` + Functions []string `json:"functions"` +} + +// sharedSources are the files more than one function bundles in. +func sharedSources(usedBy map[string][]string) []sharedSource { + var out []sharedSource + for path, users := range usedBy { + if len(users) < 2 { + continue + } + sort.Strings(users) + out = append(out, sharedSource{Path: path, Functions: users}) + } + sort.Slice(out, func(i, j int) bool { return out[i].Path < out[j].Path }) + return out +} diff --git a/internal/cmd/stack_sync.go b/internal/cmd/stack_sync.go new file mode 100644 index 0000000..08ba056 --- /dev/null +++ b/internal/cmd/stack_sync.go @@ -0,0 +1,130 @@ +package cmd + +import ( + "fmt" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + + "github.com/spf13/cobra" + + "github.com/nottelabs/notte-cli/internal/bundle" + "github.com/nottelabs/notte-cli/internal/project" + "github.com/nottelabs/notte-cli/internal/pyenv" +) + +var stackSyncCmd = &cobra.Command{ + Use: "sync", + Aliases: []string{"install"}, + Short: "Build the local Python environment that mirrors the runtime", + Long: `Create .notte/venv with the Python version the runtime runs and the +packages it ships, so an editor resolves imports the way the runtime will. + +Only the allowlisted packages your functions actually import are installed: +the runtime's list is closed, so this is an intersection rather than +dependency resolution. Deploy and check do this automatically; sync is the +explicit form, and the one to run after cloning a stack.`, + Args: cobra.NoArgs, + RunE: runStackSync, +} + +var stackSyncForce bool + +func init() { + stackCmd.AddCommand(stackSyncCmd) + stackSyncCmd.Flags().BoolVar(&stackSyncForce, "force", false, + "Rebuild even if the environment already matches the runtime") +} + +func runStackSync(cmd *cobra.Command, args []string) error { + cfg, err := loadStack() + if err != nil { + return err + } + dest, err := resolveStackTarget(cfg) + if err != nil { + return err + } + health, tc, err := stackRuntime(cmd, dest) + if err != nil { + return err + } + + imports, err := sourceImports(cfg) + if err != nil { + return err + } + + sync, err := pyenv.Sync(cmd.Context(), tc, pyenv.SyncRequest{ + VenvDir: cfg.StatePath("venv"), Health: health, Imports: imports, Force: stackSyncForce, + }) + if err != nil { + return err + } + reportEnvironment(sync) + + installed := make([]string, 0, len(sync.Installed)) + for _, p := range sync.Installed { + installed = append(installed, p.ImportName+" "+p.Version) + } + if !IsJSONOutput() { + for _, line := range installed { + PrintInfo(" " + line) + } + } + + return PrintResult( + fmt.Sprintf("\n%s is ready.", cfg.StatePath("venv")), + map[string]any{ + "venv": cfg.StatePath("venv"), + "python": sync.Python, + "installed": installed, + "allowed_but_missing": sync.AllowedButMissing, + "not_allowed": sync.NotAllowed, + "reused": sync.Reused, + }, + ) +} + +// sourceImports is every non-relative module the stack's Python files import. +// +// Read from the sources rather than from bundled artifacts on purpose: a +// function that fails to bundle still has imports, and its author still needs +// an environment in which to fix it. Scanning sources also covers shared +// modules that no function imports yet, which an artifact never mentions. +func sourceImports(cfg *project.Config) ([]string, error) { + root := cfg.FunctionsPath() + seen := map[string]bool{} + + err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() || !strings.HasSuffix(path, ".py") { + return nil + } + // Tests are included deliberately. They are never bundled, so their + // imports are not the runtime's concern — but they are the editor's, + // and an environment that cannot resolve a test file is half useful. + src, readErr := os.ReadFile(path) + if readErr != nil { + return readErr + } + for _, module := range bundle.ExternalImports(string(src)) { + seen[module] = true + } + return nil + }) + if err != nil { + return nil, err + } + + out := make([]string, 0, len(seen)) + for module := range seen { + out = append(out, module) + } + sort.Strings(out) + return out, nil +} diff --git a/internal/cmd/stack_test.go b/internal/cmd/stack_test.go new file mode 100644 index 0000000..3fdaf8d --- /dev/null +++ b/internal/cmd/stack_test.go @@ -0,0 +1,657 @@ +package cmd + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/nottelabs/notte-cli/internal/api" + "github.com/nottelabs/notte-cli/internal/bundle" + "github.com/nottelabs/notte-cli/internal/project" +) + +// initInto scaffolds a stack in a temp dir by running the command's own logic, +// so the test covers what users get rather than a reimplementation. +func initInto(t *testing.T) string { + t.Helper() + dir := t.TempDir() + wd, err := os.Getwd() + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.Chdir(wd) }) + if err := os.Chdir(dir); err != nil { + t.Fatal(err) + } + stackInitName, stackInitForce = "demo", false + if err := runStackInit(nil, nil); err != nil { + t.Fatalf("init: %v", err) + } + return dir +} + +func TestInitScaffoldsALoadableStack(t *testing.T) { + dir := initInto(t) + + for _, rel := range []string{ + "notte.toml", ".gitignore", "AGENTS.md", "pyrightconfig.json", + "functions/__init__.py", "functions/_shared/__init__.py", + "functions/hello/main.py", + } { + if _, err := os.Stat(filepath.Join(dir, rel)); err != nil { + t.Errorf("missing %s", rel) + } + } + + cfg, err := project.Load(dir) + if err != nil { + t.Fatalf("the scaffolded config must load: %v", err) + } + if cfg.Project.Name != "demo" { + t.Fatalf("name = %q", cfg.Project.Name) + } +} + +// The scaffolded project must have no [env.*] blocks. If init ever writes them +// again, a first-time user is back to thinking three credentials are a +// prerequisite for deploying anything. +func TestInitWritesNoEnvironments(t *testing.T) { + dir := initInto(t) + raw, err := os.ReadFile(filepath.Join(dir, project.ConfigName)) + if err != nil { + t.Fatal(err) + } + for _, line := range strings.Split(string(raw), "\n") { + trimmed := strings.TrimSpace(line) + if strings.HasPrefix(trimmed, "[env.") { + t.Fatalf("scaffold declares an environment: %q", trimmed) + } + } + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + if len(cfg.Envs) != 0 { + t.Fatalf("expected no environments, got %v", cfg.Envs) + } + // And the default must still resolve. + if _, err := cfg.ResolveEnv(""); err != nil { + t.Fatalf("prod must resolve with no [env.*] block: %v", err) + } +} + +// .notte holds the venv and build output and must never be committed. +func TestInitGitignoresStateAndSecrets(t *testing.T) { + dir := initInto(t) + raw, err := os.ReadFile(filepath.Join(dir, ".gitignore")) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{".notte/", ".env"} { + if !strings.Contains(string(raw), want) { + t.Errorf(".gitignore should cover %q", want) + } + } +} + +// The scaffolded function must survive the bundler, or `init` hands the user a +// stack that fails `check` immediately. +func TestScaffoldedFunctionBundles(t *testing.T) { + dir := initInto(t) + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + functions, err := project.Discover(cfg) + if err != nil { + t.Fatalf("discover: %v", err) + } + if len(functions) != 1 || functions[0].Name != "hello" { + t.Fatalf("expected one function named hello, got %+v", functions) + } + res, err := bundle.Bundle(os.DirFS(cfg.FunctionsPath()), functions[0].Entrypoint, bundle.Options{}) + if err != nil { + t.Fatalf("the scaffolded function must bundle: %v", err) + } + if !strings.Contains(res.Code, "def run(") { + t.Fatalf("artifact has no entrypoint:\n%s", res.Code) + } + // It must also satisfy the documented contract: run() returns a BaseModel + // declared in the same file. + if !strings.Contains(res.Code, "class Response(BaseModel)") { + t.Fatalf("scaffold should model the return-type contract:\n%s", res.Code) + } +} + +func TestInitIsIdempotentWithoutForce(t *testing.T) { + dir := initInto(t) + marker := "# edited by hand\n" + path := filepath.Join(dir, project.ConfigName) + raw, _ := os.ReadFile(path) + if err := os.WriteFile(path, append([]byte(marker), raw...), 0o644); err != nil { + t.Fatal(err) + } + if err := runStackInit(nil, nil); err != nil { + t.Fatal(err) + } + after, _ := os.ReadFile(path) + if !strings.HasPrefix(string(after), marker) { + t.Fatal("re-running init clobbered a hand-edited notte.toml") + } +} + +func TestNewCreatesADiscoverableFunction(t *testing.T) { + dir := initInto(t) + if err := runStackNew(nil, []string{"scraper"}); err != nil { + t.Fatalf("new: %v", err) + } + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + functions, err := project.Discover(cfg) + if err != nil { + t.Fatal(err) + } + var found bool + for _, f := range functions { + if f.Name == "scraper" { + found = true + } + } + if !found { + t.Fatalf("new function not discovered: %+v", functions) + } +} + +func TestNewRefusesToOverwrite(t *testing.T) { + initInto(t) + if err := runStackNew(nil, []string{"hello"}); err == nil { + t.Fatal("creating over an existing function should fail") + } +} + +// The lock key must describe the endpoint being written to. Recording a +// staging deploy under "prod" would make the next real prod deploy update +// whatever id happened to be filed there. +func TestEnvNameFollowsTheEndpoint(t *testing.T) { + defer func() { stackEnv = "" }() + + stackEnv = "staging" + if got := envName(); got != "staging" { + t.Fatalf("an explicit --env must win: %q", got) + } + + stackEnv = "" + t.Setenv("NOTTE_API_URL", "https://us-staging.notte.cc") + if got := envName(); got != "staging" { + t.Fatalf("with NOTTE_API_URL on staging the lock key must be staging, got %q", got) + } + + t.Setenv("NOTTE_API_URL", "https://api.notte.cc") + if got := envName(); got != project.DefaultEnv { + t.Fatalf("prod endpoint should map to %q, got %q", project.DefaultEnv, got) + } +} + +// sourceImports must read the sources, not bundled artifacts. A function that +// fails to bundle still has imports, and its author still needs an environment +// in which to fix it — otherwise every later diagnostic is a spurious +// unresolved-import piled on top of the real error. +func TestSourceImportsCoversUnbundlableAndUnimportedFiles(t *testing.T) { + dir := initInto(t) + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + + // A function that cannot bundle: star import from a relative module. + broken := filepath.Join(cfg.FunctionsPath(), "broken") + if err := os.MkdirAll(broken, 0o755); err != nil { + t.Fatal(err) + } + for name, body := range map[string]string{ + "__init__.py": "", + "main.py": "import httpx\nfrom .helpers import *\n\n\ndef run():\n return 1\n", + "helpers.py": "def h():\n return 1\n", + } { + if err := os.WriteFile(filepath.Join(broken, name), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + // A shared module no function imports. + if err := os.WriteFile(filepath.Join(cfg.FunctionsPath(), "_shared", "orphan.py"), + []byte("import requests\n"), 0o644); err != nil { + t.Fatal(err) + } + + imports, err := sourceImports(cfg) + if err != nil { + t.Fatal(err) + } + has := func(want string) bool { + for _, got := range imports { + if got == want { + return true + } + } + return false + } + if !has("httpx") { + t.Errorf("imports of a function that cannot bundle must still be collected: %v", imports) + } + if !has("requests") { + t.Errorf("imports of an unreferenced shared module must be collected: %v", imports) + } + if !has("pydantic") { + t.Errorf("the scaffolded function's imports are missing: %v", imports) + } + for _, got := range imports { + if strings.HasPrefix(got, ".") { + t.Errorf("relative import leaked into the environment list: %q", got) + } + } +} + +func TestSyncIsAliasedToInstall(t *testing.T) { + var found bool + for _, alias := range stackSyncCmd.Aliases { + if alias == "install" { + found = true + } + } + if !found { + t.Fatalf("sync should answer to install too, got %v", stackSyncCmd.Aliases) + } +} + +// Upstream names are not unique — a real workspace has several functions +// called "test". Slugging without resolving that collapses them onto one path, +// and the lock keeps whichever id was recorded last while the rest become +// unreachable. Found against staging, where 3,200 functions collapsed to 23. +func TestAssignNamesResolvesCollisionsDeterministically(t *testing.T) { + name := func(s string) *string { return &s } + remote := []api.FunctionListItemResponse{ + {FunctionId: "ccc", Name: name("test")}, + {FunctionId: "aaa", Name: name("test")}, + {FunctionId: "bbb", Name: name("test")}, + {FunctionId: "ddd", Name: name("unique")}, + } + + got := assignNames(remote) + if len(got) != 4 { + t.Fatalf("every function needs a name: %v", got) + } + seen := map[string]bool{} + for _, n := range got { + if seen[n] { + t.Fatalf("two functions share the local name %q: %v", n, got) + } + seen[n] = true + } + // Lowest id keeps the bare slug, so the mapping does not shuffle when the + // listing order changes. + if got["aaa"] != "test" { + t.Errorf("lowest id should keep the bare slug, got %q", got["aaa"]) + } + if got["ddd"] != "unique" { + t.Errorf("an uncontested name should be untouched, got %q", got["ddd"]) + } + + // Stable across a reordered listing. + shuffled := []api.FunctionListItemResponse{remote[3], remote[1], remote[0], remote[2]} + for id, n := range assignNames(shuffled) { + if got[id] != n { + t.Errorf("id %s named %q then %q — assignment is not stable", id, got[id], n) + } + } +} + +func TestFunctionSlugSanitises(t *testing.T) { + name := func(s string) *string { return &s } + cases := map[string]string{ + "HN Top Posts": "hn_top_posts", + "managed auth - bluesky": "managed_auth_bluesky", + " padded ": "padded", + } + for in, want := range cases { + if got := functionSlug(api.FunctionListItemResponse{Name: name(in)}); got != want { + t.Errorf("%q -> %q, want %q", in, got, want) + } + } + if got := functionSlug(api.FunctionListItemResponse{}); got != "" { + t.Errorf("a nameless function should slug to empty, got %q", got) + } +} + +func TestReadEnvFileParsesTheUsualForms(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, ".env.prod") + body := "# comment\n\nPLAIN=value\nexport EXPORTED=two\nQUOTED=\"three\"\nSINGLE='four'\nSPACED = five \nnot_upper=ignored\n" + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + got, err := readEnvFile(path) + if err != nil { + t.Fatal(err) + } + want := map[string]string{"PLAIN": "value", "EXPORTED": "two", "QUOTED": "three", "SINGLE": "four", "SPACED": "five"} + for k, v := range want { + if got[k] != v { + t.Errorf("%s = %q, want %q", k, got[k], v) + } + } + // Secret names are uppercase by API rule, so a lowercase key is not one. + if _, ok := got["not_upper"]; ok { + t.Error("lowercase keys are not valid secret names and should be skipped") + } +} + +func TestSharedSourcesOnlyReportsFilesUsedTwice(t *testing.T) { + got := sharedSources(map[string][]string{ + "_shared/http.py": {"b", "a"}, + "solo/main.py": {"solo"}, + }) + if len(got) != 1 || got[0].Path != "_shared/http.py" { + t.Fatalf("got %+v", got) + } + if got[0].Functions[0] != "a" { + t.Errorf("users should be sorted, got %v", got[0].Functions) + } +} + +// The environment a command records under and the endpoint it writes to must +// be resolved together. Greptile caught these apart: --env chose the lockfile +// key while the client came from the ambient NOTTE_API_URL, so +// `deploy --env staging` against a prod default wrote functions to prod and +// filed their ids under staging. +func TestNamingAnEnvironmentCannotSilentlyRetarget(t *testing.T) { + defer func() { stackEnv = "" }() + dir := initInto(t) + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + + // staging named, nothing declares it, and the endpoint is prod. + t.Setenv("NOTTE_API_URL", "https://api.notte.cc") + t.Setenv("NOTTE_API_KEY", "sk-test") + stackEnv = "staging" + + if _, err := resolveStackTarget(cfg); err == nil { + t.Fatal("naming an undeclared environment against a prod endpoint must fail, " + + "not quietly deploy to prod and record it as staging") + } else { + for _, want := range []string{"staging", "api.notte.cc"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should name %q so the mismatch is obvious: %v", want, err) + } + } + } +} + +// With no --env the label follows the endpoint, so a single-environment +// project needs no configuration at all. +func TestUnnamedEnvironmentFollowsTheEndpoint(t *testing.T) { + defer func() { stackEnv = "" }() + dir := initInto(t) + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + t.Setenv("NOTTE_API_KEY", "sk-test") + t.Setenv("NOTTE_API_URL", "https://us-staging.notte.cc") + stackEnv = "" + + dest, err := resolveStackTarget(cfg) + if err != nil { + t.Fatal(err) + } + if dest.Env != "staging" { + t.Fatalf("label = %q, want staging", dest.Env) + } + if dest.APIURL != "https://us-staging.notte.cc" { + t.Fatalf("url = %q", dest.APIURL) + } +} + +// A declared environment supplies its own endpoint, so it works regardless of +// what NOTTE_API_URL happens to be. +func TestDeclaredEnvironmentSuppliesItsOwnEndpoint(t *testing.T) { + defer func() { stackEnv = "" }() + dir := writeStack(t, map[string]string{ + "notte.toml": `[project] +name = "demo" + +[env.staging] +api_url = "https://us-staging.notte.cc" +api_key = "${env:STAGING_KEY}" +`, + "functions/__init__.py": "", + "functions/hello/__init__.py": "", + "functions/hello/main.py": "def run():\n return 1\n", + }) + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + t.Setenv("STAGING_KEY", "sk-staging") + t.Setenv("NOTTE_API_URL", "https://api.notte.cc") // ambient prod, deliberately + stackEnv = "staging" + + dest, err := resolveStackTarget(cfg) + if err != nil { + t.Fatal(err) + } + if dest.APIURL != "https://us-staging.notte.cc" { + t.Fatalf("a declared api_url must win over the ambient one, got %q", dest.APIURL) + } + if dest.Env != "staging" { + t.Fatalf("label = %q", dest.Env) + } +} + +// writeStack builds a project directory from a path->content map. +func writeStack(t *testing.T, files map[string]string) string { + t.Helper() + dir := t.TempDir() + for rel, body := range files { + p := filepath.Join(dir, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +// doctor is a diagnostic command, so an ambient client labelled with --env +// would report another environment's Python version and package list as if +// they were this one's — a confident lie from the command people run when +// nothing else works. Greptile caught this one command left unconverted. +func TestDoctorResolvesTheSelectedEnvironment(t *testing.T) { + defer func() { stackEnv = "" }() + dir := writeStack(t, map[string]string{ + "notte.toml": `[project] +name = "demo" + +[env.staging] +api_url = "https://us-staging.notte.cc" +api_key = "${env:STAGING_KEY}" +`, + }) + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + t.Setenv("STAGING_KEY", "sk-staging") + t.Setenv("NOTTE_API_URL", "https://api.notte.cc") // ambient prod + stackEnv = "staging" + + client, label, err := doctorClient(cfg, nil) + if err != nil { + t.Fatal(err) + } + if label != "staging" { + t.Fatalf("label = %q, want staging", label) + } + if client.BaseURL() != "https://us-staging.notte.cc" { + t.Fatalf("doctor would report on %q while labelling it staging", client.BaseURL()) + } +} + +// Outside a stack there is no notte.toml to resolve against, and doctor still +// has to work: it is what people run when nothing else does. +func TestDoctorWorksOutsideAStack(t *testing.T) { + defer func() { stackEnv = "" }() + stackEnv = "" + t.Setenv("NOTTE_API_KEY", "sk-test") + t.Setenv("NOTTE_API_URL", "https://us-dev.notte.cc") + + client, label, err := doctorClient(nil, errNoStackForTest) + if err != nil { + t.Fatal(err) + } + if label != "dev" { + t.Fatalf("label = %q, want dev", label) + } + if client.BaseURL() != "https://us-dev.notte.cc" { + t.Fatalf("url = %q", client.BaseURL()) + } +} + +var errNoStackForTest = fmt.Errorf("no stack here") + +// A notte.toml that exists but fails to parse is not the same as no project. +// Branching on the load error alone made them indistinguishable, so +// `doctor --env staging` beside a malformed config silently reported on the +// ambient endpoint. A flag that cannot be honoured is refused. +func TestDoctorRefusesEnvItCannotResolve(t *testing.T) { + defer func() { stackEnv = "" }() + t.Setenv("NOTTE_API_KEY", "sk-test") + t.Setenv("NOTTE_API_URL", "https://api.notte.cc") // ambient prod + stackEnv = "staging" + + _, _, err := doctorClient(nil, errNoStackForTest) + if err == nil { + t.Fatal("--env staging must not silently fall through to the prod endpoint") + } + for _, want := range []string{"staging", "api.notte.cc"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should name %q: %v", want, err) + } + } +} + +// The fallback still works when --env agrees with the endpoint, or is absent. +func TestDoctorFallbackAllowedWhenEnvMatches(t *testing.T) { + defer func() { stackEnv = "" }() + t.Setenv("NOTTE_API_KEY", "sk-test") + t.Setenv("NOTTE_API_URL", "https://us-dev.notte.cc") + + stackEnv = "dev" + if _, label, err := doctorClient(nil, errNoStackForTest); err != nil || label != "dev" { + t.Fatalf("matching --env should be honoured: label=%q err=%v", label, err) + } + stackEnv = "" + if _, label, err := doctorClient(nil, errNoStackForTest); err != nil || label != "dev" { + t.Fatalf("no --env should follow the endpoint: label=%q err=%v", label, err) + } +} + +// A project that exists but will not load is not the same as no project. Its +// [env.*] blocks may name an endpoint other than the ambient one, and being +// unable to read them is precisely why the ambient endpoint cannot stand in — +// even when the labels happen to agree. +func TestDoctorRefusesEnvWhenTheConfigIsUnreadable(t *testing.T) { + defer func() { stackEnv = "" }() + dir := writeStack(t, map[string]string{"notte.toml": "[project\nname = broken\n"}) + wd, _ := os.Getwd() + t.Cleanup(func() { _ = os.Chdir(wd) }) + if err := os.Chdir(dir); err != nil { + t.Fatal(err) + } + + t.Setenv("NOTTE_API_KEY", "sk-test") + t.Setenv("NOTTE_API_URL", "https://us-staging.notte.cc") + stackEnv = "staging" // label matches the ambient endpoint, and still must refuse + + _, _, err := doctorClient(nil, errNoStackForTest) + if err == nil { + t.Fatal("an unreadable notte.toml must not let --env fall through to the ambient endpoint") + } + if !strings.Contains(err.Error(), project.ConfigName) { + t.Errorf("error should point at the config: %v", err) + } +} + +// A credential is chosen by the endpoint it will be sent to, never by the +// section name that named it. Keyring entries are filed under +// ResolveEnvLabel(url) at write time, so "api_key:staging" is the staging +// *endpoint's* key — and a project may call any endpoint whatever it likes. +// Preferring the section name meant `[env.staging] api_url = api.notte.cc` +// would send a staging credential to production. +func TestSectionNamingAnotherEnvironmentIsRefused(t *testing.T) { + defer func() { stackEnv = "" }() + dir := writeStack(t, map[string]string{ + "notte.toml": `[project] +name = "demo" + +[env.staging] +api_url = "https://api.notte.cc" +api_key = "${env:EXPLICIT_KEY}" +`, + }) + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + t.Setenv("EXPLICIT_KEY", "sk-explicit") + stackEnv = "staging" + + // A section naming a known endpoint of another environment is refused + // outright: the lock key and every report use the section name, so this + // would file production deployments under staging. + _, err = resolveStackTarget(cfg) + if err == nil { + t.Fatal("[env.staging] pointing at the prod endpoint must be refused") + } + for _, want := range []string{"api.notte.cc", "prod"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should name the contradiction (%q): %v", want, err) + } + } +} + +// A self-hosted or preview endpoint has no canonical label, so there is +// nothing for the section name to contradict and it stands. +func TestCustomEndpointSectionIsAllowed(t *testing.T) { + defer func() { stackEnv = "" }() + dir := writeStack(t, map[string]string{ + "notte.toml": `[project] +name = "demo" + +[env.preview] +api_url = "https://my-branch.internal.example.com" +api_key = "${env:PREVIEW_KEY}" +`, + }) + cfg, err := project.Load(dir) + if err != nil { + t.Fatal(err) + } + t.Setenv("PREVIEW_KEY", "sk-preview") + stackEnv = "preview" + + dest, err := resolveStackTarget(cfg) + if err != nil { + t.Fatalf("an unknown host cannot contradict a section name: %v", err) + } + if dest.Env != "preview" || dest.APIURL != "https://my-branch.internal.example.com" { + t.Fatalf("got env=%q url=%q", dest.Env, dest.APIURL) + } +} diff --git a/internal/cmd/stacktmpl/AGENTS.md.tmpl b/internal/cmd/stacktmpl/AGENTS.md.tmpl new file mode 100644 index 0000000..6ddcf2e --- /dev/null +++ b/internal/cmd/stacktmpl/AGENTS.md.tmpl @@ -0,0 +1,47 @@ +# Writing Notte functions in this repo + +`notte stack deploy` bundles each function into a single Python file and +uploads it. These are the rules the runtime enforces — `notte stack check` +verifies them locally against the runtime's own report, so you get the error +in a second rather than after an upload. + +## Layout + +``` +functions/ + _shared/ underscore means library, never deployed + http.py + my_function/ + main.py the entrypoint — must define exactly one run() + parse.py a helper, bundled into the artifact + test_main.py a test, never bundled + quick.py a single-file function +``` + +Import shared code with relative imports: `from .parse import clean`, +`from .._shared.http import fetch`. They are flattened away at deploy time. + +## The contract + +- **Exactly one top-level `run()`.** Its parameters become invocation + variables, and their defaults are the calling convention. +- **`run()` must return a `BaseModel` declared in the same file.** Not a + `dict`, not a bare `list`, not an imported model — wrap a list in a model. +- **Only allowlisted imports.** Run `notte stack doctor` to see the exact list + the runtime ships, with versions. It is not the same as the SDK's own list. +- **No `os`.** Read environment variables with + `from notte_sdk.types import os`. Bare `import os` is rejected. +- **No `sys`, `subprocess`, `pathlib`, `importlib`, `tempfile`.** + +## Things the bundler rejects, and the fix + +| Rejected | Instead | +|---|---| +| `from . import parse` then `parse.clean()` | `from .parse import clean` | +| `from .parse import *` | import the names explicitly | +| two modules defining the same top-level name | rename one | +| an import sharing a line with anything else | put each import on its own line | +| a relative import inside a function body | move it to the top of the file | + +Aliases are preserved: `from .parse import clean as scrub` still binds +`scrub` in the artifact. diff --git a/internal/cmd/stacktmpl/gitignore.tmpl b/internal/cmd/stacktmpl/gitignore.tmpl new file mode 100644 index 0000000..4678f15 --- /dev/null +++ b/internal/cmd/stacktmpl/gitignore.tmpl @@ -0,0 +1,5 @@ +# notte stack +.notte/ +.env +.env.* +!.env.example diff --git a/internal/cmd/stacktmpl/main.py.tmpl b/internal/cmd/stacktmpl/main.py.tmpl new file mode 100644 index 0000000..f0b8402 --- /dev/null +++ b/internal/cmd/stacktmpl/main.py.tmpl @@ -0,0 +1,14 @@ +from pydantic import BaseModel + + +class Response(BaseModel): + """run()'s return type. It must be a BaseModel declared in this file — + that is what lets the deployed function document its own schema.""" + + message: str + + +def run(name: str = "world") -> Response: + """The entrypoint. Parameters become invocation variables, so their + defaults are the calling convention.""" + return Response(message=f"hello, {name}") diff --git a/internal/cmd/stacktmpl/notte.toml.tmpl b/internal/cmd/stacktmpl/notte.toml.tmpl new file mode 100644 index 0000000..5248bd6 --- /dev/null +++ b/internal/cmd/stacktmpl/notte.toml.tmpl @@ -0,0 +1,30 @@ +#:schema https://notte.cc/schema/notte-v1.json + +[project] +name = "{{.Name}}" + +# Functions live in ./functions. Anything there whose name does not start with +# an underscore is deployed — either /main.py or .py. Underscore +# means shared code: import it with `from .._shared.thing import helper`. + +# Per-function settings are optional. Uncomment to use them: +# +# [functions.{{.Example}}] +# name = "A human-readable name" +# description = "What it does" +# domain = "example.com" # the site it acts on +# instructions = "Takes ~30s. `query` is the search term." +# # notes for whoever calls it +# self_healing = true # only for functions an agent built: +# # self-heal resumes the thread that +# # created them, and a CLI deploy has none +# cron = "cron(0 9 * * ? *)" # six-field AWS EventBridge form +# cron_variables = { name = "scheduled" } # arguments the scheduled run uses; +# # without these it runs on run()'s defaults +# secrets = ["MY_TOKEN"] # beyond what the AST scan already finds + +# Environments are opt-in. Most projects deploy to prod and never add these. +# +# [env.staging] +# api_url = "https://us-staging.notte.cc" +# api_key = "${env:NOTTE_API_KEY_STAGING}" diff --git a/internal/cmd/stacktmpl/pyrightconfig.json.tmpl b/internal/cmd/stacktmpl/pyrightconfig.json.tmpl new file mode 100644 index 0000000..263c04d --- /dev/null +++ b/internal/cmd/stacktmpl/pyrightconfig.json.tmpl @@ -0,0 +1,8 @@ +{ + "venvPath": ".notte", + "venv": "venv", + "include": ["functions"], + "exclude": ["**/__pycache__", ".notte"], + "typeCheckingMode": "standard", + "reportMissingImports": "error" +} diff --git a/internal/project/config.go b/internal/project/config.go new file mode 100644 index 0000000..f583887 --- /dev/null +++ b/internal/project/config.go @@ -0,0 +1,414 @@ +// Package project reads a notte stack: the notte.toml a human writes, the +// notte.lock.json the CLI maintains, and the functions on disk between them. +// +// The split matters. notte.toml is hand-owned and never rewritten by the CLI, +// because every Go TOML library either drops comments on write or exposes them +// read-only — and the comments are where the reasoning lives. Everything the +// CLI needs to remember goes in the lock, which is JSON and machine-owned. +// Mixing the two is what left marketplace/manifest.json at 2.2 MB and dirty +// after every sync. +package project + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "regexp" + "sort" + "strings" + + "github.com/BurntSushi/toml" +) + +const ( + // ConfigName is the hand-written project file. + ConfigName = "notte.toml" + // LockName is the machine-written companion. + LockName = "notte.lock.json" + // StateDir holds build output, the venv and caches. Gitignored. + StateDir = ".notte" + // DefaultFunctionsDir is where functions live unless notte.toml says + // otherwise. Deliberately not "notte", which would shadow the real notte + // package the moment the repo root lands on sys.path — which pytest does. + DefaultFunctionsDir = "functions" + // DefaultEnv is the environment every command uses when none is named. + // Almost every project has exactly this one. + DefaultEnv = "prod" +) + +// Config is notte.toml. +type Config struct { + Project ProjectSection `toml:"project"` + Envs map[string]EnvConfig `toml:"env"` + Functions map[string]FunctionConfig `toml:"functions"` + Connectors map[string]ConnectorConfig `toml:"connectors"` + + // Root is the directory containing notte.toml. Not from the file. + Root string `toml:"-"` +} + +type ProjectSection struct { + Name string `toml:"name"` + FunctionsDir string `toml:"functions_dir"` +} + +// EnvConfig is one entry under [env.*]. A project with a single environment +// has none of these at all; prod is implied. +type EnvConfig struct { + APIURL string `toml:"api_url"` + APIKey string `toml:"api_key"` + Extends string `toml:"extends"` + Headers map[string]string `toml:"headers"` +} + +// FunctionConfig is one entry under [functions.*], keyed by function name. +type FunctionConfig struct { + Name string `toml:"name"` + Description string `toml:"description"` + Shared bool `toml:"shared"` + Cron string `toml:"cron"` + Secrets []string `toml:"secrets"` + + // Domain is the site the function acts on, used by the catalog. + Domain string `toml:"domain"` + // Instructions are notes for whoever calls the function: how long a run + // takes, what the variables mean, what it trips over. + Instructions string `toml:"instructions"` + // SelfHealing lets an agent repair the function when a run fails. A + // pointer so "unset" is distinguishable from "explicitly false" — the API + // omits absent fields, and silently turning a feature off because a + // config did not mention it would be a surprising deploy. + SelfHealing *bool `toml:"self_healing"` + + // CronVariables are the arguments a scheduled run is invoked with. + // + // A schedule carries its own variables — POST /functions/{id}/schedule + // takes {cron, variables} — and the server validates the keys against + // run()'s parameters. Without them a scheduled function runs on its + // defaults, which is rarely what a 9am job is for, and a function with a + // parameter that has no default cannot be scheduled at all. + CronVariables map[string]any `toml:"cron_variables"` +} + +// ConnectorConfig is one entry under [connectors.*], keyed by the directory +// name, which is also the catalog slug. +// +// It replaces connectors/.json entirely. Three fields from that format +// are deliberately absent: slug, because it is the directory name; the two +// entrypoint paths, because the layout says where they are; and revision, +// which is derived from the bundle hash rather than hand-maintained. +type ConnectorConfig struct { + Name string `toml:"name"` + Domain string `toml:"domain"` + Category string `toml:"category"` + Color string `toml:"color"` + Description string `toml:"description"` + Method string `toml:"method"` + LoginURL string `toml:"login_url"` + AllowedDomains []string `toml:"allowed_domains"` + SupportsTOTP bool `toml:"supports_totp"` + SupportsEmail bool `toml:"supports_email_2fa"` + SupportsSMS bool `toml:"supports_sms_2fa"` + ProxyCountry string `toml:"proxy_country"` + RequiresHeadful bool `toml:"requires_headful"` + + // Login and Verifier carry the per-role catalog copy. Neither names a + // path: the entrypoints are login.py and verifier.py by definition. + Login RoleConfig `toml:"login"` + Verifier RoleConfig `toml:"verifier"` +} + +// RoleConfig is the catalog copy for one half of a connector. +type RoleConfig struct { + Name string `toml:"name"` + Description string `toml:"description"` +} + +var colorRe = regexp.MustCompile(`^#[0-9a-fA-F]{6}$`) + +// Validate checks the fields the API constrains, so a bad value is caught +// before an upload rather than as a 422 afterwards. +func (c ConnectorConfig) Validate(slug string) []string { + var problems []string + if c.Name == "" { + problems = append(problems, fmt.Sprintf("[connectors.%s] needs a name", slug)) + } + if c.Domain == "" { + problems = append(problems, fmt.Sprintf("[connectors.%s] needs a domain", slug)) + } + if c.Color != "" && !colorRe.MatchString(c.Color) { + problems = append(problems, fmt.Sprintf("[connectors.%s] color %q must be #rrggbb", slug, c.Color)) + } + // proxy_country defaults to "us" server-side; an explicit value must still + // look like a country code rather than a country name. + if c.ProxyCountry != "" && len(c.ProxyCountry) != 2 { + problems = append(problems, fmt.Sprintf("[connectors.%s] proxy_country %q must be a two-letter code", slug, c.ProxyCountry)) + } + return problems +} + +// Param is one of run()'s parameters, as reported by the validator. +type Param struct { + Name string + HasDefault bool +} + +// ScheduleProblems reports configuration that would fail at schedule time. +// +// The server rejects an unexpected key and a missing required one, but only +// when the schedule is set — and a cron that fails at 09:00 on a Sunday is a +// bad way to learn about a typo. run()'s parameters are already known from +// validation, so the same check runs locally. +func (f FunctionConfig) ScheduleProblems(name string, params []Param) []string { + if len(f.CronVariables) == 0 && f.Cron == "" { + return nil + } + if f.Cron == "" { + return []string{fmt.Sprintf("[functions.%s] sets cron_variables but no cron, so they are never used", name)} + } + + known := make(map[string]bool, len(params)) + for _, p := range params { + known[p.Name] = true + } + + var problems []string + var unexpected []string + for key := range f.CronVariables { + if !known[key] { + unexpected = append(unexpected, key) + } + } + sort.Strings(unexpected) + for _, key := range unexpected { + problems = append(problems, fmt.Sprintf("[functions.%s] cron_variables has %q, which is not a parameter of run(%s)", + name, key, paramList(params))) + } + + for _, p := range params { + if p.HasDefault { + continue + } + if _, ok := f.CronVariables[p.Name]; !ok { + problems = append(problems, fmt.Sprintf("[functions.%s] run() requires %q and it has no default, so cron_variables must supply it", + name, p.Name)) + } + } + return problems +} + +func paramList(params []Param) string { + names := make([]string, 0, len(params)) + for _, p := range params { + names = append(names, p.Name) + } + return strings.Join(names, ", ") +} + +// Load reads notte.toml from dir. +func Load(dir string) (*Config, error) { + path := filepath.Join(dir, ConfigName) + raw, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("read %s: %w", ConfigName, err) + } + + var cfg Config + md, err := toml.Decode(string(raw), &cfg) + if err != nil { + return nil, fmt.Errorf("%s: %w", ConfigName, err) + } + // TOML's failure mode for a misspelled key is silence, so an unknown key is + // an error rather than a setting that quietly does nothing. + if undecoded := md.Undecoded(); len(undecoded) > 0 { + keys := make([]string, 0, len(undecoded)) + for _, k := range undecoded { + keys = append(keys, k.String()) + } + return nil, fmt.Errorf("%s: unknown key(s): %s", ConfigName, strings.Join(keys, ", ")) + } + + cfg.Root = dir + if cfg.Project.FunctionsDir == "" { + cfg.Project.FunctionsDir = DefaultFunctionsDir + } + if err := cfg.validate(); err != nil { + return nil, fmt.Errorf("%s: %w", ConfigName, err) + } + return &cfg, nil +} + +// Find walks up from dir looking for notte.toml, the way git finds .git. Any +// command can then run from a subdirectory of the stack. +func Find(dir string) (string, error) { + abs, err := filepath.Abs(dir) + if err != nil { + return "", err + } + for { + if _, err := os.Stat(filepath.Join(abs, ConfigName)); err == nil { + return abs, nil + } + parent := filepath.Dir(abs) + if parent == abs { + return "", fmt.Errorf("no %s found in %s or any parent directory", ConfigName, dir) + } + abs = parent + } +} + +var envNameRe = regexp.MustCompile(`^[a-z][a-z0-9-]*$`) + +func (c *Config) validate() error { + if strings.HasPrefix(c.Project.FunctionsDir, "/") || strings.Contains(c.Project.FunctionsDir, "..") { + return fmt.Errorf("functions_dir must be a relative path inside the project, got %q", c.Project.FunctionsDir) + } + for name, env := range c.Envs { + if !envNameRe.MatchString(name) { + return fmt.Errorf("env name %q must be lowercase alphanumeric with dashes", name) + } + if env.Extends != "" { + if _, ok := c.Envs[env.Extends]; !ok { + return fmt.Errorf("env %q extends %q, which is not defined", name, env.Extends) + } + if env.Extends == name { + return fmt.Errorf("env %q extends itself", name) + } + } + } + // A cycle would otherwise be an infinite loop at resolution time. + for name := range c.Envs { + seen := map[string]bool{name: true} + for cur := c.Envs[name].Extends; cur != ""; cur = c.Envs[cur].Extends { + if seen[cur] { + return fmt.Errorf("env %q has a circular extends chain", name) + } + seen[cur] = true + } + } + return nil +} + +// ResolveEnv flattens an environment, following extends, and expands +// interpolations. An environment that is not declared is not an error when it +// is the default: a single-environment project has no [env.*] block at all. +func (c *Config) ResolveEnv(name string) (EnvConfig, error) { + if name == "" { + name = DefaultEnv + } + env, declared := c.Envs[name] + if !declared { + if name != DefaultEnv { + return EnvConfig{}, fmt.Errorf("env %q is not defined in %s", name, ConfigName) + } + env = EnvConfig{} + } + + // Inherit from the base, nearest wins. + for base := env.Extends; base != ""; { + parent := c.Envs[base] + if env.APIURL == "" { + env.APIURL = parent.APIURL + } + if env.APIKey == "" { + env.APIKey = parent.APIKey + } + merged := map[string]string{} + for k, v := range parent.Headers { + merged[k] = v + } + for k, v := range env.Headers { + merged[k] = v + } + env.Headers = merged + base = parent.Extends + } + + var err error + if env.APIURL, err = expand(env.APIURL); err != nil { + return EnvConfig{}, fmt.Errorf("env %q api_url: %w", name, err) + } + if env.APIKey, err = expand(env.APIKey); err != nil { + return EnvConfig{}, fmt.Errorf("env %q api_key: %w", name, err) + } + for k, v := range env.Headers { + if env.Headers[k], err = expand(v); err != nil { + return EnvConfig{}, fmt.Errorf("env %q header %q: %w", name, k, err) + } + } + return env, nil +} + +var interpolationRe = regexp.MustCompile(`\$\{([a-z]+):([^}]+)\}`) + +// expand resolves ${env:VAR} and ${git:branch} style references. +// +// An unresolved reference is an error rather than an empty string. managed-auth +// records why: a header silently ignored meant "a silent wrong write", and +// expanding to "" produces exactly that class of failure — an api_url of "" or +// a credential of "" that fails somewhere far from the cause. +func expand(s string) (string, error) { + if s == "" { + return "", nil + } + var firstErr error + out := interpolationRe.ReplaceAllStringFunc(s, func(match string) string { + parts := interpolationRe.FindStringSubmatch(match) + ns, key := parts[1], parts[2] + value, err := lookup(ns, key) + if err != nil && firstErr == nil { + firstErr = err + } + return value + }) + return out, firstErr +} + +func lookup(ns, key string) (string, error) { + switch ns { + case "env": + v, ok := os.LookupEnv(key) + if !ok { + return "", fmt.Errorf("environment variable %s is not set", key) + } + return v, nil + case "git": + return gitValue(key) + default: + return "", fmt.Errorf("unknown interpolation namespace %q (want env or git)", ns) + } +} + +func gitValue(key string) (string, error) { + var args []string + switch key { + case "branch": + args = []string{"rev-parse", "--abbrev-ref", "HEAD"} + case "sha": + args = []string{"rev-parse", "HEAD"} + case "short_sha": + args = []string{"rev-parse", "--short", "HEAD"} + default: + return "", fmt.Errorf("unknown git value %q (want branch, sha or short_sha)", key) + } + out, err := exec.Command("git", args...).Output() + if err != nil { + return "", fmt.Errorf("git %s: %w", strings.Join(args, " "), err) + } + value := strings.TrimSpace(string(out)) + if value == "" || value == "HEAD" { + return "", fmt.Errorf("git %s returned no usable value (detached HEAD?)", key) + } + return value, nil +} + +// FunctionsPath is the absolute path to the functions package. +func (c *Config) FunctionsPath() string { + return filepath.Join(c.Root, c.Project.FunctionsDir) +} + +// StatePath is a path inside the gitignored .notte directory. +func (c *Config) StatePath(parts ...string) string { + return filepath.Join(append([]string{c.Root, StateDir}, parts...)...) +} diff --git a/internal/project/config_test.go b/internal/project/config_test.go new file mode 100644 index 0000000..88db9ab --- /dev/null +++ b/internal/project/config_test.go @@ -0,0 +1,349 @@ +package project + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// write builds a project directory from a path->content map. +func write(t *testing.T, files map[string]string) string { + t.Helper() + dir := t.TempDir() + for rel, body := range files { + p := filepath.Join(dir, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +// The common case: no environments, no credentials, prod implied. If this ever +// needs more than [project] to work, the "environments are opt-in" promise has +// been broken. +func TestMinimalConfigNeedsNoEnvironments(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": "[project]\nname = \"demo\"\n", + }) + cfg, err := Load(dir) + if err != nil { + t.Fatal(err) + } + if cfg.Project.FunctionsDir != DefaultFunctionsDir { + t.Fatalf("functions_dir = %q, want %q", cfg.Project.FunctionsDir, DefaultFunctionsDir) + } + env, err := cfg.ResolveEnv("") + if err != nil { + t.Fatalf("default env must resolve without an [env.*] block: %v", err) + } + if env.APIURL != "" { + t.Fatalf("expected an empty api_url to fall through to CLI defaults, got %q", env.APIURL) + } +} + +// TOML silently ignores keys it cannot place, so a typo would otherwise be a +// setting that never applies. +func TestUnknownKeyIsRejected(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": "[project]\nname = \"demo\"\nfunctions_dirr = \"fns\"\n", + }) + _, err := Load(dir) + if err == nil { + t.Fatal("expected an error for a misspelled key") + } + if !strings.Contains(err.Error(), "functions_dirr") { + t.Fatalf("error should name the key, got %v", err) + } +} + +func TestUndeclaredEnvIsRejectedButDefaultIsNot(t *testing.T) { + dir := write(t, map[string]string{"notte.toml": "[project]\nname = \"demo\"\n"}) + cfg, err := Load(dir) + if err != nil { + t.Fatal(err) + } + if _, err := cfg.ResolveEnv("staging"); err == nil { + t.Fatal("naming an undefined env should fail") + } + if _, err := cfg.ResolveEnv(DefaultEnv); err != nil { + t.Fatalf("the default env must resolve even when undeclared: %v", err) + } +} + +func TestEnvExtendsInherits(t *testing.T) { + t.Setenv("TEST_KEY", "sk-test") + dir := write(t, map[string]string{ + "notte.toml": `[project] +name = "demo" + +[env.dev] +api_url = "https://us-dev.notte.cc" +api_key = "${env:TEST_KEY}" +headers = { "x-a" = "1" } + +[env.preview] +extends = "dev" +headers = { "x-b" = "2" } +`, + }) + cfg, err := Load(dir) + if err != nil { + t.Fatal(err) + } + env, err := cfg.ResolveEnv("preview") + if err != nil { + t.Fatal(err) + } + if env.APIURL != "https://us-dev.notte.cc" { + t.Fatalf("api_url not inherited: %q", env.APIURL) + } + if env.APIKey != "sk-test" { + t.Fatalf("api_key not inherited/expanded: %q", env.APIKey) + } + if env.Headers["x-a"] != "1" || env.Headers["x-b"] != "2" { + t.Fatalf("headers not merged: %v", env.Headers) + } +} + +func TestEnvExtendsUnknownIsRejected(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[env.preview]\nextends = \"nope\"\n", + }) + if _, err := Load(dir); err == nil { + t.Fatal("extends of an undefined env should fail") + } +} + +func TestEnvExtendsCycleIsRejected(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[env.a]\nextends=\"b\"\n\n[env.b]\nextends=\"a\"\n", + }) + err := Load2(t, dir) + if err == nil || !strings.Contains(err.Error(), "circular") { + t.Fatalf("expected a circular-extends error, got %v", err) + } +} + +// Load2 is Load, returning only the error, for terser assertions. +func Load2(t *testing.T, dir string) error { + t.Helper() + _, err := Load(dir) + return err +} + +// An unresolved reference must fail loudly. Expanding to "" produces an +// api_url of "" or a credential of "", which fails far from the cause. +func TestUnsetInterpolationIsAnError(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[env.dev]\napi_key = \"${env:DEFINITELY_NOT_SET_XYZ}\"\n", + }) + cfg, err := Load(dir) + if err != nil { + t.Fatal(err) + } + _, err = cfg.ResolveEnv("dev") + if err == nil { + t.Fatal("an unset ${env:...} must be an error, never an empty string") + } + if !strings.Contains(err.Error(), "DEFINITELY_NOT_SET_XYZ") { + t.Fatalf("error should name the variable, got %v", err) + } +} + +func TestUnknownInterpolationNamespaceIsAnError(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[env.dev]\napi_key = \"${vault:thing}\"\n", + }) + cfg, _ := Load(dir) + if _, err := cfg.ResolveEnv("dev"); err == nil { + t.Fatal("unknown namespace should be an error") + } +} + +func TestGitInterpolation(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[env.preview]\nheaders = { \"x-db-preview\" = \"${git:branch}\" }\n", + }) + cfg, err := Load(dir) + if err != nil { + t.Fatal(err) + } + env, err := cfg.ResolveEnv("preview") + if err != nil { + t.Skipf("not in a git worktree: %v", err) + } + if env.Headers["x-db-preview"] == "" { + t.Fatal("git branch did not expand") + } +} + +func TestFunctionsDirMustStayInsideTheProject(t *testing.T) { + for _, bad := range []string{"/etc", "../outside"} { + dir := write(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\nfunctions_dir = \"" + bad + "\"\n", + }) + if _, err := Load(dir); err == nil { + t.Fatalf("functions_dir %q should be rejected", bad) + } + } +} + +// Commands must work from a subdirectory, the way git does. +func TestFindWalksUp(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n", + "functions/fn/main.py": "def run():\n return 1\n", + }) + found, err := Find(filepath.Join(dir, "functions", "fn")) + if err != nil { + t.Fatal(err) + } + if resolved, _ := filepath.EvalSymlinks(found); resolved != mustEval(t, dir) { + t.Fatalf("found %q, want %q", found, dir) + } +} + +func mustEval(t *testing.T, p string) string { + t.Helper() + r, err := filepath.EvalSymlinks(p) + if err != nil { + t.Fatal(err) + } + return r +} + +func TestFindReportsWhenThereIsNoProject(t *testing.T) { + if _, err := Find(t.TempDir()); err == nil { + t.Fatal("expected an error outside a project") + } +} + +func TestCronVariablesParse(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": `[project] +name = "d" + +[functions.hello] +cron = "cron(0 9 * * ? *)" +cron_variables = { name = "scheduled", limit = 10 } +`, + }) + cfg, err := Load(dir) + if err != nil { + t.Fatal(err) + } + fn := cfg.Functions["hello"] + if fn.CronVariables["name"] != "scheduled" { + t.Fatalf("name = %v", fn.CronVariables["name"]) + } + // TOML integers decode as int64; the schedule endpoint takes any JSON value. + if got, ok := fn.CronVariables["limit"].(int64); !ok || got != 10 { + t.Fatalf("limit = %#v", fn.CronVariables["limit"]) + } +} + +// A schedule that supplies a key run() does not take is rejected by the server +// only when the schedule is set, and a cron failing at 09:00 is a bad way to +// find a typo. +func TestScheduleProblemsRejectsUnknownVariable(t *testing.T) { + fn := FunctionConfig{Cron: "cron(0 9 * * ? *)", CronVariables: map[string]any{"naem": "x"}} + problems := fn.ScheduleProblems("hello", []Param{{Name: "name", HasDefault: true}}) + if len(problems) != 1 { + t.Fatalf("got %v", problems) + } + if !strings.Contains(problems[0], "naem") || !strings.Contains(problems[0], "name") { + t.Fatalf("error should name both the typo and the real parameter: %v", problems[0]) + } +} + +// A parameter without a default cannot be filled in by the runtime, so a +// schedule that omits it can never succeed. +func TestScheduleProblemsRequiresParametersWithoutDefaults(t *testing.T) { + fn := FunctionConfig{Cron: "cron(0 9 * * ? *)"} + problems := fn.ScheduleProblems("hello", []Param{{Name: "url"}}) + if len(problems) != 1 || !strings.Contains(problems[0], "url") { + t.Fatalf("got %v", problems) + } + + fn.CronVariables = map[string]any{"url": "https://x.test"} + if problems := fn.ScheduleProblems("hello", []Param{{Name: "url"}}); len(problems) != 0 { + t.Fatalf("supplying it should satisfy the check: %v", problems) + } +} + +func TestScheduleProblemsRejectsVariablesWithoutACron(t *testing.T) { + fn := FunctionConfig{CronVariables: map[string]any{"name": "x"}} + problems := fn.ScheduleProblems("hello", []Param{{Name: "name", HasDefault: true}}) + if len(problems) != 1 || !strings.Contains(problems[0], "never used") { + t.Fatalf("got %v", problems) + } +} + +// An unscheduled function is not required to supply anything. +func TestScheduleProblemsSilentWithoutASchedule(t *testing.T) { + fn := FunctionConfig{} + if problems := fn.ScheduleProblems("hello", []Param{{Name: "url"}}); len(problems) != 0 { + t.Fatalf("got %v", problems) + } +} + +// The fields `notte functions configure` can set are declarable, so a deploy +// pushes them rather than leaving copy editable only upstream. +func TestFunctionMetadataFieldsParse(t *testing.T) { + dir := write(t, map[string]string{ + "notte.toml": `[project] +name = "d" + +[functions.hello] +name = "Hello" +description = "Says hello" +domain = "example.com" +instructions = "Takes ~30s." +self_healing = true +`, + }) + cfg, err := Load(dir) + if err != nil { + t.Fatal(err) + } + fn := cfg.Functions["hello"] + if fn.Domain != "example.com" || fn.Instructions != "Takes ~30s." { + t.Fatalf("got %+v", fn) + } + if fn.SelfHealing == nil || !*fn.SelfHealing { + t.Fatalf("self_healing = %v", fn.SelfHealing) + } +} + +// self_healing is a pointer so "unset" differs from "explicitly false". +// Turning a feature off because a config did not mention it would be a +// surprising deploy. +func TestSelfHealingDistinguishesUnsetFromFalse(t *testing.T) { + unset := write(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[functions.hello]\ndescription = \"x\"\n", + }) + cfg, err := Load(unset) + if err != nil { + t.Fatal(err) + } + if cfg.Functions["hello"].SelfHealing != nil { + t.Fatal("an unmentioned self_healing must stay nil, not become false") + } + + explicit := write(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[functions.hello]\nself_healing = false\n", + }) + cfg, err = Load(explicit) + if err != nil { + t.Fatal(err) + } + sh := cfg.Functions["hello"].SelfHealing + if sh == nil || *sh { + t.Fatalf("an explicit false must be sent, got %v", sh) + } +} diff --git a/internal/project/discover.go b/internal/project/discover.go new file mode 100644 index 0000000..ff15bdd --- /dev/null +++ b/internal/project/discover.go @@ -0,0 +1,380 @@ +package project + +import ( + "fmt" + "io/fs" + "os" + "path" + "regexp" + "sort" + "strings" +) + +// Function is one deployable unit found on disk. +type Function struct { + // Name is the directory or file stem, and the identity used in + // notte.toml. It never contains a function id: ids are per-environment, + // so one in a filename would tie the tree to a single environment. + Name string + // Entrypoint is slash-separated and relative to the functions directory, + // which is what the bundler takes. + Entrypoint string + // Dir reports the directory form, which can carry helpers and tests. + Dir bool +} + +// Entrypoint filenames. The name of the file declares what a directory is, +// which keeps discovery to one rule and lets a connector's two roles live +// beside each other without a second reserved directory name. +const ( + // EntrypointName marks a plain function. + // + // Not function.py, which is redundant inside functions//; not + // index.py, which is a JavaScript import; not route.py, since a Notte + // function has exactly one run() and there is no route/handler split. + EntrypointName = "main.py" + // LoginEntrypoint and VerifierEntrypoint mark a managed-auth connector. + // Both must be present: a connector is the pair, and the API deploys them + // transactionally. + LoginEntrypoint = "login.py" + VerifierEntrypoint = "verifier.py" +) + +// Role is which half of a connector an entrypoint is. +type Role string + +const ( + RoleLogin Role = "login" + RoleVerifier Role = "verifier" +) + +// Connector is a managed-auth connector: two functions plus catalog metadata, +// deployed together. +type Connector struct { + // Name is the directory name, and the catalog slug. Never the path: a + // grouping directory such as functions/auth/ must not leak into a slug + // that is globally unique across the catalog. + Name string + // Dir is slash-separated and relative to the functions directory. + Dir string + // Login and Verifier are entrypoints, in the form the bundler takes. + Login string + Verifier string +} + +// Entrypoints returns the connector's two roles in a stable order. +func (c Connector) Entrypoints() []struct { + Role Role + Entrypoint string +} { + return []struct { + Role Role + Entrypoint string + }{ + {RoleLogin, c.Login}, + {RoleVerifier, c.Verifier}, + } +} + +var nameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]*$`) + +// Discover finds every function and connector under the functions directory. +// +// The rule is one sentence, and it is depth-independent: any directory whose +// name does not start with an underscore is a function if it contains main.py, +// or a connector if it contains login.py and verifier.py. A bare .py is +// a single-file function. +// +// Depth-independence is what lets grouping be a convention rather than a rule. +// functions/bluesky/ and functions/auth/bluesky/ both work, so a project can +// stay flat while it is small and group when it is not, without the CLI +// reserving a directory name or anyone migrating. +func Discover(cfg *Config) ([]Function, error) { + functions, _, err := DiscoverAll(cfg) + return functions, err +} + +// DiscoverAll returns functions and connectors together. +func DiscoverAll(cfg *Config) ([]Function, []Connector, error) { + root := cfg.FunctionsPath() + var functions []Function + var connectors []Connector + + if err := walkUnits(root, "", cfg, &functions, &connectors); err != nil { + return nil, nil, err + } + + sort.Slice(functions, func(i, j int) bool { return functions[i].Name < functions[j].Name }) + sort.Slice(connectors, func(i, j int) bool { return connectors[i].Name < connectors[j].Name }) + + if err := checkDuplicateNames(functions, connectors); err != nil { + return nil, nil, err + } + if err := checkUnknownConfig(cfg, functions, connectors); err != nil { + return nil, nil, err + } + return functions, connectors, nil +} + +// walkUnits descends until it finds a unit, then stops. A directory that is a +// function or a connector is not searched further: its subdirectories are its +// own helpers, not more units. +func walkUnits(root, rel string, cfg *Config, functions *[]Function, connectors *[]Connector) error { + entries, err := os.ReadDir(path.Join(root, rel)) + if err != nil { + return fmt.Errorf("read %s: %w", path.Join(cfg.Project.FunctionsDir, rel), err) + } + + for _, e := range entries { + name := e.Name() + if strings.HasPrefix(name, "_") || strings.HasPrefix(name, ".") { + continue + } + child := path.Join(rel, name) + + if !e.IsDir() { + // A bare .py is a single-file function only at the top level. + // Deeper down it belongs to whatever contains it — a helper inside + // a unit, or loose Python inside a directory that has not been + // finished. Reading those as functions would make an unfinished + // directory look populated and deploy its helpers. + if rel != "" || !strings.HasSuffix(name, ".py") || IsTestFile(name) { + continue + } + stem := strings.TrimSuffix(name, ".py") + if err := validateName(stem); err != nil { + return err + } + *functions = append(*functions, Function{Name: stem, Entrypoint: child}) + continue + } + + switch kind, err := classify(root, child); { + case err != nil: + return err + case kind == unitFunction: + if err := validateName(name); err != nil { + return err + } + *functions = append(*functions, Function{ + Name: name, Entrypoint: path.Join(child, EntrypointName), Dir: true, + }) + case kind == unitConnector: + if err := validateName(name); err != nil { + return err + } + *connectors = append(*connectors, Connector{ + Name: name, + Dir: child, + Login: path.Join(child, LoginEntrypoint), + Verifier: path.Join(child, VerifierEntrypoint), + }) + default: + // Not a unit itself, so it may be a grouping directory. Depth + // independence means such a directory has to be searched rather + // than rejected — but a grouping directory holds units, not loose + // Python. One with .py files and nothing beneath it is an + // unfinished unit, and skipping it silently would deploy nothing + // and say nothing. + before := len(*functions) + len(*connectors) + if err := walkUnits(root, child, cfg, functions, connectors); err != nil { + return err + } + if len(*functions)+len(*connectors) == before && containsPython(root, child) { + return fmt.Errorf( + "%s has Python files but no %s, and no %s/%s pair — "+ + "add an entrypoint, or rename it to _%s if it is shared code", + child, EntrypointName, LoginEntrypoint, VerifierEntrypoint, name) + } + } + } + return nil +} + +// containsPython reports whether a directory holds .py files directly, +// which is what separates an unfinished unit from a grouping directory. +func containsPython(root, rel string) bool { + entries, err := os.ReadDir(path.Join(root, rel)) + if err != nil { + return false + } + for _, e := range entries { + if !e.IsDir() && strings.HasSuffix(e.Name(), ".py") && e.Name() != "__init__.py" { + return true + } + } + return false +} + +type unitKind int + +const ( + unitNone unitKind = iota + unitFunction + unitConnector +) + +// classify decides what a directory is from the entrypoints it contains. +// +// Half a connector is an error rather than a grouping directory: someone who +// wrote login.py and not verifier.py has an unfinished connector, and silently +// treating it as a folder deploys neither and says nothing. +func classify(root, rel string) (unitKind, error) { + has := func(name string) bool { + _, err := os.Stat(path.Join(root, rel, name)) + return err == nil + } + + main, login, verifier := has(EntrypointName), has(LoginEntrypoint), has(VerifierEntrypoint) + + switch { + case main && (login || verifier): + return unitNone, fmt.Errorf("%s has both %s and a connector entrypoint; it must be one or the other", rel, EntrypointName) + case main: + return unitFunction, nil + case login && verifier: + return unitConnector, nil + case login: + return unitNone, fmt.Errorf("%s has %s but no %s — a connector needs both, and they deploy together", + rel, LoginEntrypoint, VerifierEntrypoint) + case verifier: + return unitNone, fmt.Errorf("%s has %s but no %s — a connector needs both, and they deploy together", + rel, VerifierEntrypoint, LoginEntrypoint) + } + return unitNone, nil +} + +// checkDuplicateNames rejects two units claiming one name. +// +// Grouping directories make this reachable: functions/a/report/ and +// functions/b/report/ both deploy as "report", and for connectors the name is +// a catalog slug that is unique across every workspace. +func checkDuplicateNames(functions []Function, connectors []Connector) error { + seen := map[string]string{} + claim := func(name, where string) error { + if prev, dup := seen[name]; dup { + return fmt.Errorf("%s and %s both deploy as %q; rename one", prev, where, name) + } + seen[name] = where + return nil + } + for _, f := range functions { + if err := claim(f.Name, f.Entrypoint); err != nil { + return err + } + } + for _, c := range connectors { + if err := claim(c.Name, c.Dir); err != nil { + return err + } + } + return nil +} + +func validateName(name string) error { + if !nameRe.MatchString(name) { + return fmt.Errorf("function name %q must be lowercase alphanumeric with underscores or dashes", name) + } + return nil +} + +// checkUnknownConfig rejects a [functions.x] or [connectors.x] block with no +// unit x. +// +// Almost always a typo or a rename, and the symptom otherwise is a cron or a +// catalog description that silently never applies. +func checkUnknownConfig(cfg *Config, functions []Function, connectors []Connector) error { + haveFn := make(map[string]bool, len(functions)) + for _, f := range functions { + haveFn[f.Name] = true + } + haveConn := make(map[string]bool, len(connectors)) + for _, c := range connectors { + haveConn[c.Name] = true + } + + var problems []string + var unknown []string + for name := range cfg.Functions { + if !haveFn[name] { + unknown = append(unknown, name) + } + } + // Sidecars need no such check: one lives inside the unit it configures, so + // it cannot name a unit that does not exist. + sort.Strings(unknown) + if len(unknown) > 0 { + problems = append(problems, fmt.Sprintf("configures function(s) that do not exist: %s", strings.Join(unknown, ", "))) + } + + unknown = nil + for name := range cfg.Connectors { + if !haveConn[name] { + unknown = append(unknown, name) + } + } + sort.Strings(unknown) + if len(unknown) > 0 { + problems = append(problems, fmt.Sprintf("configures connector(s) that do not exist: %s", strings.Join(unknown, ", "))) + } + + if len(problems) == 0 { + return nil + } + return fmt.Errorf("%s %s", ConfigName, strings.Join(problems, "; ")) +} + +// IsTestFile reports whether a path is a test, which is never bundled. +func IsTestFile(p string) bool { + base := path.Base(p) + return strings.HasPrefix(base, "test_") || strings.HasSuffix(base, "_test.py") +} + +// Select filters functions by a target: a name, a glob, a path, or "all". +func Select(functions []Function, target string) ([]Function, error) { + if target == "" || target == "all" { + return functions, nil + } + // A path is accepted so shell completion on the tree works. + target = strings.TrimSuffix(strings.TrimSuffix(strings.Trim(target, "/"), "/"+EntrypointName), ".py") + if i := strings.LastIndex(target, "/"); i >= 0 { + target = target[i+1:] + } + + var out []Function + for _, f := range functions { + ok, err := path.Match(target, f.Name) + if err != nil { + return nil, fmt.Errorf("invalid pattern %q: %w", target, err) + } + if ok { + out = append(out, f) + } + } + if len(out) == 0 { + names := make([]string, 0, len(functions)) + for _, f := range functions { + names = append(names, f.Name) + } + return nil, fmt.Errorf("no function matches %q (have: %s)", target, strings.Join(names, ", ")) + } + return out, nil +} + +// Sources lists every .py file the functions directory contributes, excluding +// tests. Used to report what a shared-module edit would touch. +func Sources(fsys fs.FS) ([]string, error) { + var out []string + err := fs.WalkDir(fsys, ".", func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() || !strings.HasSuffix(p, ".py") || IsTestFile(p) { + return nil + } + out = append(out, p) + return nil + }) + sort.Strings(out) + return out, err +} diff --git a/internal/project/discover_conn_test.go b/internal/project/discover_conn_test.go new file mode 100644 index 0000000..a8d7dbd --- /dev/null +++ b/internal/project/discover_conn_test.go @@ -0,0 +1,180 @@ +package project + +import ( + "strings" + "testing" +) + +func discoverAll(t *testing.T, files map[string]string) ([]Function, []Connector, error) { + t.Helper() + if _, ok := files["notte.toml"]; !ok { + files["notte.toml"] = "[project]\nname = \"demo\"\n" + } + dir := write(t, files) + cfg, err := Load(dir) + if err != nil { + return nil, nil, err + } + return DiscoverAll(cfg) +} + +// A directory with login.py and verifier.py is a connector. The filename +// declares the kind, so the two roles sit beside each other without needing a +// second reserved directory name. +func TestConnectorIsADirectoryWithBothRoles(t *testing.T) { + fns, conns, err := discoverAll(t, map[string]string{ + "functions/bluesky/login.py": "def run(session_id: str):\n return 1\n", + "functions/bluesky/verifier.py": "def run(session_id: str):\n return 1\n", + "functions/bluesky/helpers.py": "def h():\n return 1\n", + "functions/scraper/main.py": "def run():\n return 1\n", + }) + if err != nil { + t.Fatal(err) + } + if len(fns) != 1 || fns[0].Name != "scraper" { + t.Fatalf("functions = %+v", fns) + } + if len(conns) != 1 || conns[0].Name != "bluesky" { + t.Fatalf("connectors = %+v", conns) + } + if conns[0].Login != "bluesky/login.py" || conns[0].Verifier != "bluesky/verifier.py" { + t.Fatalf("entrypoints = %q %q", conns[0].Login, conns[0].Verifier) + } +} + +// Half a connector is an unfinished connector, not a folder. Treating it as a +// grouping directory would deploy neither role and say nothing. +func TestHalfAConnectorIsAnError(t *testing.T) { + for _, missing := range []struct{ have, want string }{ + {"login.py", "verifier.py"}, + {"verifier.py", "login.py"}, + } { + _, _, err := discoverAll(t, map[string]string{ + "functions/bluesky/" + missing.have: "def run(session_id: str):\n return 1\n", + }) + if err == nil { + t.Fatalf("%s alone should be an error", missing.have) + } + if !strings.Contains(err.Error(), missing.want) { + t.Errorf("error should name the missing half %q: %v", missing.want, err) + } + } +} + +// Grouping is a convention, not a rule: both depths work, so a project can +// stay flat while small and group when it is not. +func TestDiscoveryIsDepthIndependent(t *testing.T) { + fns, conns, err := discoverAll(t, map[string]string{ + "functions/flat/main.py": "def run():\n return 1\n", + "functions/group/nested/main.py": "def run():\n return 1\n", + "functions/auth/bluesky/login.py": "def run(session_id: str):\n return 1\n", + "functions/auth/bluesky/verifier.py": "def run(session_id: str):\n return 1\n", + }) + if err != nil { + t.Fatal(err) + } + names := map[string]string{} + for _, f := range fns { + names[f.Name] = f.Entrypoint + } + if names["flat"] != "flat/main.py" || names["nested"] != "group/nested/main.py" { + t.Fatalf("functions = %+v", fns) + } + // The slug is the directory name, never the path: a grouping directory + // must not leak into a globally unique catalog slug. + if len(conns) != 1 || conns[0].Name != "bluesky" { + t.Fatalf("connector name should be the directory, not the path: %+v", conns) + } + if conns[0].Login != "auth/bluesky/login.py" { + t.Fatalf("entrypoint = %q", conns[0].Login) + } +} + +// A unit's subdirectories are its own helpers, not more units. +func TestAUnitIsNotSearchedFurther(t *testing.T) { + fns, _, err := discoverAll(t, map[string]string{ + "functions/outer/main.py": "def run():\n return 1\n", + "functions/outer/inner/main.py": "def run():\n return 1\n", + }) + if err != nil { + t.Fatal(err) + } + if len(fns) != 1 || fns[0].Name != "outer" { + t.Fatalf("a nested main.py inside a function is a helper, not a unit: %+v", fns) + } +} + +// Grouping makes this reachable, and for connectors the name is a catalog slug +// unique across every workspace. +func TestTwoUnitsCannotClaimOneName(t *testing.T) { + _, _, err := discoverAll(t, map[string]string{ + "functions/a/report/main.py": "def run():\n return 1\n", + "functions/b/report/main.py": "def run():\n return 1\n", + }) + if err == nil { + t.Fatal("two functions deploying as the same name must be an error") + } + if !strings.Contains(err.Error(), "report") { + t.Errorf("error should name the collision: %v", err) + } +} + +func TestDirectoryWithBothKindsIsAnError(t *testing.T) { + _, _, err := discoverAll(t, map[string]string{ + "functions/confused/main.py": "def run():\n return 1\n", + "functions/confused/login.py": "def run(session_id: str):\n return 1\n", + }) + if err == nil || !strings.Contains(err.Error(), "one or the other") { + t.Fatalf("got %v", err) + } +} + +func TestUnknownConnectorConfigIsRejected(t *testing.T) { + _, _, err := discoverAll(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[connectors.typo]\nname = \"Typo\"\ndomain = \"x.test\"\n", + "functions/bluesky/login.py": "def run(session_id: str):\n return 1\n", + "functions/bluesky/verifier.py": "def run(session_id: str):\n return 1\n", + }) + if err == nil || !strings.Contains(err.Error(), "typo") { + t.Fatalf("a [connectors.x] block with no connector x should be rejected: %v", err) + } +} + +func TestConnectorConfigValidation(t *testing.T) { + ok := ConnectorConfig{Name: "Bluesky", Domain: "bsky.app", Color: "#0085ff", ProxyCountry: "us"} + if p := ok.Validate("bluesky"); len(p) != 0 { + t.Fatalf("valid config reported %v", p) + } + bad := ConnectorConfig{Color: "0085ff", ProxyCountry: "usa"} + p := bad.Validate("bluesky") + if len(p) != 4 { + t.Fatalf("expected name, domain, color and proxy_country problems, got %v", p) + } +} + +// A bare .py is a single-file function only at the top level. Deeper down it +// belongs to whatever contains it, or an unfinished directory looks populated +// and its helpers get deployed. +func TestBarePythonIsAFunctionOnlyAtTheTopLevel(t *testing.T) { + fns, _, err := discoverAll(t, map[string]string{ + "functions/quick.py": "def run():\n return 1\n", + "functions/group/deep/main.py": "def run():\n return 1\n", + "functions/group/helper.py": "def h():\n return 1\n", + }) + if err != nil { + t.Fatal(err) + } + got := map[string]bool{} + for _, f := range fns { + got[f.Name] = true + } + if !got["quick"] { + t.Error("a top-level .py should be a function") + } + if got["helper"] { + t.Error("a .py inside a grouping directory is a helper, not a function") + } + if !got["deep"] { + t.Error("the nested directory function should still be found") + } +} diff --git a/internal/project/discover_test.go b/internal/project/discover_test.go new file mode 100644 index 0000000..857f456 --- /dev/null +++ b/internal/project/discover_test.go @@ -0,0 +1,167 @@ +package project + +import ( + "strings" + "testing" +) + +func loadAndDiscover(t *testing.T, files map[string]string) ([]Function, error) { + t.Helper() + if _, ok := files["notte.toml"]; !ok { + files["notte.toml"] = "[project]\nname = \"demo\"\n" + } + dir := write(t, files) + cfg, err := Load(dir) + if err != nil { + t.Fatalf("load: %v", err) + } + return Discover(cfg) +} + +func names(fns []Function) []string { + out := make([]string, len(fns)) + for i, f := range fns { + out[i] = f.Name + } + return out +} + +func TestDiscoverDirectoryAndSingleFileForms(t *testing.T) { + fns, err := loadAndDiscover(t, map[string]string{ + "functions/amazon_search/main.py": "def run():\n return 1\n", + "functions/amazon_search/parse.py": "def clean(s):\n return s\n", + "functions/quick_check.py": "def run():\n return 1\n", + }) + if err != nil { + t.Fatal(err) + } + if got := names(fns); len(got) != 2 || got[0] != "amazon_search" || got[1] != "quick_check" { + t.Fatalf("got %v", got) + } + if !fns[0].Dir || fns[1].Dir { + t.Fatalf("directory/file forms not distinguished: %+v", fns) + } + if fns[0].Entrypoint != "amazon_search/main.py" || fns[1].Entrypoint != "quick_check.py" { + t.Fatalf("entrypoints: %q, %q", fns[0].Entrypoint, fns[1].Entrypoint) + } +} + +// The underscore prefix is the whole configuration story for shared code. +func TestUnderscoreIsNotAFunction(t *testing.T) { + fns, err := loadAndDiscover(t, map[string]string{ + "functions/_shared/http.py": "def fetch():\n return 1\n", + "functions/_shared/__init__.py": "", + "functions/_helper.py": "x = 1\n", + "functions/real/main.py": "def run():\n return 1\n", + }) + if err != nil { + t.Fatal(err) + } + if got := names(fns); len(got) != 1 || got[0] != "real" { + t.Fatalf("got %v, want [real]", got) + } +} + +// A directory with no main.py is a mistake, and skipping it silently would +// deploy nothing and say nothing. +func TestDirectoryWithoutEntrypointIsAnError(t *testing.T) { + _, err := loadAndDiscover(t, map[string]string{ + "functions/halfdone/parse.py": "def clean(s):\n return s\n", + }) + if err == nil { + t.Fatal("expected an error") + } + if !strings.Contains(err.Error(), "main.py") || !strings.Contains(err.Error(), "_halfdone") { + t.Fatalf("error should suggest both fixes, got %v", err) + } +} + +func TestInvalidFunctionNameIsRejected(t *testing.T) { + _, err := loadAndDiscover(t, map[string]string{ + "functions/Amazon Search/main.py": "def run():\n return 1\n", + }) + if err == nil { + t.Fatal("expected an error for a name with a space and capitals") + } +} + +// A [functions.x] block with no function x is almost always a typo or a +// rename, and the symptom is otherwise a cron that silently never applies. +func TestConfigForMissingFunctionIsRejected(t *testing.T) { + _, err := loadAndDiscover(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[functions.typo_name]\ncron = \"cron(0 9 * * ? *)\"\n", + "functions/real/main.py": "def run():\n return 1\n", + }) + if err == nil { + t.Fatal("expected an error") + } + if !strings.Contains(err.Error(), "typo_name") { + t.Fatalf("error should name the stale block, got %v", err) + } +} + +func TestDiscoverIsSorted(t *testing.T) { + fns, err := loadAndDiscover(t, map[string]string{ + "functions/zebra.py": "def run():\n return 1\n", + "functions/alpha.py": "def run():\n return 1\n", + "functions/mid.py": "def run():\n return 1\n", + }) + if err != nil { + t.Fatal(err) + } + if got := names(fns); got[0] != "alpha" || got[1] != "mid" || got[2] != "zebra" { + t.Fatalf("not sorted: %v", got) + } +} + +func TestSelectByNameGlobAndPath(t *testing.T) { + fns := []Function{ + {Name: "amazon_search", Entrypoint: "amazon_search/main.py", Dir: true}, + {Name: "amazon_deals", Entrypoint: "amazon_deals/main.py", Dir: true}, + {Name: "other", Entrypoint: "other.py"}, + } + cases := []struct { + target string + want int + }{ + {"", 3}, + {"all", 3}, + {"other", 1}, + {"amazon_*", 2}, + {"functions/amazon_search", 1}, + {"functions/amazon_search/main.py", 1}, + {"functions/other.py", 1}, + } + for _, tc := range cases { + got, err := Select(fns, tc.target) + if err != nil { + t.Fatalf("%q: %v", tc.target, err) + } + if len(got) != tc.want { + t.Fatalf("%q selected %d, want %d", tc.target, len(got), tc.want) + } + } +} + +func TestSelectUnknownListsWhatExists(t *testing.T) { + _, err := Select([]Function{{Name: "alpha"}}, "nope") + if err == nil { + t.Fatal("expected an error") + } + if !strings.Contains(err.Error(), "alpha") { + t.Fatalf("error should list available functions, got %v", err) + } +} + +func TestIsTestFile(t *testing.T) { + for _, p := range []string{"fn/test_main.py", "fn/main_test.py"} { + if !IsTestFile(p) { + t.Errorf("%q should be a test file", p) + } + } + for _, p := range []string{"fn/main.py", "fn/latest.py", "fn/contest.py"} { + if IsTestFile(p) { + t.Errorf("%q should not be a test file", p) + } + } +} diff --git a/internal/project/lock.go b/internal/project/lock.go new file mode 100644 index 0000000..f2e03b0 --- /dev/null +++ b/internal/project/lock.go @@ -0,0 +1,165 @@ +package project + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" +) + +// LockVersion is the schema version of notte.lock.json. +const LockVersion = 1 + +// Lock maps functions in the tree to the function they became, per environment. +// +// Path is the identity and ids live per environment, which is what lets one +// tree serve dev, staging and prod. marketplace established both halves: an id +// in a filename ties the tree to one environment, and a tree-wide content hash +// means pushing to prod silently marks dev up to date. +type Lock struct { + Version int `json:"version"` + Functions []LockEntry `json:"functions"` +} + +// LockEntry is one function's per-environment state. +type LockEntry struct { + // Path is the entrypoint relative to the functions directory. + Path string `json:"path"` + // Envs is keyed by environment name. + Envs map[string]EnvState `json:"envs"` +} + +// EnvState is what one environment knows about one function. +type EnvState struct { + FunctionID string `json:"function_id"` + // Version is the server-assigned version last seen, e.g. v20260821_162138. + Version string `json:"version,omitempty"` + Versions []string `json:"versions,omitempty"` + + // SourceSHA256 covers the set of contributing source files and answers + // "does this need deploying?". + SourceSHA256 string `json:"source_sha256"` + // ArtifactSHA256 covers the bundled bytes and answers "what changed + // upstream?". Two hashes rather than one because bundling is lossy: the + // artifact cannot be turned back into the sources that produced it. + ArtifactSHA256 string `json:"artifact_sha256"` +} + +// LoadLock reads notte.lock.json. A missing lock is an empty one, not an +// error: that is a project that has never deployed. +func LoadLock(dir string) (*Lock, error) { + raw, err := os.ReadFile(filepath.Join(dir, LockName)) + if os.IsNotExist(err) { + return &Lock{Version: LockVersion}, nil + } + if err != nil { + return nil, fmt.Errorf("read %s: %w", LockName, err) + } + + var lock Lock + if err := json.Unmarshal(raw, &lock); err != nil { + return nil, fmt.Errorf("%s: %w", LockName, err) + } + if lock.Version > LockVersion { + return nil, fmt.Errorf("%s was written by a newer notte (schema v%d, this build understands v%d)", + LockName, lock.Version, LockVersion) + } + lock.Version = LockVersion + return &lock, nil +} + +// Save writes the lock, sorted, one function per line. +// +// The formatting is deliberate. marketplace carries 2,049 entries and stays +// reviewable in a diff only because each is a single line, so a change shows +// as one changed line rather than a reflowed block. +func (l *Lock) Save(dir string) error { + sort.Slice(l.Functions, func(i, j int) bool { return l.Functions[i].Path < l.Functions[j].Path }) + + var buf bytes.Buffer + buf.WriteString("{\n") + fmt.Fprintf(&buf, " \"version\": %d,\n", l.Version) + buf.WriteString(" \"functions\": [\n") + for i, entry := range l.Functions { + line, err := json.Marshal(entry) + if err != nil { + return fmt.Errorf("encode %s: %w", entry.Path, err) + } + buf.WriteString(" ") + buf.Write(line) + if i < len(l.Functions)-1 { + buf.WriteByte(',') + } + buf.WriteByte('\n') + } + buf.WriteString(" ]\n}\n") + + return os.WriteFile(filepath.Join(dir, LockName), buf.Bytes(), 0o644) +} + +// Entry returns the entry for a path, or nil. +func (l *Lock) Entry(path string) *LockEntry { + for i := range l.Functions { + if l.Functions[i].Path == path { + return &l.Functions[i] + } + } + return nil +} + +// State returns what env knows about path. +func (l *Lock) State(path, env string) (EnvState, bool) { + entry := l.Entry(path) + if entry == nil { + return EnvState{}, false + } + st, ok := entry.Envs[env] + return st, ok +} + +// Record stores the outcome of a write to env. +// +// The content hashes always advance to what was pushed, even when the caller +// could not read the version back. marketplace learned this the hard way: +// tying the hash to a successful read-back meant a transient error on the +// confirmation request minted a duplicate upstream version on the next run. +// Stale version strings are recoverable; a re-push is not. +func (l *Lock) Record(path, env string, state EnvState) { + entry := l.Entry(path) + if entry == nil { + l.Functions = append(l.Functions, LockEntry{Path: path, Envs: map[string]EnvState{}}) + entry = &l.Functions[len(l.Functions)-1] + } + if entry.Envs == nil { + entry.Envs = map[string]EnvState{} + } + prev := entry.Envs[env] + if state.Version == "" { + state.Version = prev.Version + } + if state.Versions == nil { + state.Versions = prev.Versions + } + entry.Envs[env] = state +} + +// Prune drops entries whose path is no longer in the tree. +// +// Only safe to call after a complete walk. A partial run — a --limit, or a +// failed download — must never reach this, or absence gets read as deletion. +func (l *Lock) Prune(known map[string]bool) []string { + var dropped []string + kept := l.Functions[:0] + for _, entry := range l.Functions { + if known[entry.Path] { + kept = append(kept, entry) + continue + } + dropped = append(dropped, entry.Path) + } + l.Functions = kept + sort.Strings(dropped) + return dropped +} diff --git a/internal/project/lock_test.go b/internal/project/lock_test.go new file mode 100644 index 0000000..f8aa271 --- /dev/null +++ b/internal/project/lock_test.go @@ -0,0 +1,174 @@ +package project + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// A project that has never deployed has no lock, which is not an error. +func TestMissingLockIsEmptyNotAnError(t *testing.T) { + lock, err := LoadLock(t.TempDir()) + if err != nil { + t.Fatal(err) + } + if len(lock.Functions) != 0 || lock.Version != LockVersion { + t.Fatalf("got %+v", lock) + } +} + +func TestRecordAndReadBackPerEnvironment(t *testing.T) { + lock := &Lock{Version: LockVersion} + lock.Record("fn/main.py", "prod", EnvState{FunctionID: "prod-id", SourceSHA256: "aaa", ArtifactSHA256: "111"}) + lock.Record("fn/main.py", "dev", EnvState{FunctionID: "dev-id", SourceSHA256: "bbb", ArtifactSHA256: "222"}) + + prod, ok := lock.State("fn/main.py", "prod") + if !ok || prod.FunctionID != "prod-id" || prod.SourceSHA256 != "aaa" { + t.Fatalf("prod = %+v ok=%v", prod, ok) + } + dev, _ := lock.State("fn/main.py", "dev") + if dev.FunctionID != "dev-id" { + t.Fatalf("dev = %+v", dev) + } + if len(lock.Functions) != 1 { + t.Fatalf("two environments should share one entry, got %d", len(lock.Functions)) + } +} + +// The property that lets one tree serve every environment: deploying to prod +// must not mark dev as up to date. +func TestPushingToOneEnvironmentDoesNotTouchAnother(t *testing.T) { + lock := &Lock{Version: LockVersion} + lock.Record("fn/main.py", "dev", EnvState{FunctionID: "dev-id", SourceSHA256: "old"}) + lock.Record("fn/main.py", "prod", EnvState{FunctionID: "prod-id", SourceSHA256: "new"}) + + dev, _ := lock.State("fn/main.py", "dev") + if dev.SourceSHA256 != "old" { + t.Fatalf("dev hash moved to %q when prod was written", dev.SourceSHA256) + } +} + +// Version metadata is allowed to go stale; the content hash is not. Tying the +// hash to a successful read-back is what minted duplicate upstream versions in +// marketplace when the confirmation request failed. +func TestRecordKeepsPreviousVersionWhenReadBackFailed(t *testing.T) { + lock := &Lock{Version: LockVersion} + lock.Record("fn/main.py", "prod", EnvState{ + FunctionID: "id", Version: "v1", Versions: []string{"v1"}, SourceSHA256: "aaa", + }) + // A write that landed, but whose follow-up read failed: no version known. + lock.Record("fn/main.py", "prod", EnvState{FunctionID: "id", SourceSHA256: "bbb"}) + + st, _ := lock.State("fn/main.py", "prod") + if st.SourceSHA256 != "bbb" { + t.Fatalf("hash must advance to what was pushed, got %q", st.SourceSHA256) + } + if st.Version != "v1" || len(st.Versions) != 1 { + t.Fatalf("version metadata should survive as stale, got %+v", st) + } +} + +func TestSaveAndReload(t *testing.T) { + dir := t.TempDir() + lock := &Lock{Version: LockVersion} + lock.Record("b/main.py", "prod", EnvState{FunctionID: "b", SourceSHA256: "2", ArtifactSHA256: "22"}) + lock.Record("a/main.py", "prod", EnvState{FunctionID: "a", SourceSHA256: "1", ArtifactSHA256: "11"}) + if err := lock.Save(dir); err != nil { + t.Fatal(err) + } + + back, err := LoadLock(dir) + if err != nil { + t.Fatal(err) + } + if len(back.Functions) != 2 || back.Functions[0].Path != "a/main.py" { + t.Fatalf("not sorted on save: %+v", back.Functions) + } + st, _ := back.State("b/main.py", "prod") + if st.ArtifactSHA256 != "22" { + t.Fatalf("round trip lost data: %+v", st) + } +} + +// One function per line is what keeps a 2,000-entry lock reviewable: a change +// shows as one changed line rather than a reflowed block. +func TestSaveWritesOneFunctionPerLine(t *testing.T) { + dir := t.TempDir() + lock := &Lock{Version: LockVersion} + for _, p := range []string{"a.py", "b.py", "c.py"} { + lock.Record(p, "prod", EnvState{FunctionID: p, SourceSHA256: "x"}) + } + if err := lock.Save(dir); err != nil { + t.Fatal(err) + } + raw, err := os.ReadFile(filepath.Join(dir, LockName)) + if err != nil { + t.Fatal(err) + } + entries := 0 + for _, line := range strings.Split(string(raw), "\n") { + if strings.HasPrefix(strings.TrimSpace(line), `{"path"`) { + entries++ + } + } + if entries != 3 { + t.Fatalf("expected 3 single-line entries, got %d:\n%s", entries, raw) + } +} + +func TestSaveIsDeterministic(t *testing.T) { + build := func() string { + dir := t.TempDir() + lock := &Lock{Version: LockVersion} + lock.Record("z.py", "prod", EnvState{FunctionID: "z", SourceSHA256: "1"}) + lock.Record("a.py", "dev", EnvState{FunctionID: "a", SourceSHA256: "2"}) + lock.Record("a.py", "prod", EnvState{FunctionID: "a2", SourceSHA256: "3"}) + if err := lock.Save(dir); err != nil { + t.Fatal(err) + } + raw, _ := os.ReadFile(filepath.Join(dir, LockName)) + return string(raw) + } + first := build() + for i := 0; i < 5; i++ { + if build() != first { + t.Fatal("lock output is not stable across runs") + } + } +} + +func TestLockFromNewerSchemaIsRejected(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, LockName), []byte(`{"version":99,"functions":[]}`), 0o644); err != nil { + t.Fatal(err) + } + _, err := LoadLock(dir) + if err == nil || !strings.Contains(err.Error(), "newer notte") { + t.Fatalf("expected a schema-version error, got %v", err) + } +} + +func TestPruneDropsOnlyUnknownPaths(t *testing.T) { + lock := &Lock{Version: LockVersion} + for _, p := range []string{"keep.py", "gone.py"} { + lock.Record(p, "prod", EnvState{FunctionID: p}) + } + dropped := lock.Prune(map[string]bool{"keep.py": true}) + if len(dropped) != 1 || dropped[0] != "gone.py" { + t.Fatalf("dropped = %v", dropped) + } + if len(lock.Functions) != 1 || lock.Functions[0].Path != "keep.py" { + t.Fatalf("remaining = %+v", lock.Functions) + } +} + +func TestCorruptLockIsReportedNotIgnored(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, LockName), []byte("{not json"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := LoadLock(dir); err == nil { + t.Fatal("a corrupt lock must be an error, not silently an empty one") + } +} diff --git a/internal/project/sidecar.go b/internal/project/sidecar.go new file mode 100644 index 0000000..8b1e757 --- /dev/null +++ b/internal/project/sidecar.go @@ -0,0 +1,128 @@ +package project + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/BurntSushi/toml" +) + +// Sidecar filenames. A unit may carry its configuration beside its code +// instead of in the project file, which is what makes a tree of thousands +// workable: adding one touches a single directory rather than appending to a +// file everyone else is also editing. +// +// Deliberately not named notte.toml. Find walks up looking for that name, so a +// sidecar sharing it would make any command run from inside a function treat +// that function's directory as the project root. +const ( + // FunctionSidecar sits inside a directory function, beside main.py. + FunctionSidecar = "function.toml" + // ConnectorSidecar sits inside a connector, beside login.py. + ConnectorSidecar = "connector.toml" +) + +// SingleFileSidecar is the sidecar for functions/.py, which has no +// directory to put one in. Matches the convention marketplace already uses, +// where list_event_exhibitors.toml sits beside list_event_exhibitors.py. +func SingleFileSidecar(entrypoint string) string { + return strings.TrimSuffix(entrypoint, ".py") + ".toml" +} + +// FunctionConfigFor resolves one function's configuration. +// +// Central [functions.] and a sidecar are both supported, and having both +// is an error rather than a precedence rule. Picking a winner would mean the +// loser's edits silently do nothing, which is the failure this design refuses +// everywhere else — an unknown key, a stale [functions.x] block, a cron +// variable that names nothing. +func (c *Config) FunctionConfigFor(fn Function) (FunctionConfig, error) { + central, hasCentral := c.Functions[fn.Name] + + path := c.sidecarPath(fn) + sidecar, hasSidecar, err := loadSidecar[FunctionConfig](path) + if err != nil { + return FunctionConfig{}, err + } + + if hasCentral && hasSidecar { + return FunctionConfig{}, fmt.Errorf( + "%s is configured twice: [functions.%s] in %s and %s.\n"+ + " keep one — whichever loses would silently stop applying", + fn.Name, fn.Name, ConfigName, c.relative(path)) + } + if hasSidecar { + return sidecar, nil + } + return central, nil +} + +// ConnectorConfigFor resolves one connector's configuration. +func (c *Config) ConnectorConfigFor(conn Connector) (ConnectorConfig, error) { + central, hasCentral := c.Connectors[conn.Name] + + path := filepath.Join(c.FunctionsPath(), filepath.FromSlash(conn.Dir), ConnectorSidecar) + sidecar, hasSidecar, err := loadSidecar[ConnectorConfig](path) + if err != nil { + return ConnectorConfig{}, err + } + + if hasCentral && hasSidecar { + return ConnectorConfig{}, fmt.Errorf( + "%s is configured twice: [connectors.%s] in %s and %s.\n"+ + " keep one — whichever loses would silently stop applying", + conn.Name, conn.Name, ConfigName, c.relative(path)) + } + if hasSidecar { + return sidecar, nil + } + return central, nil +} + +// sidecarPath is where a function's sidecar would live. +func (c *Config) sidecarPath(fn Function) string { + root := c.FunctionsPath() + if fn.Dir { + return filepath.Join(root, filepath.FromSlash(filepath.Dir(fn.Entrypoint)), FunctionSidecar) + } + return filepath.Join(root, filepath.FromSlash(SingleFileSidecar(fn.Entrypoint))) +} + +// relative renders a path for an error message, relative to the project root. +func (c *Config) relative(path string) string { + if rel, err := filepath.Rel(c.Root, path); err == nil { + return filepath.ToSlash(rel) + } + return path +} + +// loadSidecar decodes a sidecar, reporting whether one exists. +// +// An unknown key is an error here for the same reason it is in the project +// file: TOML's failure mode for a misspelling is silence, and a setting that +// quietly does nothing is worse than one that refuses. +func loadSidecar[T any](path string) (T, bool, error) { + var out T + raw, err := os.ReadFile(path) + if os.IsNotExist(err) { + return out, false, nil + } + if err != nil { + return out, false, fmt.Errorf("read %s: %w", filepath.Base(path), err) + } + + md, err := toml.Decode(string(raw), &out) + if err != nil { + return out, false, fmt.Errorf("%s: %w", filepath.Base(path), err) + } + if undecoded := md.Undecoded(); len(undecoded) > 0 { + keys := make([]string, 0, len(undecoded)) + for _, k := range undecoded { + keys = append(keys, k.String()) + } + return out, false, fmt.Errorf("%s: unknown key(s): %s", filepath.Base(path), strings.Join(keys, ", ")) + } + return out, true, nil +} diff --git a/internal/project/sidecar_test.go b/internal/project/sidecar_test.go new file mode 100644 index 0000000..6ed26fd --- /dev/null +++ b/internal/project/sidecar_test.go @@ -0,0 +1,157 @@ +package project + +import ( + "strings" + "testing" +) + +func loadCfg(t *testing.T, files map[string]string) *Config { + t.Helper() + if _, ok := files["notte.toml"]; !ok { + files["notte.toml"] = "[project]\nname = \"demo\"\n" + } + cfg, err := Load(write(t, files)) + if err != nil { + t.Fatal(err) + } + return cfg +} + +// A sidecar beside the code is what makes a tree of thousands workable: +// marketplace carries 2,524 of them, and a central block per function would be +// one file everyone edits at once. +func TestSidecarConfiguresADirectoryFunction(t *testing.T) { + cfg := loadCfg(t, map[string]string{ + "functions/amazon/main.py": "def run():\n return 1\n", + "functions/amazon/function.toml": `name = "Amazon search" +description = "Searches" +domain = "amazon.com" +cron = "cron(0 9 * * ? *)" +`, + }) + fns, _, err := DiscoverAll(cfg) + if err != nil { + t.Fatal(err) + } + fc, err := cfg.FunctionConfigFor(fns[0]) + if err != nil { + t.Fatal(err) + } + if fc.Name != "Amazon search" || fc.Domain != "amazon.com" || fc.Cron == "" { + t.Fatalf("got %+v", fc) + } +} + +// A single-file function has no directory, so its sidecar sits beside it — +// the convention marketplace already uses. +func TestSidecarConfiguresASingleFileFunction(t *testing.T) { + cfg := loadCfg(t, map[string]string{ + "functions/quick.py": "def run():\n return 1\n", + "functions/quick.toml": "description = \"A quick one\"\n", + }) + fns, _, err := DiscoverAll(cfg) + if err != nil { + t.Fatal(err) + } + fc, err := cfg.FunctionConfigFor(fns[0]) + if err != nil { + t.Fatal(err) + } + if fc.Description != "A quick one" { + t.Fatalf("got %+v", fc) + } +} + +// Central still works, so a small project needs no sidecars at all. +func TestCentralConfigStillApplies(t *testing.T) { + cfg := loadCfg(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[functions.amazon]\ndescription = \"central\"\n", + "functions/amazon/main.py": "def run():\n return 1\n", + }) + fns, _, err := DiscoverAll(cfg) + if err != nil { + t.Fatal(err) + } + fc, err := cfg.FunctionConfigFor(fns[0]) + if err != nil { + t.Fatal(err) + } + if fc.Description != "central" { + t.Fatalf("got %+v", fc) + } +} + +// Configuring a unit in both places is an error rather than a precedence rule. +// Picking a winner means the loser's edits silently do nothing, which is the +// failure this design refuses everywhere else. +func TestConfiguringAUnitTwiceIsAnError(t *testing.T) { + cfg := loadCfg(t, map[string]string{ + "notte.toml": "[project]\nname=\"d\"\n\n[functions.amazon]\ndescription = \"central\"\n", + "functions/amazon/main.py": "def run():\n return 1\n", + "functions/amazon/function.toml": "description = \"sidecar\"\n", + }) + fns, _, err := DiscoverAll(cfg) + if err != nil { + t.Fatal(err) + } + _, err = cfg.FunctionConfigFor(fns[0]) + if err == nil { + t.Fatal("expected an error") + } + for _, want := range []string{"amazon", "function.toml", ConfigName} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error should name %q: %v", want, err) + } + } +} + +// TOML's failure mode for a typo is silence, so a sidecar gets the same +// unknown-key treatment the project file does. +func TestSidecarUnknownKeyIsRejected(t *testing.T) { + cfg := loadCfg(t, map[string]string{ + "functions/amazon/main.py": "def run():\n return 1\n", + "functions/amazon/function.toml": "descriptoin = \"typo\"\n", + }) + fns, _, err := DiscoverAll(cfg) + if err != nil { + t.Fatal(err) + } + if _, err := cfg.FunctionConfigFor(fns[0]); err == nil || !strings.Contains(err.Error(), "descriptoin") { + t.Fatalf("expected the key to be named, got %v", err) + } +} + +func TestConnectorSidecar(t *testing.T) { + cfg := loadCfg(t, map[string]string{ + "functions/bluesky/login.py": "def run(session_id: str):\n return 1\n", + "functions/bluesky/verifier.py": "def run(session_id: str):\n return 1\n", + "functions/bluesky/connector.toml": `name = "Bluesky" +domain = "bsky.app" +category = "Social" +color = "#0085ff" +`, + }) + _, conns, err := DiscoverAll(cfg) + if err != nil { + t.Fatal(err) + } + cc, err := cfg.ConnectorConfigFor(conns[0]) + if err != nil { + t.Fatal(err) + } + if cc.Name != "Bluesky" || cc.Domain != "bsky.app" { + t.Fatalf("got %+v", cc) + } + if p := cc.Validate("bluesky"); len(p) != 0 { + t.Fatalf("valid connector reported %v", p) + } +} + +// A sidecar must not be named notte.toml: Find walks up looking for that name, +// so any command run from inside a function would treat the function's +// directory as the project root. +func TestSidecarNameCannotShadowTheProjectFile(t *testing.T) { + if FunctionSidecar == ConfigName || ConnectorSidecar == ConfigName { + t.Fatal("a sidecar sharing the project filename would break upward discovery") + } +} diff --git a/internal/pyenv/health.go b/internal/pyenv/health.go new file mode 100644 index 0000000..df2ef47 --- /dev/null +++ b/internal/pyenv/health.go @@ -0,0 +1,212 @@ +// Package pyenv turns the runtime's self-description into a local Python +// environment that matches it, and runs the checks that need an interpreter. +// +// Everything here follows from one fact: the CLI must not carry its own copy of +// the runtime's rules. Earlier drafts vendored the import allowlist, a denylist +// and a stdlib set generated from a pinned CPython, and all three drifted — +// one of them shipping a list from CPython 3.14 that would have rejected +// modules the 3.12 runner actually has. GET /functions/health makes the runner +// authoritative instead. +package pyenv + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "sort" + "strings" +) + +// Status values reported by GET /functions/health. +const ( + // StatusOK means the runner answered and the report is complete. + StatusOK = "ok" + // StatusDegraded means the API is healthy but the runner did not answer + // the contract question — normally because an API deploy has landed and + // the runner image has not been rebuilt yet. It is a routine state, not + // an outage, and must never block a deploy. + StatusDegraded = "degraded" + // StatusUnreachable means the runner could not be reached at all. + StatusUnreachable = "unreachable" +) + +// Health is the runtime's description of itself. +type Health struct { + Status string `json:"status"` + Reachable bool `json:"reachable"` + LatencyMS float64 `json:"latency_ms"` + Error string `json:"error,omitempty"` + + // PythonVersion is the interpreter the runner executes, e.g. "3.12.0". + // Empty when degraded. + PythonVersion string `json:"python_version"` + // Packages are the non-stdlib imports the runner allows. Empty when + // degraded. + Packages []Package `json:"packages"` + // StdlibModules are the standard-library imports the runner allows, + // post-patch — so tempfile, which upload accepts and the runner discards, + // is absent. Empty when degraded. + StdlibModules []string `json:"stdlib_modules"` + // ReservedEnvNames are function_env names the API refuses to store. This + // is the API's own rule rather than the runner's, so it is populated even + // when degraded, and `secrets` validation keeps working through the window. + ReservedEnvNames []string `json:"reserved_env_names"` + // RuntimeDigest covers the contract fields only, so it moves if and only + // if an environment built against the previous answer would now be wrong. + // Null when the report is partial: a validator for "no answer" is worse + // than none. + RuntimeDigest string `json:"runtime_digest"` +} + +// Package is one importable non-stdlib name. +type Package struct { + // ImportName is what appears in an import statement, e.g. "bs4". + ImportName string `json:"import_name"` + // Package is the distribution name, e.g. "beautifulsoup4". + Package string `json:"package"` + Version string `json:"version"` + // Installed reports whether the image actually ships it. A name can be + // allowed and absent, which passes upload validation and then dies on + // ModuleNotFoundError mid-run. + Installed bool `json:"installed"` + // Source is the PEP 610 direct_url of the install when it did not come + // from an index — the runner installs notte-sdk and notte-core from git + // SHAs, so for those a version is not an identity. Empty for index + // installs, where the version is the whole identity. + Source string `json:"source"` +} + +// Complete reports whether the report carries the contract fields. A degraded +// report has reserved names and nothing else usable. +func (h *Health) Complete() bool { + return h.Status == StatusOK && h.PythonVersion != "" && h.RuntimeDigest != "" +} + +// Allows reports whether an import is permitted by the runtime. +// +// Submodules follow their root, which is how safe_import resolves them: a +// denied root denies its children, an allowed root allows them. +func (h *Health) Allows(module string) bool { + root := module + if i := strings.IndexByte(module, '.'); i >= 0 { + root = module[:i] + } + for _, m := range h.StdlibModules { + if m == root || m == module { + return true + } + } + for _, p := range h.Packages { + if p.ImportName == root { + return true + } + } + return false +} + +// Package returns the entry for an import name. +func (h *Health) Package(importName string) (Package, bool) { + for _, p := range h.Packages { + if p.ImportName == importName { + return p, true + } + } + return Package{}, false +} + +// Installable is the subset of wanted imports that the runner both allows and +// actually ships, plus the names it allows but does not ship. +// +// The second return is the case worth surfacing: allowed-but-absent passes +// upload validation and fails at run time, and nothing else makes it visible. +func (h *Health) Installable(wanted []string) (install []Package, allowedButMissing []string) { + seen := map[string]bool{} + for _, name := range wanted { + root := name + if i := strings.IndexByte(name, '.'); i >= 0 { + root = name[:i] + } + if seen[root] { + continue + } + seen[root] = true + + p, ok := h.Package(root) + if !ok { + continue // stdlib, or not allowed at all — the caller reports that + } + if p.Installed { + install = append(install, p) + } else { + allowedButMissing = append(allowedButMissing, root) + } + } + sort.Slice(install, func(i, j int) bool { return install[i].ImportName < install[j].ImportName }) + sort.Strings(allowedButMissing) + return install, allowedButMissing +} + +// Requirement renders a package as a uv/pip install argument. +// +// Source wins over version when present. The runner installs notte-sdk and +// notte-core from git, and the published package under the same version number +// is different code — a version-only install produces a near-miss environment +// that reports confident, wrong answers. +func (p Package) Requirement() string { + name := p.Package + if name == "" { + name = p.ImportName + } + if p.Source != "" { + // PEP 508 direct reference. The git+ prefix is required — uv rejects a + // bare https URL outright — and naming the distribution keeps the + // resolver's messages legible. + return name + " @ " + p.Source + } + if p.Version == "" { + return name + } + return name + "==" + p.Version +} + +// FetchHealth reads GET /functions/health. +// +// The endpoint always answers 200 and carries the answer in `status`, so a +// caller that branches on the HTTP code learns nothing. Only transport and +// decoding failures surface as errors here. +func FetchHealth(ctx context.Context, client *http.Client, baseURL, apiKey string) (*Health, error) { + url := strings.TrimSuffix(baseURL, "/") + "/functions/health" + req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + if err != nil { + return nil, err + } + req.Header.Set("Authorization", "Bearer "+apiKey) + + resp, err := client.Do(req) + if err != nil { + return nil, fmt.Errorf("GET /functions/health: %w", err) + } + defer func() { _ = resp.Body.Close() }() + + body, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20)) + if err != nil { + return nil, fmt.Errorf("GET /functions/health: %w", err) + } + if resp.StatusCode == http.StatusNotFound { + // The route is matched by GET /functions/{function_id} on an API old + // enough to lack it, so the error talks about a function called + // "health". Say what it means instead. + return nil, fmt.Errorf("this Notte API does not have GET /functions/health yet; upgrade it or use an environment that does") + } + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("GET /functions/health: HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(body))) + } + + var h Health + if err := json.Unmarshal(body, &h); err != nil { + return nil, fmt.Errorf("GET /functions/health: %w", err) + } + return &h, nil +} diff --git a/internal/pyenv/health_test.go b/internal/pyenv/health_test.go new file mode 100644 index 0000000..5caebef --- /dev/null +++ b/internal/pyenv/health_test.go @@ -0,0 +1,232 @@ +package pyenv + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" +) + +// loadFixture reads a payload captured from a live environment. These are real +// responses, not hand-written approximations — the point is to be tested +// against what the API actually sends. +func loadFixture(t *testing.T, name string) *Health { + t.Helper() + raw, err := os.ReadFile(filepath.Join("testdata", name)) + if err != nil { + t.Fatal(err) + } + var h Health + if err := json.Unmarshal(raw, &h); err != nil { + t.Fatal(err) + } + return &h +} + +func TestDecodeRealOKResponse(t *testing.T) { + h := loadFixture(t, "health_ok.json") + if h.Status != StatusOK || !h.Reachable { + t.Fatalf("status=%q reachable=%v", h.Status, h.Reachable) + } + if h.PythonVersion != "3.12.0" { + t.Fatalf("python_version = %q", h.PythonVersion) + } + if !strings.HasPrefix(h.RuntimeDigest, "sha256:") { + t.Fatalf("runtime_digest = %q", h.RuntimeDigest) + } + if !h.Complete() { + t.Fatal("an ok report with a version and a digest should be complete") + } + if len(h.Packages) == 0 || len(h.StdlibModules) == 0 { + t.Fatalf("packages=%d stdlib=%d", len(h.Packages), len(h.StdlibModules)) + } +} + +// A degraded report carries reserved names and nothing else usable. Treating it +// as authoritative would build an empty venv and reject every import. +func TestDecodeRealDegradedResponse(t *testing.T) { + h := loadFixture(t, "health_degraded.json") + if h.Status != StatusDegraded { + t.Fatalf("status = %q", h.Status) + } + if !h.Reachable { + t.Fatal("degraded still means the API answered") + } + if h.Complete() { + t.Fatal("a degraded report must not be treated as complete") + } + if h.RuntimeDigest != "" { + t.Fatalf("digest must be null when partial, got %q", h.RuntimeDigest) + } + if len(h.ReservedEnvNames) == 0 { + t.Fatal("reserved_env_names must survive degradation — it is the API's rule, not the runner's") + } +} + +// tempfile is allowed at upload and discarded by the runner. If it leaks back +// into stdlib_modules, the CLI accepts something that dies at run time. +func TestRuntimeStdlibExcludesTempfileAndProcessControl(t *testing.T) { + h := loadFixture(t, "health_ok.json") + for _, denied := range []string{"tempfile", "os", "sys", "subprocess", "pathlib", "socket"} { + if h.Allows(denied) { + t.Errorf("%q must not be allowed by the runtime", denied) + } + } + for _, allowed := range []string{"json", "re", "datetime", "asyncio"} { + if !h.Allows(allowed) { + t.Errorf("%q should be allowed", allowed) + } + } +} + +func TestAllowsFollowsTheRootOfADottedImport(t *testing.T) { + h := loadFixture(t, "health_ok.json") + if !h.Allows("notte_sdk.types") { + t.Error("a submodule of an allowed package should be allowed") + } + if h.Allows("os.path") { + t.Error("a submodule of a denied root must stay denied") + } +} + +// The runner installs notte-sdk from a git SHA, and the published package under +// the same version number is different code. A version-only install builds a +// near-miss environment that reports confident, wrong answers. +func TestRequirementPrefersSourceOverVersion(t *testing.T) { + h := loadFixture(t, "health_ok.json") + sdk, ok := h.Package("notte_sdk") + if !ok { + t.Fatal("notte_sdk missing from the report") + } + if sdk.Source == "" { + t.Skip("this capture has no git source for notte_sdk") + } + req := sdk.Requirement() + if !strings.Contains(req, "github.com/nottelabs/notte") { + t.Fatalf("requirement should install from the git source, got %q", req) + } + // uv rejects a bare https URL: the git+ prefix is what marks it a VCS + // reference. Verified against uv directly. + if !strings.Contains(req, "git+https://") { + t.Fatalf("the git+ prefix must survive, got %q", req) + } + if !strings.HasPrefix(req, "notte-sdk @ ") { + t.Fatalf("PEP 508 direct reference should name the distribution, got %q", req) + } +} + +func TestRequirementUsesPinnedVersionForIndexInstalls(t *testing.T) { + p := Package{ImportName: "bs4", Package: "beautifulsoup4", Version: "4.12.3", Installed: true} + if got := p.Requirement(); got != "beautifulsoup4==4.12.3" { + t.Fatalf("got %q", got) + } + // The import name is not always the distribution name. + if p.ImportName == p.Package { + t.Fatal("fixture should exercise the import-name/package-name split") + } +} + +// Allowed-but-absent is the case that passes upload validation and dies at run +// time. The real staging report has three of them. +func TestInstallableSeparatesAllowedFromShipped(t *testing.T) { + h := loadFixture(t, "health_ok.json") + install, missing := h.Installable([]string{"requests", "notte", "notte_sdk", "json", "pandas"}) + + var installed []string + for _, p := range install { + installed = append(installed, p.ImportName) + } + if len(installed) == 0 { + t.Fatal("expected requests and notte_sdk to be installable") + } + for _, want := range []string{"notte_sdk", "requests"} { + found := false + for _, got := range installed { + if got == want { + found = true + } + } + if !found { + t.Errorf("%q should be installable, got %v", want, installed) + } + } + // notte is allowed by the runtime but not shipped in the image. + found := false + for _, m := range missing { + if m == "notte" { + found = true + } + } + if !found { + t.Errorf("notte is allowed-but-absent in the real report; got missing=%v", missing) + } + // stdlib and unknown names are neither installable nor "missing". + for _, m := range missing { + if m == "json" || m == "pandas" { + t.Errorf("%q should not be reported as allowed-but-missing", m) + } + } +} + +func TestFetchHealthReadsStatusNotHTTPCode(t *testing.T) { + body, err := os.ReadFile(filepath.Join("testdata", "health_degraded.json")) + if err != nil { + t.Fatal(err) + } + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if got := r.Header.Get("Authorization"); got != "Bearer k" { + t.Errorf("Authorization = %q", got) + } + if r.URL.Path != "/functions/health" { + t.Errorf("path = %q", r.URL.Path) + } + w.WriteHeader(http.StatusOK) // always 200, even when degraded + _, _ = w.Write(body) + })) + defer srv.Close() + + h, err := FetchHealth(context.Background(), srv.Client(), srv.URL, "k") + if err != nil { + t.Fatalf("degraded must not surface as a transport error: %v", err) + } + if h.Status != StatusDegraded { + t.Fatalf("status = %q", h.Status) + } +} + +// An API without the route matches it against GET /functions/{function_id}, +// producing an error about a function called "health". +func TestFetchHealthExplainsAnOldAPI(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusNotFound) + _, _ = w.Write([]byte(`{"detail":"The function 'health' does not exist or the user does not have access to it"}`)) + })) + defer srv.Close() + + _, err := FetchHealth(context.Background(), srv.Client(), srv.URL, "k") + if err == nil { + t.Fatal("expected an error") + } + if !strings.Contains(err.Error(), "does not have GET /functions/health yet") { + t.Fatalf("error should explain the real cause, got %v", err) + } +} + +func TestFetchHealthTrimsTrailingSlash(t *testing.T) { + var path string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + path = r.URL.Path + _, _ = w.Write([]byte(`{"status":"ok"}`)) + })) + defer srv.Close() + if _, err := FetchHealth(context.Background(), srv.Client(), srv.URL+"/", "k"); err != nil { + t.Fatal(err) + } + if path != "/functions/health" { + t.Fatalf("path = %q", path) + } +} diff --git a/internal/pyenv/testdata/health_degraded.json b/internal/pyenv/testdata/health_degraded.json new file mode 100644 index 0000000..d71e456 --- /dev/null +++ b/internal/pyenv/testdata/health_degraded.json @@ -0,0 +1,18 @@ +{ + "status": "degraded", + "reachable": true, + "latency_ms": 318.2, + "python_version": null, + "packages": [], + "stdlib_modules": [], + "reserved_env_names": [ + "ENVIRONMENT", + "NOTTE_API_KEY", + "NOTTE_API_URL", + "NOTTE_BASE_URL", + "NOTTE_DB_PREVIEW_BRANCH", + "NOTTE_ENV" + ], + "runtime_digest": null, + "error": "HTTPStatusError: Client error '404 Not Found' for url 'https://.lambda-url..on.aws/runtime'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404" +} diff --git a/internal/pyenv/testdata/health_ok.json b/internal/pyenv/testdata/health_ok.json new file mode 100644 index 0000000..4cb22ad --- /dev/null +++ b/internal/pyenv/testdata/health_ok.json @@ -0,0 +1,362 @@ +{ + "status": "ok", + "reachable": true, + "latency_ms": 4888.9, + "python_version": "3.12.0", + "packages": [ + { + "import_name": "google", + "package": "google-auth", + "version": "2.57.0", + "source": null, + "installed": true + }, + { + "import_name": "gspread", + "package": "gspread", + "version": "6.2.1", + "source": null, + "installed": true + }, + { + "import_name": "httpcloak", + "package": "httpcloak", + "version": "1.6.8", + "source": null, + "installed": true + }, + { + "import_name": "httpx", + "package": "httpx", + "version": "0.28.1", + "source": null, + "installed": true + }, + { + "import_name": "litellm", + "package": "litellm", + "version": "1.80.0", + "source": null, + "installed": true + }, + { + "import_name": "loguru", + "package": "loguru", + "version": "0.7.3", + "source": null, + "installed": true + }, + { + "import_name": "notte", + "package": null, + "version": null, + "source": null, + "installed": false + }, + { + "import_name": "notte_agent", + "package": null, + "version": null, + "source": null, + "installed": false + }, + { + "import_name": "notte_browser", + "package": null, + "version": null, + "source": null, + "installed": false + }, + { + "import_name": "notte_core", + "package": "notte-core", + "version": "1.4.4.dev0", + "source": "git+https://github.com/nottelabs/notte.git@95d00c16449470775503a57cc9c065003c357b3c#subdirectory=packages/notte-core", + "installed": true + }, + { + "import_name": "notte_sdk", + "package": "notte-sdk", + "version": "1.4.4.dev0", + "source": "git+https://github.com/nottelabs/notte.git@95d00c16449470775503a57cc9c065003c357b3c#subdirectory=packages/notte-sdk", + "installed": true + }, + { + "import_name": "playwright", + "package": "playwright", + "version": "1.50.0", + "source": null, + "installed": true + }, + { + "import_name": "pydantic", + "package": "pydantic", + "version": "2.13.4", + "source": null, + "installed": true + }, + { + "import_name": "requests", + "package": "requests", + "version": "2.34.2", + "source": null, + "installed": true + }, + { + "import_name": "typing_extensions", + "package": "typing_extensions", + "version": "4.16.0", + "source": null, + "installed": true + } + ], + "stdlib_modules": [ + "__future__", + "_abc", + "_aix_support", + "_ast", + "_asyncio", + "_bisect", + "_blake2", + "_bz2", + "_codecs", + "_codecs_cn", + "_codecs_hk", + "_codecs_iso2022", + "_codecs_jp", + "_codecs_kr", + "_codecs_tw", + "_collections", + "_collections_abc", + "_compat_pickle", + "_compression", + "_contextvars", + "_crypt", + "_csv", + "_curses", + "_curses_panel", + "_datetime", + "_dbm", + "_decimal", + "_frozen_importlib", + "_frozen_importlib_external", + "_functools", + "_gdbm", + "_hashlib", + "_heapq", + "_json", + "_locale", + "_lsprof", + "_lzma", + "_markupbase", + "_md5", + "_msi", + "_multibytecodec", + "_opcode", + "_operator", + "_osx_support", + "_overlapped", + "_py_abc", + "_pydatetime", + "_pydecimal", + "_pyio", + "_pylong", + "_queue", + "_random", + "_scproxy", + "_sha1", + "_sha2", + "_sha3", + "_sitebuiltins", + "_sre", + "_ssl", + "_stat", + "_statistics", + "_string", + "_strptime", + "_struct", + "_symtable", + "_threading_local", + "_tkinter", + "_tokenize", + "_tracemalloc", + "_typing", + "_uuid", + "_warnings", + "_weakref", + "_weakrefset", + "_winapi", + "_zoneinfo", + "abc", + "aifc", + "antigravity", + "argparse", + "array", + "ast", + "asyncio", + "atexit", + "audioop", + "base64", + "bdb", + "binascii", + "bisect", + "bz2", + "cProfile", + "calendar", + "cgi", + "cgitb", + "chunk", + "cmath", + "cmd", + "codecs", + "collections", + "colorsys", + "concurrent", + "contextlib", + "contextvars", + "copy", + "copyreg", + "crypt", + "csv", + "curses", + "dataclasses", + "datetime", + "decimal", + "difflib", + "dis", + "doctest", + "email", + "encodings", + "ensurepip", + "enum", + "errno", + "faulthandler", + "fnmatch", + "fractions", + "ftplib", + "functools", + "genericpath", + "getopt", + "getpass", + "gettext", + "graphlib", + "gzip", + "hashlib", + "heapq", + "hmac", + "html", + "http", + "idlelib", + "imaplib", + "imghdr", + "io", + "ipaddress", + "itertools", + "json", + "keyword", + "lib2to3", + "locale", + "logging", + "lzma", + "mailbox", + "mailcap", + "math", + "mimetypes", + "msilib", + "msvcrt", + "netrc", + "nis", + "nntplib", + "ntpath", + "nturl2path", + "numbers", + "opcode", + "operator", + "optparse", + "ossaudiodev", + "pdb", + "pipes", + "platform", + "plistlib", + "poplib", + "posixpath", + "pprint", + "profile", + "pstats", + "pyclbr", + "pydoc", + "pydoc_data", + "pyexpat", + "queue", + "quopri", + "random", + "re", + "readline", + "reprlib", + "rlcompleter", + "sched", + "secrets", + "select", + "selectors", + "shlex", + "site", + "smtplib", + "sndhdr", + "spwd", + "sre_compile", + "sre_constants", + "sre_parse", + "ssl", + "stat", + "statistics", + "string", + "stringprep", + "struct", + "sunau", + "symtable", + "syslog", + "tabnanny", + "telnetlib", + "textwrap", + "this", + "time", + "timeit", + "tkinter", + "token", + "tokenize", + "tomllib", + "trace", + "traceback", + "tracemalloc", + "turtle", + "turtledemo", + "types", + "typing", + "unicodedata", + "unittest", + "urllib", + "urllib.parse", + "uu", + "uuid", + "warnings", + "wave", + "weakref", + "webbrowser", + "winsound", + "wsgiref", + "xdrlib", + "xmlrpc", + "zlib", + "zoneinfo" + ], + "reserved_env_names": [ + "ENVIRONMENT", + "NOTTE_API_KEY", + "NOTTE_API_URL", + "NOTTE_BASE_URL", + "NOTTE_DB_PREVIEW_BRANCH", + "NOTTE_ENV" + ], + "runtime_digest": "sha256:c9b804136706978a4c554404f0a9b55fa6cfae4cfadadd4b8bbe951a58b43244", + "error": null +} diff --git a/internal/pyenv/typecheck.go b/internal/pyenv/typecheck.go new file mode 100644 index 0000000..d862a4f --- /dev/null +++ b/internal/pyenv/typecheck.go @@ -0,0 +1,209 @@ +package pyenv + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "os" + "os/exec" + "sort" + "strings" +) + +// TyVersion pins the type checker. +// +// ty is on 0.0.x with no stable API and breaking changes between releases, so +// a floating version would let an upstream change turn a passing stack red +// without anything local moving. Pinned rather than avoided: it is what +// anything-api already gates its build agent on, and its 10-100x advantage on +// cold checks is exactly the case a CLI hits on every invocation. +const TyVersion = "0.0.75" + +// UnresolvedImport is the rule that must never be treated as noise. +// +// anything-api's ty-config.ts records why: with no interpreter configured, ty +// resolved against the first python on PATH, every import came back +// unresolved, and the build agent deployed straight through a mandatory type +// check. A checker that cannot resolve imports does not fail — it goes green. +const UnresolvedImport = "unresolved-import" + +// EnvironmentError is a failure of the environment rather than of the code +// being checked. +// +// It carries the rebuild command because the obvious one does not work: a +// plain sync reuses the environment, since the stamp records what it was built +// from and that still matches. Only --force gets past the reuse check. +type EnvironmentError struct { + VenvDir string + Detail string +} + +func (e *EnvironmentError) Error() string { + return fmt.Sprintf("the environment in %s is unusable, so nothing could be checked:\n %s\n"+ + " rebuild it with:\n notte stack sync --force", e.VenvDir, e.Detail) +} + +// Diagnostic is one finding. +type Diagnostic struct { + Rule string + Message string + Path string + Line int + Column int +} + +func (d Diagnostic) String() string { + return fmt.Sprintf("%s:%d:%d: %s [%s]", d.Path, d.Line, d.Column, d.Message, d.Rule) +} + +// TypeCheckResult is the outcome of a ty run. +type TypeCheckResult struct { + Diagnostics []Diagnostic + // Unresolved are the module names ty could not resolve. Split out because + // their meaning depends on whether the runtime claims to ship them. + Unresolved []string +} + +// OK reports whether the artifact is clean. +func (r *TypeCheckResult) OK() bool { return len(r.Diagnostics) == 0 } + +// Misconfigured reports whether ty failed to resolve something the runtime +// says it ships, which means the environment wiring is broken rather than the +// user's code. Reporting that as a code error would send someone to fix a file +// that is fine. +func (r *TypeCheckResult) Misconfigured(h *Health) []string { + var broken []string + for _, module := range r.Unresolved { + if p, ok := h.Package(module); ok && p.Installed { + broken = append(broken, module) + } + } + sort.Strings(broken) + return broken +} + +// gitlabDiagnostic is ty's GitLab Code Quality output. ty has no plain JSON +// format; this is the structured one, and parsing it beats scraping the +// human-readable lines. +type gitlabDiagnostic struct { + CheckName string `json:"check_name"` + Description string `json:"description"` + Severity string `json:"severity"` + Location struct { + Path string `json:"path"` + Positions struct { + Begin struct { + Line int `json:"line"` + Column int `json:"column"` + } `json:"begin"` + } `json:"positions"` + } `json:"location"` +} + +// TypeCheck runs ty over targets, relative to dir, resolving imports against +// the environment in venvDir. +// +// ty is run through uvx rather than installed into the venv on purpose: the +// venv mirrors the runtime image, and putting a package in it that the runtime +// does not have weakens the property that makes the venv the enforcement. +// +// The interpreter is passed explicitly on every invocation. Left to itself ty +// resolves against the first python on PATH, and anything-api records what that +// costs: every import came back unresolved and a mandatory type check went +// green while checking nothing. Passing --python rather than writing a ty.toml +// keeps it at the call site, where it cannot be omitted by a caller that +// forgot to generate the file — and leaves no machine-specific absolute path +// in the user's repository. +func TypeCheck(ctx context.Context, tc *Toolchain, dir, venvDir string, targets []string) (*TypeCheckResult, error) { + if len(targets) == 0 { + return &TypeCheckResult{}, nil + } + // ty treats an unusable --python as fatal for the whole run, which is worse + // than the misconfiguration it prevents, so it is checked first. + if _, err := os.Stat(PythonPath(venvDir)); err != nil { + return nil, fmt.Errorf("no interpreter at %s: %w", PythonPath(venvDir), err) + } + + args := append([]string{ + "ty@" + TyVersion, "check", + "--python", venvDir, + "--output-format", "gitlab", + // Diagnostics are read from stdout, never from the exit code — the + // same reason the health endpoint always answers 200. + "--exit-zero", + }, targets...) + + cmd := exec.CommandContext(ctx, tc.UV+"x", args...) + cmd.Dir = dir + var stderr bytes.Buffer + cmd.Stderr = &stderr + out, err := cmd.Output() + if err != nil { + // --exit-zero means diagnostics never fail the process, so a non-zero + // exit is ty itself refusing to run — overwhelmingly a broken + // environment rather than anything about the code. Returning a bare + // "exit status 2" sends someone hunting through their functions for a + // problem that is not there. + detail := strings.TrimSpace(stderr.String()) + if detail == "" { + detail = err.Error() + } + return nil, &EnvironmentError{VenvDir: venvDir, Detail: detail} + } + + var raw []gitlabDiagnostic + if err := json.Unmarshal(out, &raw); err != nil { + return nil, fmt.Errorf("decode ty output: %w (got %q)", err, truncate(string(out), 200)) + } + + res := &TypeCheckResult{} + seenUnresolved := map[string]bool{} + for _, d := range raw { + message := strings.TrimPrefix(d.Description, d.CheckName+": ") + res.Diagnostics = append(res.Diagnostics, Diagnostic{ + Rule: d.CheckName, + Message: message, + Path: d.Location.Path, + Line: d.Location.Positions.Begin.Line, + Column: d.Location.Positions.Begin.Column, + }) + if d.CheckName == UnresolvedImport { + if module := moduleFromUnresolved(message); module != "" && !seenUnresolved[module] { + seenUnresolved[module] = true + res.Unresolved = append(res.Unresolved, module) + } + } + } + sort.Strings(res.Unresolved) + return res, nil +} + +// moduleFromUnresolved pulls the module name out of ty's message, which reads +// "Cannot resolve imported module `foo`". +func moduleFromUnresolved(message string) string { + start := strings.IndexByte(message, '`') + if start < 0 { + return "" + } + rest := message[start+1:] + end := strings.IndexByte(rest, '`') + if end < 0 { + return "" + } + root := rest[:end] + if i := strings.IndexByte(root, '.'); i >= 0 { + root = root[:i] + } + return root +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] + "…" +} + +// UvxPath is uv's tool runner, which sits beside uv. +func (t *Toolchain) UvxPath() string { return t.UV + "x" } diff --git a/internal/pyenv/typecheck_test.go b/internal/pyenv/typecheck_test.go new file mode 100644 index 0000000..52bab6d --- /dev/null +++ b/internal/pyenv/typecheck_test.go @@ -0,0 +1,147 @@ +package pyenv + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestModuleFromUnresolvedMessage(t *testing.T) { + cases := map[string]string{ + "Cannot resolve imported module `httpcloak`": "httpcloak", + "Cannot resolve imported module `notte_sdk.types`": "notte_sdk", + "something else entirely": "", + } + for message, want := range cases { + if got := moduleFromUnresolved(message); got != want { + t.Errorf("%q -> %q, want %q", message, got, want) + } + } +} + +// An unresolved import of something the runtime says it ships means the venv +// or ty.toml wiring is broken, not the user's code. Reporting it as a code +// error sends someone to fix a file that is fine. +func TestMisconfiguredDistinguishesWiringFromUserError(t *testing.T) { + h := loadFixture(t, "health_ok.json") + res := &TypeCheckResult{Unresolved: []string{"requests", "pandas", "notte"}} + + broken := res.Misconfigured(h) + if len(broken) != 1 || broken[0] != "requests" { + t.Fatalf("misconfigured = %v, want [requests]", broken) + } + // pandas is not allowed at all — a genuine user error. + // notte is allowed but installed:false, so it is expected to be missing + // and must not be blamed on the environment. +} + +func TestTypeCheckWithNoTargetsIsClean(t *testing.T) { + res, err := TypeCheck(context.Background(), &Toolchain{UV: "uv"}, t.TempDir(), t.TempDir(), nil) + if err != nil { + t.Fatal(err) + } + if !res.OK() { + t.Fatal("no targets should be clean") + } +} + +// The real thing: ty against a venv built from the real staging report. +func TestTypeCheckAgainstARealEnvironment(t *testing.T) { + venv, h := realVenv(t) + tc := toolchain(t) + + dir := t.TempDir() + + // requests and pydantic are in the venv; nonexistent_pkg is not. + src := `import requests +from pydantic import BaseModel +import nonexistent_pkg + + +class Response(BaseModel): + ok: bool + + +def run() -> Response: + return Response(ok=bool(requests) and bool(nonexistent_pkg)) +` + if err := os.WriteFile(filepath.Join(dir, "artifact.py"), []byte(src), 0o644); err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) + defer cancel() + res, err := TypeCheck(ctx, tc, dir, venv, []string{"artifact.py"}) + if err != nil { + t.Fatalf("typecheck: %v", err) + } + + // The point of WriteTyConfig: packages that ARE installed must resolve. + // If these come back unresolved, ty is pointed at the wrong interpreter + // and every check it performs is meaningless. + for _, module := range res.Unresolved { + if module == "requests" || module == "pydantic" { + t.Fatalf("ty could not resolve %q from the venv — the interpreter is not configured, "+ + "which is the failure that lets a mandatory type check pass while checking nothing", module) + } + } + if len(res.Misconfigured(h)) != 0 { + t.Fatalf("environment reported broken: %v", res.Misconfigured(h)) + } + + // And a genuinely missing package must be reported. + found := false + for _, module := range res.Unresolved { + if module == "nonexistent_pkg" { + found = true + } + } + if !found { + t.Fatalf("an import of a package not in the venv should be unresolved; got %v", res.Unresolved) + } + if res.OK() { + t.Fatal("a file with an unresolvable import is not clean") + } +} + +// A type error unrelated to imports must be reported, which is what makes ty +// worth running at all: it catches the redefinitions a flattener could +// silently introduce. +func TestTypeCheckReportsRealTypeErrors(t *testing.T) { + venv, _ := realVenv(t) + tc := toolchain(t) + + dir := t.TempDir() + src := "def run() -> int:\n return \"not an int\"\n" + if err := os.WriteFile(filepath.Join(dir, "artifact.py"), []byte(src), 0o644); err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) + defer cancel() + res, err := TypeCheck(ctx, tc, dir, venv, []string{"artifact.py"}) + if err != nil { + t.Fatal(err) + } + if res.OK() { + t.Fatal("a wrong return type should be reported") + } + joined := strings.Join(diagnosticStrings(res), " ") + if !strings.Contains(joined, "invalid-return-type") { + t.Fatalf("expected invalid-return-type, got %v", diagnosticStrings(res)) + } + if res.Diagnostics[0].Line == 0 { + t.Fatal("diagnostics must carry a line number") + } +} + +func diagnosticStrings(res *TypeCheckResult) []string { + out := make([]string, len(res.Diagnostics)) + for i, d := range res.Diagnostics { + out[i] = d.String() + } + return out +} diff --git a/internal/pyenv/validate.go b/internal/pyenv/validate.go new file mode 100644 index 0000000..3345dbb --- /dev/null +++ b/internal/pyenv/validate.go @@ -0,0 +1,68 @@ +package pyenv + +import ( + "bytes" + "context" + _ "embed" + "encoding/json" + "fmt" + "os/exec" + "strings" +) + +//go:embed validate.py +var validateScript string + +// Verdict is the result of validating one artifact. +type Verdict struct { + OK bool `json:"ok"` + Stage string `json:"stage"` + Errors []string `json:"errors"` + // Variables are run()'s parameters, which become invocation variables. + Variables []Variable `json:"variables"` +} + +// Variable is one parameter of run(). +type Variable struct { + Name string `json:"name"` + Type *string `json:"type"` + Default *string `json:"default"` +} + +// Validate runs the SDK's ScriptValidator against an artifact, with the +// runtime's import list substituted for the SDK's own. +// +// The split is deliberate: the endpoint owns which imports are allowed, and +// the validator owns structure. Trusting the SDK's list would reject +// httpcloak, which ~333 deployed functions import. +func Validate(ctx context.Context, venvDir string, health *Health, source string) (*Verdict, error) { + if !health.Complete() { + return nil, fmt.Errorf("cannot validate against a %s runtime report: it carries no import list", health.Status) + } + + allowed := make([]string, 0, len(health.StdlibModules)+len(health.Packages)) + allowed = append(allowed, health.StdlibModules...) + for _, p := range health.Packages { + allowed = append(allowed, p.ImportName) + } + + request, err := json.Marshal(map[string]any{"source": source, "allowed_imports": allowed}) + if err != nil { + return nil, err + } + + cmd := exec.CommandContext(ctx, PythonPath(venvDir), "-c", validateScript) + cmd.Stdin = bytes.NewReader(request) + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + + if err := cmd.Run(); err != nil { + return nil, fmt.Errorf("run validator: %w: %s", err, strings.TrimSpace(stderr.String())) + } + + var v Verdict + if err := json.Unmarshal(stdout.Bytes(), &v); err != nil { + return nil, fmt.Errorf("decode validator output: %w (stdout: %q)", err, stdout.String()) + } + return &v, nil +} diff --git a/internal/pyenv/validate.py b/internal/pyenv/validate.py new file mode 100644 index 0000000..839c420 --- /dev/null +++ b/internal/pyenv/validate.py @@ -0,0 +1,67 @@ +"""Validate a bundled artifact against the runtime's own rules. + +Reads {"source": str, "allowed_imports": [str]} on stdin and writes a verdict +as JSON on stdout. Never raises for a rejected script: a non-zero exit would +be indistinguishable from the interpreter or the SDK being broken. + +Why the allow list is injected rather than trusted: + +The published notte-sdk carries its own ALLOWED_IMPORTS, and it is stale. At +1.8.31 it is an explicit 41-entry list with no httpcloak, httpx, bs4 or tqdm, +and it *includes* tempfile, which the runner discards. Running it unmodified +rejects functions that are deployed and serving traffic. So the validator's +machinery is reused for what it gets right - the entry point, forbidden nodes, +forbidden calls, relative imports - while the import question is answered from +GET /functions/health, which asks the runner. +""" + +import ast +import json +import sys + + +def main() -> None: + request = json.load(sys.stdin) + source = request["source"] + + try: + from notte_core.ast import ScriptValidator + except Exception as exc: # pragma: no cover - environment, not input + json.dump({"ok": False, "stage": "import", "errors": [f"cannot import notte_core.ast: {exc}"]}, sys.stdout) + return + + # The runtime's list replaces the SDK's, and the SDK's denylist is emptied + # so it cannot veto a name the runtime allows. Anything not in the runtime + # list is simply absent, and check_valid_import rejects it. + ScriptValidator.ALLOWED_IMPORTS = set(request["allowed_imports"]) + if hasattr(ScriptValidator, "DISALLOWED_STDLIB_IMPORTS"): + ScriptValidator.DISALLOWED_STDLIB_IMPORTS = set() + + errors = [] + + # parse_script accepts two top-level run() definitions where the server + # rejects them, so that is checked here rather than delegated. + try: + tree = ast.parse(source) + runs = [n for n in tree.body if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) and n.name == "run"] + if len(runs) > 1: + lines = ", ".join(str(n.lineno) for n in runs) + errors.append(f"multiple top-level run() definitions (lines {lines}); the runtime accepts exactly one") + except SyntaxError as exc: + json.dump({"ok": False, "stage": "syntax", "errors": [f"line {exc.lineno}: {exc.msg}"]}, sys.stdout) + return + + variables = [] + try: + info = ScriptValidator.parse_script(source, restricted=True) + variables = [ + {"name": v.name, "type": v.type, "default": v.default} + for v in info.variables + ] + except Exception as exc: + errors.append(f"{type(exc).__name__}: {exc}") + + json.dump({"ok": not errors, "stage": "validate", "errors": errors, "variables": variables}, sys.stdout) + + +main() diff --git a/internal/pyenv/validate_test.go b/internal/pyenv/validate_test.go new file mode 100644 index 0000000..de5e42f --- /dev/null +++ b/internal/pyenv/validate_test.go @@ -0,0 +1,141 @@ +package pyenv + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// realVenv builds an environment from the captured staging report once per +// test binary. Slow, networked and skipped under -short. +func realVenv(t *testing.T) (string, *Health) { + t.Helper() + if testing.Short() { + t.Skip("network") + } + tc := toolchain(t) + h := loadFixture(t, "health_ok.json") + + // Shared across tests in this package run; building it per test would add + // minutes for no extra coverage. + venv := filepath.Join(os.TempDir(), "notte-pyenv-test-venv") + ctx, cancel := context.WithTimeout(context.Background(), 6*time.Minute) + defer cancel() + if _, err := Sync(ctx, tc, SyncRequest{ + VenvDir: venv, Health: h, + Imports: []string{"requests", "pydantic", "notte_sdk", "httpcloak"}, + }); err != nil { + t.Fatalf("sync: %v", err) + } + return venv, h +} + +func validate(t *testing.T, venv string, h *Health, src string) *Verdict { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) + defer cancel() + v, err := Validate(ctx, venv, h, src) + if err != nil { + t.Fatalf("validate: %v", err) + } + return v +} + +// The whole reason the allow list is injected. The published SDK rejects +// httpcloak, and ~333 deployed functions import it. With the runtime's list +// substituted, a real production function must pass. +func TestValidateAcceptsARealProductionFunction(t *testing.T) { + venv, h := realVenv(t) + + src, err := os.ReadFile(filepath.Join(os.Getenv("HOME"), + "Desktop/projects/anything-api/marketplace/99.co/list_condos_by_letter.py")) + if err != nil { + t.Skipf("marketplace checkout not available: %v", err) + } + + v := validate(t, venv, h, string(src)) + if !v.OK { + t.Fatalf("a deployed, serving function was rejected: %v", v.Errors) + } + if len(v.Variables) == 0 { + t.Fatal("run() parameters should be extracted as invocation variables") + } +} + +// Same source, unpatched SDK list: this is the bug the injection avoids. +func TestUnpatchedSDKWouldRejectThatSameFunction(t *testing.T) { + venv, _ := realVenv(t) + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + + // A report whose only allowed imports are the SDK's own stale list. + stale := &Health{ + Status: StatusOK, PythonVersion: "3.12.0", RuntimeDigest: "sha256:x", + StdlibModules: []string{"typing"}, + Packages: []Package{{ImportName: "pydantic"}}, + } + v, err := Validate(ctx, venv, stale, "import httpcloak\ndef run():\n return 1\n") + if err != nil { + t.Fatal(err) + } + if v.OK { + t.Fatal("with httpcloak absent from the allow list the validator must reject it — " + + "if this passes, the injection is not taking effect") + } +} + +func TestValidateRejectsStructuralProblems(t *testing.T) { + venv, h := realVenv(t) + + cases := map[string]string{ + "no run()": "def other():\n return 1\n", + "relative import": "from .util import x\ndef run():\n return 1\n", + "denied stdlib": "import subprocess\ndef run():\n return 1\n", + "forbidden call": "def run():\n exec(\"x=1\")\n return 1\n", + "unknown package": "import pandas\ndef run():\n return 1\n", + } + for name, src := range cases { + t.Run(name, func(t *testing.T) { + if v := validate(t, venv, h, src); v.OK { + t.Fatalf("%s should have been rejected", name) + } + }) + } +} + +// The one gap in the SDK validator: it accepts two top-level run() definitions +// where the server rejects them, so the bridge checks it directly. +func TestValidateRejectsTwoRunDefinitions(t *testing.T) { + venv, h := realVenv(t) + v := validate(t, venv, h, "def run():\n return 1\n\n\ndef run():\n return 2\n") + if v.OK { + t.Fatal("two run() definitions must be rejected") + } + joined := strings.Join(v.Errors, " ") + if !strings.Contains(joined, "multiple top-level run()") { + t.Fatalf("error should name the problem, got %v", v.Errors) + } +} + +// A rejected script is a verdict, not a crash: a non-zero exit would be +// indistinguishable from the interpreter or the SDK being broken. +func TestValidateReportsSyntaxErrorsAsAVerdict(t *testing.T) { + venv, h := realVenv(t) + v := validate(t, venv, h, "def run(:\n") + if v.OK { + t.Fatal("a syntax error must not pass") + } + if v.Stage != "syntax" { + t.Fatalf("stage = %q, want syntax", v.Stage) + } +} + +func TestValidateRefusesADegradedReport(t *testing.T) { + _, err := Validate(context.Background(), t.TempDir(), loadFixture(t, "health_degraded.json"), "def run(): pass") + if err == nil { + t.Fatal("a degraded report carries no import list, so validation cannot be authoritative") + } +} diff --git a/internal/pyenv/venv.go b/internal/pyenv/venv.go new file mode 100644 index 0000000..5854831 --- /dev/null +++ b/internal/pyenv/venv.go @@ -0,0 +1,241 @@ +package pyenv + +import ( + "context" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" +) + +// StampName records which runtime a venv was built for, so a rebuild happens +// when the runtime moves and not merely when time passes. +const StampName = ".notte-runtime.json" + +// Toolchain is the external tooling the stack commands need. +type Toolchain struct { + // UV is the path to uv, which supplies the interpreter as well as the + // packages — so the requirement is "have uv", not "have Python 3.12". + UV string +} + +// ErrNoToolchain is returned when uv is absent. +type ErrNoToolchain struct{} + +func (ErrNoToolchain) Error() string { + return "uv is required by `notte stack` and was not found on PATH\n" + + " install it with: curl -LsSf https://astral.sh/uv/install.sh | sh\n" + + " (uv downloads the Python the runtime uses, so nothing else is needed)" +} + +// FindToolchain locates uv. +func FindToolchain() (*Toolchain, error) { + path, err := exec.LookPath("uv") + if err != nil { + return nil, ErrNoToolchain{} + } + return &Toolchain{UV: path}, nil +} + +// Stamp is written into the venv to record what it was built from. +type Stamp struct { + RuntimeDigest string `json:"runtime_digest"` + PythonVersion string `json:"python_version"` + Requirements []string `json:"requirements"` +} + +// AlwaysInstall are put into every environment regardless of what the +// functions currently import. +// +// notte_core carries ScriptValidator, so an environment without it can build a +// perfectly good artifact and then fail to check it. +// +// notte_sdk is what people actually write against — it is the single most +// common import in real stacks, and someone typing `from notte_sdk import +// NotteClient` into a new function wants completion before they have saved and +// re-synced. Measured at 1 MB on top of notte_core's 46 MB, in a third of a +// second, with no browser dependencies, so the mirroring argument wins easily. +// +// Neither loosens the mirror: both are in the runtime image regardless, so +// installing them unconditionally makes the venv a closer match, not a looser +// one. +var AlwaysInstall = []string{"notte_core", "notte_sdk"} + +// SyncRequest describes the environment to build. +type SyncRequest struct { + // VenvDir is where the environment lives, normally .notte/venv. + VenvDir string + // Health is the runtime's report. Must be Complete(). + Health *Health + // Imports are the non-relative module names the functions use. Only the + // intersection with what the runtime ships is installed: the allowlist is + // closed, so this is a set intersection rather than dependency resolution. + Imports []string + // Force rebuilds even when the stamp matches. + // + // The stamp records what an environment was built *from*, which is not the + // same as it still being intact — a half-finished install or a deleted + // site-packages leaves a venv that reuse happily accepts. Without this + // there is no way to recover except deleting the directory by hand. + Force bool +} + +// SyncResult describes what was built. +type SyncResult struct { + VenvDir string + Python string + // Installed are the packages put into the environment. + Installed []Package + // AllowedButMissing are imports the runtime permits but does not ship. + // They pass upload validation and fail at run time, so they are surfaced + // rather than silently skipped. + AllowedButMissing []string + // NotAllowed are imports the runtime rejects outright. + NotAllowed []string + // Reused reports that an existing venv already matched the runtime digest. + Reused bool +} + +// PythonPath is the interpreter inside a venv. +func PythonPath(venvDir string) string { + if runtime.GOOS == "windows" { + return filepath.Join(venvDir, "Scripts", "python.exe") + } + return filepath.Join(venvDir, "bin", "python") +} + +// Sync builds an environment matching the runtime. +// +// It refuses to work from a partial report. A degraded response carries no +// packages and no stdlib list, so building from it would produce an empty +// environment in which every import fails — a confident, wrong answer. +func Sync(ctx context.Context, tc *Toolchain, req SyncRequest) (*SyncResult, error) { + if !req.Health.Complete() { + return nil, fmt.Errorf("cannot build an environment from a %s runtime report: it carries no package list", req.Health.Status) + } + + install, missing := req.Health.Installable(append(append([]string{}, AlwaysInstall...), req.Imports...)) + res := &SyncResult{ + VenvDir: req.VenvDir, + Python: req.Health.PythonVersion, + Installed: install, + AllowedButMissing: missing, + NotAllowed: notAllowed(req.Health, req.Imports), + } + + requirements := make([]string, 0, len(install)) + for _, p := range install { + requirements = append(requirements, p.Requirement()) + } + want := Stamp{ + RuntimeDigest: req.Health.RuntimeDigest, + PythonVersion: req.Health.PythonVersion, + Requirements: requirements, + } + + // The digest covers the contract fields only, so it moves if and only if an + // environment built against the previous answer would now be wrong. A + // rebuild that changes nothing observable does not invalidate this venv. + if have, err := readStamp(req.VenvDir); !req.Force && err == nil && have.matches(want) { + res.Reused = true + return res, nil + } + + if err := os.MkdirAll(filepath.Dir(req.VenvDir), 0o755); err != nil { + return nil, err + } + // Reuse was already ruled out, so anything here is stale or a half-built + // environment from an interrupted run. uv will not reliably recreate over + // one, and a partial venv that looks present is worse than none. + if err := os.RemoveAll(req.VenvDir); err != nil { + return nil, err + } + if err := run(ctx, tc.UV, "venv", "--quiet", "--python", req.Health.PythonVersion, req.VenvDir); err != nil { + return nil, fmt.Errorf("create venv: %w", err) + } + if len(requirements) > 0 { + args := append([]string{"pip", "install", "--quiet", "--python", req.VenvDir}, requirements...) + if err := run(ctx, tc.UV, args...); err != nil { + return nil, fmt.Errorf("install runtime packages: %w", err) + } + } + if err := writeStamp(req.VenvDir, want); err != nil { + return nil, err + } + return res, nil +} + +// notAllowed reports imports the runtime rejects outright, so the caller can +// fail with the file and line rather than let ty report a bare +// unresolved-import. +func notAllowed(h *Health, imports []string) []string { + var out []string + seen := map[string]bool{} + for _, name := range imports { + if h.Allows(name) || seen[name] { + continue + } + seen[name] = true + out = append(out, name) + } + return out +} + +func (s Stamp) matches(other Stamp) bool { + if s.RuntimeDigest == "" || s.RuntimeDigest != other.RuntimeDigest { + return false + } + if s.PythonVersion != other.PythonVersion || len(s.Requirements) != len(other.Requirements) { + return false + } + for i := range s.Requirements { + if s.Requirements[i] != other.Requirements[i] { + return false + } + } + return true +} + +// ReadStamp reports what an environment was built from, for diagnostics. +func ReadStamp(venvDir string) (Stamp, error) { return readStamp(venvDir) } + +func readStamp(venvDir string) (Stamp, error) { + raw, err := os.ReadFile(filepath.Join(venvDir, StampName)) + if err != nil { + return Stamp{}, err + } + var s Stamp + if err := json.Unmarshal(raw, &s); err != nil { + return Stamp{}, err + } + // A venv whose interpreter has been removed is not reusable regardless of + // what the stamp claims. + if _, err := os.Stat(PythonPath(venvDir)); err != nil { + return Stamp{}, err + } + return s, nil +} + +func writeStamp(venvDir string, s Stamp) error { + raw, err := json.MarshalIndent(s, "", " ") + if err != nil { + return err + } + return os.WriteFile(filepath.Join(venvDir, StampName), append(raw, '\n'), 0o644) +} + +func run(ctx context.Context, name string, args ...string) error { + cmd := exec.CommandContext(ctx, name, args...) + out, err := cmd.CombinedOutput() + if err != nil { + msg := strings.TrimSpace(string(out)) + if msg == "" { + return err + } + return fmt.Errorf("%w: %s", err, msg) + } + return nil +} diff --git a/internal/pyenv/venv_test.go b/internal/pyenv/venv_test.go new file mode 100644 index 0000000..ae9defb --- /dev/null +++ b/internal/pyenv/venv_test.go @@ -0,0 +1,206 @@ +package pyenv + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// toolchain skips a test when uv is absent. These tests exercise real +// subprocesses on purpose: the failure modes worth catching here are the ones +// a fake would define away. +func toolchain(t *testing.T) *Toolchain { + t.Helper() + tc, err := FindToolchain() + if err != nil { + t.Skip("uv not installed") + } + return tc +} + +func TestFindToolchainErrorExplainsTheFix(t *testing.T) { + err := ErrNoToolchain{} + if !strings.Contains(err.Error(), "astral.sh/uv/install.sh") { + t.Fatalf("error should name the install command, got %v", err) + } + if !strings.Contains(err.Error(), "downloads the Python") { + t.Fatal("error should say uv supplies the interpreter, so users do not go hunting for Python 3.12") + } +} + +// A degraded report has no package list. Building from it would produce an +// empty environment in which every import fails — a confident, wrong answer. +func TestSyncRefusesADegradedReport(t *testing.T) { + h := loadFixture(t, "health_degraded.json") + _, err := Sync(context.Background(), &Toolchain{UV: "uv"}, SyncRequest{ + VenvDir: t.TempDir(), Health: h, Imports: []string{"requests"}, + }) + if err == nil { + t.Fatal("expected a refusal") + } + if !strings.Contains(err.Error(), "degraded") { + t.Fatalf("error should name the status, got %v", err) + } +} + +// Classification happens before any subprocess runs, so it can be asserted +// without uv. +func TestSyncClassifiesImportsBeforeInstalling(t *testing.T) { + h := loadFixture(t, "health_ok.json") + install, missing := h.Installable([]string{"requests", "notte", "pandas", "json"}) + + if len(install) == 0 { + t.Fatal("requests should be installable") + } + if len(missing) != 1 || missing[0] != "notte" { + t.Fatalf("allowed-but-absent = %v, want [notte]", missing) + } + if got := notAllowed(h, []string{"requests", "pandas", "os", "json"}); len(got) != 2 { + t.Fatalf("notAllowed = %v, want pandas and os", got) + } +} + +func TestStampMatching(t *testing.T) { + base := Stamp{RuntimeDigest: "sha256:a", PythonVersion: "3.12.0", Requirements: []string{"requests==1"}} + if !base.matches(base) { + t.Fatal("identical stamps should match") + } + for _, other := range []Stamp{ + {RuntimeDigest: "sha256:b", PythonVersion: "3.12.0", Requirements: []string{"requests==1"}}, + {RuntimeDigest: "sha256:a", PythonVersion: "3.11.0", Requirements: []string{"requests==1"}}, + {RuntimeDigest: "sha256:a", PythonVersion: "3.12.0", Requirements: []string{"requests==2"}}, + {RuntimeDigest: "sha256:a", PythonVersion: "3.12.0"}, + } { + if base.matches(other) { + t.Errorf("should not match: %+v", other) + } + } + // An empty digest is what a degraded report yields; it must never match. + empty := Stamp{PythonVersion: "3.12.0"} + if empty.matches(empty) { + t.Fatal("an empty digest must never satisfy a reuse check") + } +} + +// ty treats an unusable --python as fatal for the entire run, so an absent +// interpreter must be caught before ty is invoked rather than surfacing as a +// wall of unresolved imports. +func TestTypeCheckRefusesAMissingInterpreter(t *testing.T) { + dir := t.TempDir() + _, err := TypeCheck(context.Background(), &Toolchain{UV: "uv"}, dir, + filepath.Join(dir, "nonexistent-venv"), []string{"x.py"}) + if err == nil { + t.Fatal("a missing interpreter must be reported before ty runs") + } + if !strings.Contains(err.Error(), "no interpreter at") { + t.Fatalf("error should name the path, got %v", err) + } +} + +// The slow path: build a real environment from the real staging report. +func TestSyncBuildsAndReusesARealEnvironment(t *testing.T) { + if testing.Short() { + t.Skip("network") + } + tc := toolchain(t) + h := loadFixture(t, "health_ok.json") + venv := filepath.Join(t.TempDir(), "venv") + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + res, err := Sync(ctx, tc, SyncRequest{VenvDir: venv, Health: h, Imports: []string{"requests", "pydantic"}}) + if err != nil { + t.Fatalf("sync: %v", err) + } + if res.Reused { + t.Fatal("a fresh directory cannot be a reuse") + } + if _, err := os.Stat(PythonPath(venv)); err != nil { + t.Fatalf("no interpreter in the venv: %v", err) + } + + // The digest covers contract fields only, so an unchanged runtime reuses. + again, err := Sync(ctx, tc, SyncRequest{VenvDir: venv, Health: h, Imports: []string{"requests", "pydantic"}}) + if err != nil { + t.Fatal(err) + } + if !again.Reused { + t.Fatal("an unchanged runtime digest should reuse the environment") + } + + // A moved runtime must not reuse. + moved := *h + moved.RuntimeDigest = "sha256:different" + third, err := Sync(ctx, tc, SyncRequest{VenvDir: venv, Health: &moved, Imports: []string{"requests", "pydantic"}}) + if err != nil { + t.Fatal(err) + } + if third.Reused { + t.Fatal("a changed runtime digest must rebuild") + } +} + +// A matching stamp says the environment was built from the same inputs, not +// that it is still intact. Without --force a corrupted venv is unrecoverable +// except by deleting the directory by hand. +func TestSyncForceRebuildsAMatchingEnvironment(t *testing.T) { + if testing.Short() { + t.Skip("network") + } + tc := toolchain(t) + h := loadFixture(t, "health_ok.json") + venv := filepath.Join(t.TempDir(), "venv") + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + req := SyncRequest{VenvDir: venv, Health: h, Imports: []string{"requests"}} + + if _, err := Sync(ctx, tc, req); err != nil { + t.Fatal(err) + } + reused, err := Sync(ctx, tc, req) + if err != nil { + t.Fatal(err) + } + if !reused.Reused { + t.Fatal("an unchanged runtime should reuse") + } + + req.Force = true + forced, err := Sync(ctx, tc, req) + if err != nil { + t.Fatal(err) + } + if forced.Reused { + t.Fatal("--force must rebuild even when the stamp matches") + } + if _, err := os.Stat(PythonPath(venv)); err != nil { + t.Fatalf("forced rebuild left no interpreter: %v", err) + } +} + +// The environment must carry notte_core and notte_sdk whether or not the +// functions import them: the first is what Validate runs, and the second is +// what people write against, so an editor should resolve it before a function +// has been saved and re-synced. +func TestSyncAlwaysIncludesTheNotteBasePackages(t *testing.T) { + h := loadFixture(t, "health_ok.json") + install, _ := h.Installable(append(append([]string{}, AlwaysInstall...), "requests")) + + got := map[string]bool{} + for _, p := range install { + got[p.ImportName] = true + } + for _, want := range AlwaysInstall { + if !got[want] { + t.Errorf("%q must be installed regardless of imports; got %v", want, got) + } + } + if !got["requests"] { + t.Errorf("the function's own imports must still be installed; got %v", got) + } +}