Run a local model on your own GPU or CPU and open a coding agent on it. Local AI picks a model that was tested on your card (NVIDIA, Intel Arc Pro, AMD) or an x86-64 AVX2 CPU, downloads the weights from a pinned Hugging Face revision and checks them, and serves the model in Docker behind a keyed, OpenAI-compatible gateway on a Docker network of its own on this machine. From the bar you open pi, Claude Code, Codex, OpenCode, omp, Crush, Grok, Copilot or Hermes on it in a folder you choose, without editing their config. The panel has two tabs (home: your activity, what runs, your pinned models; models: every model this machine can run, to search, download, start and pin), a hardware page with each card's memory and temperature, a full-screen view with each model's Hugging Face card, keyboard navigation, and can share a running model on your tailnet.
omarchy plugin add https://github.com/0xSero/omarchy-local-ai --enableThen open Local AI in the bar and click Set up Local AI (once per machine; Omarchy asks for your password), pick a model on the models tab (or just type its name) and Start it, and open a coding agent on it. Install says what setup does, Requirements what it needs, and Remove how to take it all back out.
omarchy plugin add https://github.com/0xSero/omarchy-local-ai --enableOpen Local AI in the bar and click Set up Local AI. A terminal opens and runs two of Omarchy's own commands; Local AI itself runs nothing as root, and adding or updating the plugin never runs setup.
omarchy-setup-security-sudoless-docker, Omarchy's Sudoless Docker: it explains what that means, asks you to confirm, then asks "Password for you to turn on Sudoless Docker for Local AI". This login reaches Docker through the new group at once, so Local AI needs no logout or reboot (Omarchy may still suggest one at its next update).- NVIDIA only:
omarchy-pkg-add nvidia-container-toolkit, asking "Password for you to install NVIDIA container support for Local AI"; already installed, it asks nothing. The package describes your GPUs to Docker in/etc/cdi/nvidia.yaml, which Docker reads when a model starts, so Docker's configuration is untouched and it is not restarted.
If Local AI 6.0-6.4 was ever set up on this machine, by any account, it left two polkit files for that account that let its programs get root without a password. Local AI stops while any are there: the panel shows a red banner, and starting, sharing and setup refuse. Only root can delete them, so Local AI does not; docs/legacy-polkit.md says what they are and gives the commands to run in a terminal. The banner clears by itself once they are gone.
Setup ends by checking what the panel will show. Sharing a model on your tailnet needs you as Tailscale's operator; Omarchy's Tailscale install sets that, and setup tells you when it is not. After setup, starting, stopping, sharing, refreshing and removing never ask for a password. Sudoless Docker is root-equivalent, as Omarchy's own warning says; turn it off in Setup > Security.
- Omarchy with the Quattro shell and its bar. Local AI calls Omarchy's own commands:
omarchy-setup-security-sudoless-docker,omarchy-pkg-add,omarchy-hw-nvidia,omarchy-launch-tui,omarchy-launch-browser,omarchy-notification-sendandomarchy-cmd-present. - Docker,
jq,curl,flock,sha256sumandsha1sum. - A supported GPU (below), or an x86-64 CPU with AVX2 and enough RAM for its recipe. NVIDIA needs the driver and
nvidia-smi; setup adds the container toolkit. Docker 28 or later (Omarchy ships 29). AMD needs ROCm'samd-smi. - Optional: Tailscale, to share a model; a Hugging Face token in
~/.cache/huggingface/token, used for downloads when it exists. - Network access to
huggingface.co(weights),ghcr.io/sybil-solutions, historicalghcr.io/0xseropins, andghcr.io/ggml-org(the engine and gateway images, pinned by digest), andapi.github.comandraw.githubusercontent.com(the automatic catalog check),huggingface.coagain for model cards in full screen, andusage.sybilsolutions.ai(the install count, below, only once you allow it).
Qwen3.8-Flash-Next on one Intel Arc Pro B70 32 GB can keep its routed experts and complete n-gram table in host RAM. This recipe needs 128 GB available RAM and 85.14 GB disk, has a 64K context window, and does not require NVMe. It supports images, not video. Its memory and private shared-memory limits are carried by the v3 catalog; the panel checks host RAM before starting it. Update to 6.13.4 or newer to get this engine repository in the bundled catalog. Automatic catalog refreshes intentionally cannot add a recipe or change what one runs; those come with plugin updates.
- Validated models per card, or one across several.
recipes.jsonholds, for each supported hardware kind, every recipe accepted on that exact card or across 2 or 4 of them in local-ai-registry: download, load, a correctness check and speed at several context lengths. EXL3 weights on SGLang or vLLM come first and are recommended. The models tab lists every model this machine can run, on whichever of its cards it lands, as one searchable table; home shows the ones you pinned (running or downloading one pins it). A machine with no tested model says so and links the supported list. - What the machine needs. A recipe that keeps experts in system RAM or reads from disk while it serves (Qwen3.8-Flash-Next on a 3090 or a B70) says how much free RAM and disk it needs and whether the models folder must be on NVMe; it is offered only on a machine that has them; the others are listed as too big, saying what is missing. A recipe may also name the CPU threads its engine pins work to.
- Weights are downloaded as you, from the pinned revision, and every file is checked against the hash Hugging Face lists for it before it is used: the SHA-256 of a large (LFS) file, the Git blob SHA-1 of a small one (configs, tokenizers, chat templates, model code). A copy in your Hugging Face cache is reused only when it passes the same check.
- Containers. The engine runs on its own internal Docker network that the host has no address on, shared only with its gateway, with no published port and
no-new-privileges. A keyed gateway runs as you on the model's own Docker network, speaks the OpenAI, Anthropic and Responses APIs, and logs one line per answer; the tokens, speeds, charts and the activity grid on Home come from that log, summed once per new line rather than on every refresh. - Agents. pi, Claude Code, Codex, OpenCode, omp, Crush, Grok, Copilot and Hermes open in a terminal, in the folder you pick, pointed at the gateway. Nothing in their own config is touched. A model's details list the agents once, the chosen one checked; on the agents page the chosen one is the default for new models. Update to appears when mise or npm has a newer version of the agent. Click the folder row to browse folders inside the panel (click into a folder, or up, then Use this folder; recent folders are listed too). Open launches the chosen agent from home or the model's page. Agent choices on a running model do not change the default; the last folder picked remains the folder default. Updates use mise when it manages that agent, the native updater for standalone omp, Hermes, Claude Code and OpenCode, or npm for an existing npm installation. Existing sessions keep running.
- Share a running model on your tailnet with
tailscale serve(tailnet only, still keyed). Stop sharing on its page removes that share. Tailscale must be running and logged in. Setup preserves another account’s operator; ask that account to manage sharing. A failed unshare is logged and does not prevent stopping the model.
Supported: x86-64 AVX2 CPUs (LFM2.5-2.6B in system RAM), NVIDIA RTX 30, 40 and 50 series, RTX A6000, RTX Ada and RTX Pro Blackwell, Intel Arc Pro B70, and AMD Instinct MI300X and Radeon RX 6800 XT (with ROCm's amd-smi).
bin/omarchy-local-ai setup # one-time machine setup
bin/omarchy-local-ai registry # refresh the validated catalog
bin/omarchy-local-ai forget <recipe> # remove stopped managed weights
bin/omarchy-local-ai snapshot # what the card draws, as JSON
bin/omarchy-local-ai run <recipe> <gpu>[,<gpu>] # e.g. run qwen38-27b-exl3-3bpw-rtx3090-sglang-tp1 nvidia:0
bin/omarchy-local-ai stop <recipe>
bin/omarchy-local-ai open <recipe> # the chosen agent on it, in a terminal
bin/omarchy-local-ai update <agent> # update the installed harness
bin/omarchy-local-ai set agent|folder <value> [recipe]
bin/omarchy-local-ai set ping on|off # whether this install is counted (off until you turn it on)
bin/omarchy-local-ai share <recipe> [off]
bin/omarchy-local-ai logCPU recipes use system RAM and start without GPU devices. Recipes that offload GPU work into RAM or NVMe remain hardware-specific and list their requirements.
Each running model's gateway answers at its address on the model's own Docker network, http://<address>:12434/v1: the model's page shows it under this machine, hidden until clicked, with a copy. The key is in ~/.local/state/omarchy/local-ai/gateway.key. Send it there and nowhere else, as Local AI itself and the agents it opens do: only Docker assigns addresses on that bridge and no host interface holds them, so no other account on the machine can listen there, while 127.0.0.1:<port> is free for any account to bind the moment the gateway stops. The gateway's port on 127.0.0.1 (from 12434) carries no key: Local AI only checks there that the gateway is up and that it refuses a request without the key.
bin/omarchy-remove-ai-local stops this user’s managed models and deletes their containers, engine images, weights and settings. Then omarchy plugin remove sero.local-ai.
| File | Role |
|---|---|
bin/omarchy-local-ai |
The backend, one bash file |
Model.js |
Pure functions: the snapshot in, the view out |
Panel.qml |
Draws the view and runs the backend's verbs |
recipes.json |
The vendored recipes, one card kind per line (make sync) |
docs/design.md has the design and test/all runs the tests.
Each engine is on its own internal Docker network, isolated from the host
(com.docker.network.bridge.gateway_mode_ipv4=isolated, Docker 28 and later):
the host has no address on it and no route to it, so no account on the machine
can reach the engine, whatever it listens on or whether it checks a key of its
own. Only the model's gateway joins that network; it is the one way in, and it
requires the key. A model an older version started is moved to its private
network while it runs, the first time the panel refreshes after the update.
Images are pinned by SHA-256. The gateway runs as the calling user, with no Linux capabilities, a read-only root filesystem, a bounded temporary directory and external DNS disabled. Docker service discovery still resolves the engine; usage records remain writable in their dedicated directory. Existing containers receive these restrictions when stopped and started again.
Model engines still have writable container filesystems; the internal network leaves them no outbound network access. Digest pins and build attestations establish image identity; they do not make untrusted code safe. Engine restrictions need per-engine hardware validation before rollout. Disabling gateway DNS does not block outbound IP connections.
Nothing, until you allow it. Home asks once whether to share minimal, anonymous
usage counts (No thanks or Allow; either answer is kept, and Settings
changes it later). An install from before this question is treated as not having
answered: off until you allow it. Once allowed, to count how many machines use it,
Local AI sends one small message each time it checks the registry for new models
(every 12 hours while the bar runs) to
https://usage.sybilsolutions.ai/v1/local-ai/ping:
{"id": "9f2c…", "version": "6.15.0", "hw": ["rtx-3090"], "running": 1, "launched": true,
"actions": {"started": 2, "failed": 0, "opened": 1, "downloaded": 1}}id is a random number made on the first message and kept in
~/.local/state/omarchy/local-ai/install-id; it is not derived from the machine
or you, and Remove deletes it. hw lists the card kinds Local AI has recipes for
that it found, running how many models run, launched whether a model has ever
answered on this machine. actions are how many models started, starts that
failed, agents opened and models downloaded since the last message that arrived:
totals only, never which model, agent or folder. They wait in
~/.local/state/omarchy/local-ai/counts.json until a message arrives, so no
action is lost or sent twice, and nothing extra is sent when you use Local AI.
The server keeps those fields and the day, and no IP address or other header.
Its code, the D1 migrations and the deploy config are in usage/ (the
worker local-ai-usage and its usage.sybilsolutions.ai route); the public
totals are the same worker's GET /v1/local-ai/stats.
Turn it on or off in the panel under Settings (send usage reports), or with
omarchy-local-ai set ping on|off; DO_NOT_TRACK=1 in the bar's environment
keeps it off whatever the setting. Turning it off also deletes the actions not
yet sent, and nothing is counted while it is off or not yet allowed. A failed or
refused message changes nothing.
New models, and any change to how a model runs, come with plugin updates, which
the panel offers on home. Between updates, once the catalog is 12 hours old,
Local AI checks the latest published registry (at most once an hour) and says so
only when it changed the list; the catalog lives in your cache
(~/.cache/omarchy/local-ai/v4). A refresh can only withdraw, reorder or relabel
the recipes the plugin ships: each recipe it lists that the plugin's own
recipes.json has (same id, same card kind) becomes that shipped recipe, and only
its name, family, format, size, score and capability flags are taken from the
registry, each when it is short plain text, a finite number or a boolean.
Everything that decides what runs or how an agent is set up (image, launch,
weights, context, needs and the rest) is always the shipped recipe's, and a recipe
the plugin does not ship is left out until a plugin update ships it. Downloads are
pinned to a registry commit, validated before an atomic replacement, and failures
keep the previous catalog. A catalog an earlier version refreshed is not read; a
refresh installs a catalog only once the panel can be drawn from it, and a catalog
that later cannot be read or drawn is skipped for the shipped recipes (the log
says why) until the next refresh replaces it. A model whose recipe an update withdrew keeps Stop; its download,
and any folder no recipe here uses, is listed on home under No longer offered
with Remove.
Refreshing does not stop running models or download model weights. Offload
recipes are offered only when their RAM, disk and storage requirements fit.
A recipe that generates a pack first declares its output budget separately from raw weights. Local AI prepares with the pinned serving image, verifies the pack and publishes it from private staging. Each reuse is verified. Stop retains the completed pack; Remove download removes it only when no managed model uses it. The v3 contract supports these operations; research models still require their own accepted image, hardware and fidelity evidence before they appear here.
On the models tab, open a model's row to Start it, Download it without starting, Pin it to home, or see its details. Remove download deletes that model's managed weights after it is stopped; shared weights in use by another managed model are protected. The recipe stays available to download again. Files in your separate Hugging Face cache are kept, so hard-linked files there may continue to occupy disk space.
The equivalent commands are omarchy-local-ai registry, download <recipe> [off],
pin <recipe> [off] and forget <recipe> (or the plugin's bin/omarchy-local-ai).
MIT. Model-maker marks in Logos.js come from lobe-icons (MIT) and, for Intel and AMD, Simple Icons (CC0). The agent, model-maker and GPU-maker logos are trademarks of their owners, shown to identify compatibility only; Local AI is not affiliated with or endorsed by any of them.