-
Notifications
You must be signed in to change notification settings - Fork 233
Add SurrealDB as a community world #1579
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 2 commits
3e67b36
4b50729
fcf15af
9351bed
9dd1fa8
ce96114
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| { | ||
| "title": "World", | ||
| "pages": ["local-world", "vercel-world", "postgres-world"] | ||
| "pages": ["local-world", "vercel-world", "postgres-world", "surrealdb-world"] | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,251 @@ | ||
| --- | ||
| title: SurrealDB World | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. for community world, the docs page are hardcoded (for example see this). As long as the repo link is set in the worlds-manifest.json, our docs will point to your README so it's easier for you to just maintain the README on your repo, and so we can standardize and easily maintain the structure/formatting including the performance benchmark stats |
||
| description: Community world using SurrealDB with LIVE SELECT for queueing and real-time streaming. | ||
| type: integration | ||
| summary: Deploy workflows to your own infrastructure using SurrealDB. | ||
| prerequisites: | ||
| - /docs/deploying | ||
| related: | ||
| - /docs/deploying/world/local-world | ||
| - /docs/deploying/world/postgres-world | ||
| --- | ||
|
|
||
| The SurrealDB World is a community-maintained backend for self-hosted deployments. It uses [SurrealDB](https://surrealdb.com) for durable storage and leverages `LIVE SELECT` for reactive queue processing and real-time streaming. | ||
|
|
||
| Use the SurrealDB World when you want a single-binary database that handles storage, queuing, and streaming without additional infrastructure like a separate job queue. | ||
|
|
||
| <Callout type="info"> | ||
| This is a community world maintained at [sepcnt/workflow-world-surrealdb](https://github.com/sepcnt/workflow-world-surrealdb). For bug reports and feature requests, please use that repository. | ||
| </Callout> | ||
|
|
||
| ## Installation | ||
|
|
||
| Install the SurrealDB World package in your workflow project: | ||
|
|
||
| ```package-install | ||
| workflow-world-surrealdb | ||
| ``` | ||
|
|
||
| Configure the required environment variables: | ||
|
|
||
| ```bash title=".env" | ||
| WORKFLOW_TARGET_WORLD="workflow-world-surrealdb" | ||
| WORKFLOW_SURREAL_URL="ws://127.0.0.1:8000/rpc" | ||
| WORKFLOW_SURREAL_NAMESPACE="workflow" | ||
| WORKFLOW_SURREAL_DATABASE="workflow" | ||
| WORKFLOW_SURREAL_USERNAME="root" | ||
| WORKFLOW_SURREAL_PASSWORD="root" | ||
| ``` | ||
|
|
||
| Run the setup script to create the necessary tables and functions in your database: | ||
|
|
||
| ```bash | ||
| npx workflow-surreal-setup | ||
| ``` | ||
|
|
||
| <Callout type="info"> | ||
| The setup is idempotent and can safely be run as a post-deployment lifecycle script. | ||
| </Callout> | ||
|
|
||
| ## Running SurrealDB | ||
|
|
||
| Start a SurrealDB instance using Docker: | ||
|
|
||
| ```bash | ||
| docker run -d -p 8000:8000 \ | ||
| surrealdb/surrealdb:v3 \ | ||
| start --user root --pass root --bind 0.0.0.0:8000 memory | ||
| ``` | ||
|
|
||
| Or use the included `docker-compose.yml`: | ||
|
|
||
| ```bash | ||
| docker compose up -d | ||
| ``` | ||
|
|
||
| For persistent storage, replace `memory` with a file path: | ||
|
|
||
| ```bash | ||
| docker run -d -p 8000:8000 -v surrealdb-data:/data \ | ||
| surrealdb/surrealdb:v3 \ | ||
| start --user root --pass root --bind 0.0.0.0:8000 \ | ||
| "surrealkv:///data/workflow.db?versioned=true" | ||
| ``` | ||
|
|
||
| ## Starting the World | ||
|
|
||
| To subscribe to the queue via LIVE SELECT, your workflow app needs to start the world on server start. Here are examples for a few frameworks: | ||
|
|
||
| <Tabs items={["Next.js", "SvelteKit", "Nitro"]}> | ||
|
|
||
| <Tab value="Next.js"> | ||
|
|
||
| Create an `instrumentation.ts` file in your project root: | ||
|
|
||
| ```ts title="instrumentation.ts" lineNumbers | ||
| export async function register() { | ||
| if (process.env.NEXT_RUNTIME !== "edge") { | ||
| const { getWorld } = await import("workflow/runtime"); | ||
| await getWorld().start?.(); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| <Callout type="info"> | ||
| Learn more about [Next.js Instrumentation](https://nextjs.org/docs/app/guides/instrumentation). | ||
| </Callout> | ||
|
|
||
| </Tab> | ||
|
|
||
| <Tab value="SvelteKit"> | ||
|
|
||
| Create a `src/hooks.server.ts` file: | ||
|
|
||
| ```ts title="src/hooks.server.ts" lineNumbers | ||
| import type { ServerInit } from "@sveltejs/kit"; | ||
|
|
||
| export const init: ServerInit = async () => { | ||
| const { getWorld } = await import("workflow/runtime"); | ||
| await getWorld().start?.(); | ||
| }; | ||
| ``` | ||
|
|
||
| <Callout type="info"> | ||
| Learn more about [SvelteKit Hooks](https://svelte.dev/docs/kit/hooks). | ||
| </Callout> | ||
|
|
||
| </Tab> | ||
|
|
||
| <Tab value="Nitro"> | ||
|
|
||
| Create a plugin to start the world on server initialization: | ||
|
|
||
| ```ts title="plugins/start-surreal-world.ts" lineNumbers | ||
| import { defineNitroPlugin } from "nitro/~internal/runtime/plugin"; | ||
|
|
||
| export default defineNitroPlugin(async () => { | ||
| const { getWorld } = await import("workflow/runtime"); | ||
| await getWorld().start?.(); | ||
| }); | ||
| ``` | ||
|
|
||
| Register the plugin in your config: | ||
|
|
||
| ```ts title="nitro.config.ts" | ||
| import { defineNitroConfig } from "nitropack"; | ||
|
|
||
| export default defineNitroConfig({ | ||
| modules: ["workflow/nitro"], | ||
| plugins: ["plugins/start-surreal-world.ts"], | ||
| }); | ||
| ``` | ||
|
|
||
| <Callout type="info"> | ||
| Learn more about [Nitro Plugins](https://v3.nitro.build/docs/plugins). | ||
| </Callout> | ||
|
|
||
| </Tab> | ||
|
|
||
| </Tabs> | ||
|
|
||
| <Callout type="info"> | ||
| The SurrealDB World requires a long-lived worker process that maintains LIVE SELECT subscriptions. This does not work on serverless environments. For Vercel deployments, use the [Vercel World](/worlds/vercel) instead. | ||
| </Callout> | ||
|
|
||
| ## Configuration | ||
|
|
||
| All configuration options can be set via environment variables or programmatically via `createWorld()`. | ||
|
|
||
| ### `WORKFLOW_SURREAL_URL` | ||
|
|
||
| SurrealDB WebSocket URL. Supports `ws://`, `wss://`, `http://`, and `https://` protocols (HTTP is auto-converted to WS). | ||
|
|
||
| Default: `ws://127.0.0.1:8000/rpc` | ||
|
|
||
| ### `WORKFLOW_SURREAL_NAMESPACE` | ||
|
|
||
| SurrealDB namespace. Default: `workflow` | ||
|
|
||
| ### `WORKFLOW_SURREAL_DATABASE` | ||
|
|
||
| SurrealDB database. Default: `workflow` | ||
|
|
||
| ### `WORKFLOW_SURREAL_USERNAME` / `WORKFLOW_SURREAL_PASSWORD` | ||
|
|
||
| Root or namespace-level credentials for authentication. | ||
|
|
||
| ### `WORKFLOW_SURREAL_TOKEN` | ||
|
|
||
| Alternative to username/password. Use a JWT token for authentication. | ||
|
|
||
| ### `WORKFLOW_SURREAL_QUEUE_CONCURRENCY` | ||
|
|
||
| Number of concurrent jobs to process. Default: `10` | ||
|
|
||
| ### `WORKFLOW_SURREAL_QUEUE_LANE_SHARDS` | ||
|
|
||
| Number of queue lane partitions for horizontal scalability. Default: `16` | ||
|
|
||
| ### `WORKFLOW_SURREAL_QUEUE_LEASE_MS` | ||
|
|
||
| Job lease duration in milliseconds before a job is considered stale. Default: `300000` (5 minutes) | ||
|
|
||
| ### `WORKFLOW_SURREAL_QUEUE_HEARTBEAT_MS` | ||
|
|
||
| Heartbeat interval for extending job leases. Default: `10000` (10 seconds) | ||
|
|
||
| ### Programmatic configuration | ||
|
|
||
| {/* @skip-typecheck: incomplete code sample */} | ||
| ```typescript title="workflow.config.ts" lineNumbers | ||
| import { createWorld } from "workflow-world-surrealdb"; | ||
|
|
||
| const world = createWorld({ | ||
| url: "ws://127.0.0.1:8000/rpc", | ||
| namespace: "workflow", | ||
| database: "workflow", | ||
| auth: { | ||
| username: "root", | ||
| password: "root", | ||
| }, | ||
| queueConcurrency: 20, | ||
| queueLaneShards: 32, | ||
| }); | ||
| ``` | ||
|
|
||
| ## How It Works | ||
|
|
||
| The SurrealDB World uses SurrealDB as a unified backend: | ||
|
|
||
| - **Storage** — Workflow runs, events, steps, and hooks are stored in SurrealDB tables with SCHEMAFULL validation | ||
| - **Job Queue** — Lane-sharded queue with `LIVE SELECT` subscriptions for instant wake-up and automatic lease recovery | ||
| - **Streaming** — `LIVE SELECT` on stream chunk tables enables real-time event distribution with periodic reconciliation | ||
|
|
||
| This architecture keeps the infrastructure minimal — a single SurrealDB instance handles storage, queuing, and streaming without additional services. For implementation details, see the [source code](https://github.com/sepcnt/workflow-world-surrealdb). | ||
|
|
||
| ## Deployment | ||
|
|
||
| Deploy your application to any cloud that supports long-running servers: | ||
|
|
||
| - Docker containers | ||
| - Kubernetes clusters | ||
| - Virtual machines | ||
| - Platform-as-a-Service providers (Railway, Render, Fly.io, etc.) | ||
|
|
||
| Ensure your deployment has: | ||
| 1. Network access to your SurrealDB instance | ||
| 2. Environment variables configured correctly | ||
| 3. The `start()` function called on server initialization | ||
|
|
||
| <Callout type="info"> | ||
| The SurrealDB World is not compatible with Vercel deployments. On Vercel, workflows automatically use the [Vercel World](/worlds/vercel) with zero configuration. | ||
| </Callout> | ||
|
|
||
| ## Limitations | ||
|
|
||
| - **Requires long-running process** — Must call `start()` on server initialization; not compatible with serverless platforms | ||
| - **SurrealDB infrastructure** — Requires a SurrealDB instance (self-hosted or [Surreal Cloud](https://surrealdb.com/cloud)) | ||
| - **Not compatible with Vercel** — Use the [Vercel World](/worlds/vercel) for Vercel deployments | ||
| - **Community-maintained** — Not an official Vercel package; support via [GitHub Issues](https://github.com/sepcnt/workflow-world-surrealdb/issues) | ||
|
|
||
| For local development, use the [Local World](/worlds/local) which requires no external services. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this isn't needed. this page should probably be deprecated tbh since the worlds top level docs pages replaced the deploying docs