Add Building Comfort and Curbside Pickup tutorials (Python + Streamlit) by danielgerlag · Pull Request #1 · drasi-project/drasi-python

danielgerlag · 2026-07-31T18:51:43Z

Why

drasi-python has API docs and small examples, but no end-to-end, hands-on tutorials that show someone building a complete, reactive app with drasi-lib. This ports two of the Drasi Server tutorials to the Python library so there is a runnable, guided path from a data source through continuous queries to a live reaction.

Both tutorials keep the upstream data story but replace the built-in dashboard reaction (and, for Curbside, the separate change-driving console) with a single Streamlit UI powered by a Python reaction — the reaction is the application.

What's here

Two self-contained tutorials under a new top-level tutorials/ directory:

Building Comfort — PostgreSQL CDC over a Building -> Floor -> Room hierarchy, six continuous queries with synthetic joins and aggregation, and a Streamlit UI with simulation mode plus per-room set/reset controls.
Curbside Pickup — a cross-database demo: a PostgreSQL orders store and a MySQL vehicles store joined by license plate, six queries (two that join across both databases, plus a temporal drasi.trueFor delay), rendered and driven from one integrated Streamlit UI (inline row buttons, a sidebar reset, and a color-coded SQL activity log).

Approach and shared design

Embedded engine, one process. Each app runs the Drasi engine on a background asyncio thread created once via @st.cache_resource; a single add_python_reaction over all queries maintains a thread-safe, per-query snapshot that Streamlit renders on a ~1s refresh. Controls write plain UPDATEs to the databases (psycopg for Postgres, PyMySQL for MySQL), so Drasi observes them through CDC exactly like a real app.
Authored once, rendered twice. Each tutorial's write-up lives in _index.md, mounted into the existing Hugo docs site; scripts/render-tutorials.py generates the GitHub-facing README.md from it, gated in CI with --check.
Reproducible envs. Dev containers, docker-compose for the databases, and bash + PowerShell setup/start/cleanup scripts.

Notes for reviewers

Everything was verified end-to-end against real databases: bootstrap, driving changes via CDC, the aggregate roll-ups, the cross-database join, and the drasi.trueFor delay firing at ~10s. The screenshots in each tutorial are captured from the running apps.
A couple of docs/infra touches support the tutorials: the website mounts ../tutorials/<name> into the content tree, and a small Hugo render hook adds Cypher syntax highlighting (Chroma has no Cypher lexer).
.streamlit/config.toml sets headless = true / address = 0.0.0.0 so the app starts non-interactively and is reachable through a dev container's forwarded port (without headless, Streamlit's first-run email prompt blocks startup).
The tutorials/ layout is set up so more tutorials can be added the same way.

Opening as a draft for feedback on structure and copy before finalizing.

Port the upstream Drasi Server Building Comfort tutorial to drasi-lib. Keeps PostgreSQL CDC, six continuous queries with synthetic joins and aggregation, but replaces the dashboard reaction with a Python reaction that drives a Streamlit UI. The UI also drives the demo (simulation toggle, reset, and per-room set), writing SQL UPDATEs to Postgres that Drasi observes via CDC. - tutorials/building-comfort/: runnable app (app.py + demo/ engine, queries, config), database assets, env/setup/start/cleanup scripts, requirements. - Authored once in _index.md (mounted into the website/ Hugo docs) with the GitHub README generated by scripts/render-tutorials.py (CI --check gate). - Dev container config, tutorials index, and repo README pointer. Verified end-to-end against the real engine and Postgres: bootstrap, break / reset / set a room via CDC, and simulation; screenshot captured from the app. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Rewrite each of the six queries as a named `Query` (id, key, cypher, joins) with the Cypher and comfort formula written out in full, instead of splicing a shared formula string into positional tuples. Drop the derived ALL_QUERY_IDS and KEY_FIELDS module globals; the engine reads id/cypher/joins/key off each Query. Behaviour is unchanged (re-verified end-to-end). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Port the upstream Curbside Pickup tutorial to drasi-lib. It joins a PostgreSQL orders store and a MySQL vehicles store by license plate, with six continuous queries — two that join across both databases, and a temporal `delay` query using drasi.trueFor. Upstream used the dashboard reaction plus a separate Node operations console; this collapses both into one integrated Streamlit UI powered by a single Python reaction: the panels render live and their inline buttons (plus a sidebar reset and a color-coded SQL activity log) drive the changes. - tutorials/curbside-pickup/: app.py + demo/ (engine, queries, config), two-DB database assets, env/setup/start/cleanup scripts, requirements (psycopg + PyMySQL). - Authored in _index.md (mounted into the website), README generated. - Website mount + tutorials index card, tutorials/README row, dev container. Verified end-to-end against real PostgreSQL and MySQL: seed, cross-DB delivery match, the drasi.trueFor delay firing at ~10s, and reset; screenshot captured. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Without `--server.headless=true`, Streamlit stops at its interactive first-run email prompt on a fresh machine and never binds its port, so the tutorial app is unreachable (ERR_CONNECTION_REFUSED) in a dev container or Codespace. Add a `.streamlit/config.toml` (headless = true, address = 0.0.0.0) to each tutorial so any launch is non-interactive and reachable through a forwarded port, and pass the same flags explicitly in the start-demo scripts. Verified with a clean HOME that the app now starts with no prompt and serves on 8501. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Wrap the six panels in st.container(border=True) so each order/vehicle/matched/ delayed box is visually distinct, and refresh the screenshot. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Chroma has no Cypher lexer, so ```cypher fences rendered unhighlighted (language-fallback) on the site. Add a code-block render hook that applies a lightweight, Cypher-aware highlight using the same Chroma CSS classes the theme already styles (keywords, functions incl. Drasi's drasi.* extensions, labels, numbers, comments). The generated GitHub READMEs keep their ```cypher fences. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds two end-to-end, runnable tutorials to drasi-python (with Streamlit UIs powered by Python reactions), and wires them into both the Hugo docs site and GitHub-facing READMEs so the narrative and code stay in sync.

Changes:

Add two self-contained tutorial projects under tutorials/ (Building Comfort + Curbside Pickup), each with dockerized databases, setup/cleanup scripts, and a Streamlit UI that embeds the Drasi engine.
Add a renderer (scripts/render-tutorials.py) to generate GitHub-friendly README.md files from Hugo/Docsy _index.md sources, and gate freshness in CI.
Extend the docs site to surface tutorials and improve Cypher code block highlighting.

Reviewed changes

Copilot reviewed 53 out of 55 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
website/layouts/_default/_markup/render-codeblock-cypher.html	Hugo render hook to provide Cypher highlighting using Chroma-like CSS classes.
website/content/docs/tutorials/_index.md	Adds Tutorials landing page and cards for the two tutorials.
website/config.toml	Mounts `tutorials/**/_index.md` and images into the Hugo content tree.
tutorials/README.md	Top-level tutorials index and explanation of single-source authoring workflow.
tutorials/curbside-pickup/scripts/start-demo.sh	Starts databases + Streamlit app for Curbside tutorial (bash).
tutorials/curbside-pickup/scripts/start-demo.ps1	Starts databases + Streamlit app for Curbside tutorial (PowerShell).
tutorials/curbside-pickup/scripts/setup-database.sh	Brings up Postgres/MySQL via compose and seeds both (bash).
tutorials/curbside-pickup/scripts/setup-database.ps1	Brings up Postgres/MySQL via compose and seeds both (PowerShell).
tutorials/curbside-pickup/scripts/cleanup.sh	Tears down Curbside containers (bash).
tutorials/curbside-pickup/scripts/cleanup.ps1	Tears down Curbside containers (PowerShell).
tutorials/curbside-pickup/requirements.txt	Python deps for Curbside tutorial (drasi-lib, streamlit, psycopg, PyMySQL).
tutorials/curbside-pickup/README.md	Generated GitHub README for Curbside tutorial.
tutorials/curbside-pickup/demo/queries.py	Defines Curbside’s six continuous queries + synthetic join.
tutorials/curbside-pickup/demo/engine.py	Embedded Drasi engine + snapshot + DB write helpers for the Streamlit UI.
tutorials/curbside-pickup/demo/config.py	Env-driven configuration for Curbside sources and DB connections.
tutorials/curbside-pickup/demo/init.py	Demo package export surface.
tutorials/curbside-pickup/database/postgres-init.sql	Postgres schema/seed/publication for orders (CDC).
tutorials/curbside-pickup/database/mysql-init.sql	MySQL schema/seed/replication grants for vehicles (binlog CDC).
tutorials/curbside-pickup/database/docker-compose.yml	Compose stack for Postgres + MySQL configured for CDC.
tutorials/curbside-pickup/app.py	Streamlit UI that renders query snapshots and drives DB updates.
tutorials/curbside-pickup/.streamlit/config.toml	Streamlit server config for headless + bind-all in containers.
tutorials/curbside-pickup/.gitignore	Tutorial-local ignores (.env, venv, engine data/logs).
tutorials/curbside-pickup/.env.example	Example env overrides for ports/credentials.
tutorials/curbside-pickup/_index.md	Source-of-truth Docsy/Hugo tutorial write-up for Curbside.
tutorials/building-comfort/scripts/start-demo.sh	Starts database + Streamlit app for Building Comfort (bash).
tutorials/building-comfort/scripts/start-demo.ps1	Starts database + Streamlit app for Building Comfort (PowerShell).
tutorials/building-comfort/scripts/setup-database.sh	Brings up Postgres with logical replication and seeds building hierarchy (bash).
tutorials/building-comfort/scripts/setup-database.ps1	Brings up Postgres with logical replication and seeds building hierarchy (PowerShell).
tutorials/building-comfort/scripts/cleanup.sh	Tears down Building Comfort container (bash).
tutorials/building-comfort/scripts/cleanup.ps1	Tears down Building Comfort container (PowerShell).
tutorials/building-comfort/requirements.txt	Python deps for Building Comfort tutorial.
tutorials/building-comfort/README.md	Generated GitHub README for Building Comfort tutorial.
tutorials/building-comfort/demo/queries.py	Defines Building Comfort’s six continuous queries + synthetic joins.
tutorials/building-comfort/demo/engine.py	Embedded Drasi engine + snapshot + simulation + DB write helpers.
tutorials/building-comfort/demo/config.py	Env-driven configuration for Building Comfort.
tutorials/building-comfort/demo/init.py	Demo package export surface.
tutorials/building-comfort/database/init.sql	Postgres schema/seed/publication/slot for building hierarchy (CDC).
tutorials/building-comfort/database/docker-compose.yml	Compose stack for Postgres configured for logical replication.
tutorials/building-comfort/app.py	Streamlit UI for Building Comfort (rendering + controls + simulation toggle).
tutorials/building-comfort/.streamlit/config.toml	Streamlit server config for headless + bind-all in containers.
tutorials/building-comfort/.gitignore	Tutorial-local ignores (.env, venv, engine data/logs).
tutorials/building-comfort/.env.example	Example env overrides for Building Comfort ports/credentials.
tutorials/building-comfort/_index.md	Source-of-truth Docsy/Hugo tutorial write-up for Building Comfort.
scripts/render-tutorials.py	Renders Docsy/Hugo shortcodes into plain Markdown READMEs; supports `--check`.
README.md	Adds a Tutorials section pointing at `tutorials/` and docs site.
.github/workflows/website.yml	Ensures docs build runs when `tutorials/**` changes.
.github/workflows/ci.yml	Adds a CI gate to ensure generated tutorial READMEs are current.
.devcontainer/curbside-pickup/post-create.sh	Installs tutorial dependencies in the Curbside dev container.
.devcontainer/curbside-pickup/devcontainer.json	Dev container definition for Curbside tutorial (ports, DinD, env).
.devcontainer/curbside-pickup/devcontainer-lock.json	Locks devcontainer feature versions for Curbside tutorial.
.devcontainer/building-comfort/post-create.sh	Installs tutorial dependencies in the Building Comfort dev container.
.devcontainer/building-comfort/devcontainer.json	Dev container definition for Building Comfort tutorial (ports, DinD, env).
.devcontainer/building-comfort/devcontainer-lock.json	Locks devcontainer feature versions for Building Comfort tutorial.

Suppressed comments (3)

tutorials/curbside-pickup/scripts/setup-database.sh:75

Same as PostgreSQL: this loop can time out without stopping the script. If MySQL never becomes ready, the next mysql seed step will fail with a generic docker exec/mysql error rather than a clear timeout message.
tutorials/curbside-pickup/scripts/setup-database.ps1:63
After the wait loop, the script doesn't verify MySQL became ready. If it never responds, the script continues and the seed step fails with a generic error. Add an explicit timeout check and exit early.
tutorials/building-comfort/demo/engine.py:235
Prefer asyncio.get_running_loop() inside a coroutine for clarity; get_event_loop() can behave differently depending on thread/loop policy.

        loop = asyncio.get_event_loop()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

danielgerlag · 2026-07-31T22:36:24Z

+echo "Waiting for PostgreSQL to be ready..."
+for i in $(seq 1 30); do
+    if docker exec curbside-pickup-postgres pg_isready -h localhost -U postgres -d RetailOperations &> /dev/null; then
+        echo "PostgreSQL is ready!"
+        break
+    fi
+    echo "  Waiting... ($i/30)"
+    sleep 2
+done
+


Fixed in b24a2f5. Both loops now set a readiness flag and the script exits with a clear message (and a 'docker logs ...' hint) if Postgres/MySQL never become ready, before it reaches seeding. Applied the same fix to the building-comfort and getting-started setup-database.sh scripts.

danielgerlag · 2026-07-31T22:36:26Z

+        docker exec curbside-pickup-postgres pg_isready -h localhost -U postgres -d RetailOperations 2>&1 | Out-Null
+        if ($LASTEXITCODE -eq 0) { Write-Host "PostgreSQL is ready!"; break }
+        Write-Host "  Waiting... ($i/30)"; Start-Sleep -Seconds 2
+    }


Fixed in b24a2f5. Added $pgReady/$mysqlReady flags with an explicit '-not ready' check that exits early. The getting-started .ps1 got the same treatment; building-comfort's .ps1 already had this pattern.

danielgerlag · 2026-07-31T22:36:27Z

+The [Building Comfort](https://github.com/drasi-project/drasi-python/tree/main/tutorials/building-comfort)
+tutorial builds a smart-building monitor from PostgreSQL CDC, six continuous
+queries with synthetic joins, and a Python reaction that drives a live Streamlit
+UI. They also render on the [documentation site](https://drasi-project.github.io/drasi-python/docs/tutorials/).


Fixed in b24a2f5 - reworded to 'The tutorials also render on the documentation site' so it reads as plural/collective.

danielgerlag · 2026-07-31T22:36:28Z

+                    cur.execute('SELECT id FROM "Room" ORDER BY id')
+                    return [row[0] for row in cur.fetchall()]
+
+        return await asyncio.get_event_loop().run_in_executor(None, _query)


Fixed in b24a2f5 - switched both call sites (_load_room_ids and _simulate) to asyncio.get_running_loop().

danielgerlag · 2026-07-31T22:36:30Z

+        if enabled and self._sim_task is None:
+            if not self._room_ids:
+                self._room_ids = await self._load_room_ids()
+            self._sim_task = asyncio.ensure_future(self._simulate())


Fixed in b24a2f5 - now uses asyncio.create_task(self._simulate()).

Port the Drasi Server Getting Started tutorial to drasi-lib as a no-UI console app. Drasi's PostgreSQL source watches a `message` table and three continuous queries build up in complexity -- a Hello World filter, a count aggregation, and a temporal `inactive-people` (drasi.trueLater + datetime.realtime). Instead of the log + SSE reactions, a single Python reaction prints each query change to the console; you drive changes by running SQL directly (add/delete-message scripts and raw psql), in the same two-terminal model as the other tutorials. - tutorials/getting-started/: app.py + demo/ (queries, config), database assets, setup/start/add-message/delete-message/cleanup scripts, requirements (drasi-lib). - Authored in _index.md (mounted into the website) with a real console transcript; README generated. Website mount + tutorials index card (weight 10) + tutorials/README row + dev container. - Extend the Cypher render hook to also highlight drasi.trueLater / realtime. Verified end-to-end against a real PostgreSQL: the filter and aggregation react to inserts/deletes, and inactive-people fires ~20s after a sender's last message. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The drive-change steps had the bash tab call a dedicated script while the PowerShell tab ran docker exec. Standardize on a single `docker exec ... psql` command that is identical in both shells, and remove the dedicated add-message/delete-message scripts. To keep the command free of shell-specific quote escaping, rename the reserved `from` column to `sender` (so no quoted identifier is needed); the query property becomes m.sender. Re-verified end-to-end: the documented docker exec INSERT/DELETE drive the filter, aggregation, and the temporal inactive-people query. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Replace the if/elif chains in the reaction callbacks with match/case, which reads the diff's type and destructures its payload in one step. Handle the real diff shapes carefully: aggregation changes (e.g. a count) arrive with type "aggregation" and a null `data`, carrying the row in `after`, and every diff always has data/before/after keys (null where N/A). So the printing reaction matches "UPDATE" | "aggregation" on before/after, and the keyed-store reactions use dict() patterns (after-then-data) so a null payload key never wins. Verified equivalent to the previous logic across all shapes, and end-to-end: message-count aggregation prints, and building-comfort's avg roll-ups update without error. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The docs already add Docsy's click-to-copy button to every code block, but the project styles set it to opacity:0 (visible only when hovering the block), so it was easy to miss. Show it at rest (opacity 0.7, with a subtle border/background) and at full strength on hover. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Inline the Postgres source config and the three queries directly in app.py and drop the demo/ package, so the whole console demo is one self-contained file. Add a "Putting It All Together" section to the tutorial that shows the complete program. Verified end-to-end: bootstrap, the Hello World filter, and the count aggregation all react as before. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

`npm run serve` used a bare `hugo server`, which keeps the config's `/drasi-python/` base path, so http://localhost:1313/ 404s and it collides with other local servers. Pass `--baseURL http://localhost:1313/ --appendPort=false` so local dev serves at the root. Production (`npm run build`, used by CI) is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge config.py, queries.py and engine.py into one engine.py (the two source configs, the synthetic join, the six queries, and CurbsideEngine), and drop the demo/ package; app.py imports CurbsideEngine and the query ids from it. Add a "Putting It All Together" section to the tutorial showing the assembled engine. Verified end-to-end against real PostgreSQL and MySQL: bootstrap, the cross-DB delivery join, the drasi.trueFor delay, and reset all work. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Drop the standalone PICKUP_BY constant and declare the join dict directly in each query's joins list, so the delivery and delay queries are self-contained. Update the docs (The Synthetic Join and Putting It All Together) to match and regenerate the README. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Register each of the three continuous queries with its own add_query call, inline where its explanatory comment lives, rather than storing them in a QUERIES dict and looping. Update the docs (The Python Reaction and Putting It All Together) to match and regenerate the README. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge demo/config.py, demo/queries.py and demo/engine.py into one engine.py at the tutorial root (config, the six queries, and ComfortEngine), mirroring the curbside layout. app.py now imports everything from engine. Add a Putting It All Together section to the docs showing the whole engine.py and regenerate the README. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The building-comfort tutorial linked to https://drasi.io/reference/sources/postgresql/, which 404s. Point it at the correct drasi-server source configuration page and regenerate the README. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…mmar - setup-database scripts: track whether Postgres/MySQL actually became ready and exit with a clear error instead of proceeding to seeding (all three .sh and the curbside/getting-started .ps1; building-comfort .ps1 already did). - building-comfort engine.py: use asyncio.get_running_loop() over get_event_loop(), and asyncio.create_task() over ensure_future(). - README.md: fix 'They also render' to refer to the tutorials collectively. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

danielgerlag and others added 6 commits July 30, 2026 16:19

Give each Curbside panel its own bordered box

837124b

Wrap the six panels in st.container(border=True) so each order/vehicle/matched/ delayed box is visually distinct, and refresh the screenshot. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 31, 2026 18:51

Copilot started reviewing on behalf of danielgerlag July 31, 2026 18:52 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

danielgerlag and others added 13 commits July 31, 2026 13:11

Add getting-started devcontainer lock file

9f228da

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

danielgerlag marked this pull request as ready for review July 31, 2026 23:09

danielgerlag merged commit 5312b3d into main Jul 31, 2026
24 of 25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Building Comfort and Curbside Pickup tutorials (Python + Streamlit) - #1

Add Building Comfort and Curbside Pickup tutorials (Python + Streamlit)#1
danielgerlag merged 19 commits into
mainfrom
danielgerlag-building-comfort-python-tutorial

danielgerlag commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

danielgerlag Jul 31, 2026

Uh oh!

danielgerlag Jul 31, 2026

Uh oh!

danielgerlag Jul 31, 2026

Uh oh!

danielgerlag Jul 31, 2026

Uh oh!

danielgerlag Jul 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

danielgerlag commented Jul 31, 2026

Why

What's here

Approach and shared design

Notes for reviewers

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

danielgerlag Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

danielgerlag Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

danielgerlag Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

danielgerlag Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

danielgerlag Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants