Add cross-org usage dashboard to Grafana - #2476
Conversation
![peqy[bot]](https://avatars.githubusercontent.com/in/1930020?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overall this is a solid approach: using the DB as the source of truth avoids the cardinality / multi-replica problems of in-process metrics, and the env-agnostic datasource variable in the dashboard makes the base placement make sense.
Main things to consider are (1) using org ids (not display names) for template variable values to avoid collisions/quoting issues, and (2) forcing UTC when computing month boundaries in SQL so “current month” is stable regardless of DB session timezone.
Found 3 issues (0 critical, 2 improvements, 1 nit).
| "label": "Organization", | ||
| "type": "query", | ||
| "datasource": { "type": "postgres", "uid": "${datasource}" }, | ||
| "query": "SELECT COALESCE(display_name, name) FROM public.organization WHERE deleted_at IS NULL ORDER BY 1", |
There was a problem hiding this comment.
Improvement Org variable uses COALESCE(display_name, name) as the value.
If two orgs share a display name/name (or a name contains quotes), series can merge and the IN ($org) filter can get tricky.
Fix: return ids + labels (Grafana convention
SELECT id AS __value, COALESCE(display_name,name) AS __text ...), and filter/group byo.id(e.g.o.id IN (${org:csv})) while still displaying the name in the legend.
There was a problem hiding this comment.
One extra reason to switch the org variable/value to ids: several panels also GROUP BY / filter on COALESCE(o.display_name, o.name), so two orgs sharing a display_name would get merged into one series/row.
Using SELECT id AS __value, COALESCE(display_name,name) AS __text ... and grouping/filtering by o.id keeps the labels nice while guaranteeing correctness.
There was a problem hiding this comment.
Fixed now: org templating query returns id AS __value + __text, so filters/grouping can safely key on o.id without name collisions.
| "editorMode": "code", | ||
| "rawQuery": true, | ||
| "format": "table", | ||
| "rawSql": "SELECT\n COALESCE(SUM(u.used) FILTER (WHERE u.resource = 'test_executions'), 0) AS \"Test executions\",\n COALESCE(SUM(u.used) FILTER (WHERE u.resource = 'tracing_spans'), 0) AS \"Tracing spans\",\n COALESCE(SUM(u.used) FILTER (WHERE u.resource = 'test_generation'), 0) AS \"Tests generated\",\n COALESCE(SUM(u.used) FILTER (WHERE u.resource = 'model_tokens'), 0) AS \"Model tokens\"\nFROM public.usage u\nJOIN public.organization o ON o.id = u.organization_id\nWHERE u.deleted_at IS NULL\n AND o.deleted_at IS NULL\n AND u.period_start = date_trunc('month', now())::date" |
There was a problem hiding this comment.
Improvement Month boundaries use date_trunc('month', now()) / now().
The dashboard pins Grafana timezone to UTC, but the SQL month calculation uses the DB session timezone (which isn’t configured to UTC in stg/prd CNPG manifests). That can shift “this month” around UTC midnight.
Fix: compute bounds in UTC in SQL (e.g.
date_trunc('month', now() AT TIME ZONE 'UTC')::dateand similarly for prev month), or ensure the datasource/session setsTimeZone=UTC.
There was a problem hiding this comment.
Good catch — this looks fixed now with date_trunc('month', now() AT TIME ZONE 'UTC') / UTC bounds.
One remaining timezone-dependent spot: the trend panels use u.period_start::timestamptz for the time column (and $__timeFilter(u.period_start) on a date). If the DB session TimeZone isn’t UTC, the rendered timestamps can shift across UTC day/month boundaries. Consider anchoring those as UTC too, e.g. u.period_start::timestamp AT TIME ZONE 'UTC' (and apply the macro to the same expression).
There was a problem hiding this comment.
Fixed in fcc6b09/a7b8b02: month bounds are now computed with now() AT TIME ZONE 'UTC', so “this month” won’t depend on the DB session TimeZone.
There was a problem hiding this comment.
Looks fixed now: trend panels anchor the plotted time via u.period_start::timestamp AT TIME ZONE 'UTC'.
One small follow-up: the queries still use AND $__timeFilter(u.period_start) (a date). To avoid reintroducing session-TimeZone casts at the range boundaries, it’d be safer to apply the macro to the same UTC expression, e.g. AND $__timeFilter(u.period_start::timestamp AT TIME ZONE 'UTC').
| matchLabels: | ||
| dashboards: grafana | ||
| datasource: | ||
| name: "Rhesis DEV Postgres (RW)" |
There was a problem hiding this comment.
Nit Datasource name says (RW) but the role is intended to be read-only.
Fix: rename to something like
Rhesis DEV Postgres (RO)to avoid confusion when selecting datasources.
There was a problem hiding this comment.
👍 Confirmed: datasource name is now Rhesis DEV Postgres (RO), which matches the intended read-only role.
There was a problem hiding this comment.
Confirmed fixed: datasource is now labeled (RO).
Usage accrues per org into the `usage` table, but the only view of it is the in-app Usage page, which shows one org its own numbers. This reads across every tenant: platform totals, per-org breakdown, top consumers, and a 12-month trend. SQL rather than Prometheus, because current usage is database state, not process state. The backend runs several replicas, so gauges updated on increment would each hold a different subset of it, and there is no Celery beat process to resync them. Grafana already has a read-only Postgres datasource, so the data is one query away. Lives in base/ even though each env names its datasource differently, because the datasource is chosen by a template variable. That variable queries type "postgres", which is what all three GrafanaDatasource CRs declare, so it matches whether or not the running Grafana resolves the newer grafana-postgresql-datasource plugin alias. Every panel joins organization and filters deleted_at on both sides. Without the join, a deleted org's live rows inflate the platform total and make it disagree with the per-org table directly below it. Limits are absent on purpose: they live in the EE tier config and the org's signed license, neither of which is readable from SQL. seats, projects and endpoints are absent too, being live counts with no history to chart. Both are called out in a panel on the dashboard. Signed-off-by: Harry Cruz <harry@rhesis.ai>
The stg and prd datasources landed in #2348, but dev never got one, so the shared dashboard in base/grafana-resources has nothing to query there, which is the environment you would want to check it in first. Dev has no CNPG, so the role cannot come from managed.roles. It is created by a Bitnami initdb script instead, mirroring how rhesis-user and rhesis-analytics-user are already created there. BYPASSRLS is required, not convenience: `usage` is FORCE ROW LEVEL SECURITY and Grafana never sets app.current_organization, so without it every cross-org query returns zero rows. Verified against a scratch Postgres, where an identically granted role without BYPASSRLS sees zero rows while this one reads every tenant and is still refused writes. GRAFANA_VIEWER_PASSWORD was the one env missing from gsm-secrets.json.example, and the name matches the ExternalSecret's secretKey so gsm-secrets-sync.sh can resolve it. That makes the GSM secret and its ESO IAM binding a sync run rather than a manual step. initdb only runs on a fresh PVC, so the already-initialized dev cluster still needs the CREATE ROLE and GRANT block run once by hand. Same caveat the stg and prd manifests already carry for their own grants. Signed-off-by: Harry Cruz <harry@rhesis.ai>
3b407e6 to
812534b
Compare
Three fixes from review on #2476. date_trunc('month', now()) truncates in the DB session TimeZone, which no CNPG manifest sets, so it was relying on the Postgres image happening to default to Etc/UTC. Under a non-UTC session the month boundary moves: at 00:30 UTC on the 1st with TimeZone=America/Los_Angeles it reports the previous month. Periods are UTC-anchored calendar months, so the SQL now says so explicitly and matches services/usage.py:_current_period. The $org filter used bare interpolation, which quotes each value without escaping quotes inside it, so an organization named like O'Brien Corp turned panel 5 into a syntax error. ${org:sqlstring} escapes properly. Renamed the dev datasource from (RW) to (RO). In stg and prd the suffix says which CNPG service is targeted, but dev has no CNPG and so no -rw/-ro services, leaving the role's SELECT-only grant as the only thing the suffix can usefully describe there. Signed-off-by: Harry Cruz <harry@rhesis.ai>
There was a problem hiding this comment.
Month-boundary fix to UTC looks good.
Improvement: consider scoping grafana-viewer privileges tighter than SELECT ON ALL TABLES (esp. combined with BYPASSRLS).
Improvement: in trend panels, u.period_start::timestamptz can still shift timestamps by DB session timezone—anchor as UTC.
Improvement: use org ids for the org template var + grouping/filtering to avoid merging orgs with the same display name.
Found 3 issues (0 critical, 3 improvements).
| --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<'EOSQL' | ||
| GRANT CONNECT ON DATABASE :"postgres_db" TO "grafana-viewer"; | ||
| GRANT USAGE ON SCHEMA public TO "grafana-viewer"; | ||
| GRANT SELECT ON ALL TABLES IN SCHEMA public TO "grafana-viewer"; |
There was a problem hiding this comment.
Improvement grafana-viewer gets GRANT SELECT ON ALL TABLES IN SCHEMA public + default privileges for future tables.
That’s broader than this dashboard needs (usage + organization) and, combined with BYPASSRLS, means a Grafana compromise can read any table in public across all tenants.
Fix: if feasible, scope privileges to the minimal set (e.g.
GRANT SELECT ON TABLE public.usage, public.organization TO "grafana-viewer";and avoid granting default privileges broadly), or document explicitly why full-schema read is required.
There was a problem hiding this comment.
Confirmed this matches the existing stg/prd pattern (CNPG postInitApplicationSQL also grants SELECT ON ALL TABLES IN SCHEMA public + default privileges).
Still worth calling out as a conscious tradeoff: BYPASSRLS + full-schema SELECT means a Grafana compromise can read any tenant data in public. If scoping isn’t feasible operationally (because usage is created later), maybe add an explicit comment in the script explaining why the broader grant is intentional.
| }, | ||
| { | ||
| "id": 5, | ||
| "type": "timeseries", |
There was a problem hiding this comment.
Improvement Trends panel casts u.period_start::timestamptz.
period_start is a date; casting to timestamptz interprets midnight in the DB session timezone, which can shift the plotted timestamp away from the intended UTC month boundary.
Fix: anchor as UTC, e.g.
u.period_start::timestamp AT TIME ZONE 'UTC' AS "time"(and apply$__timeFilterto the same expression).
There was a problem hiding this comment.
Fixed: period_start is now anchored as UTC (::timestamp AT TIME ZONE 'UTC').
Same minor note as above: consider changing AND $__timeFilter(u.period_start) to AND $__timeFilter(u.period_start::timestamp AT TIME ZONE 'UTC') so the filter uses the same timezone/type as the plotted time.
| "fieldConfig": { | ||
| "defaults": { "unit": "short", "decimals": 0, "custom": { "align": "auto", "filterable": true } }, | ||
| "overrides": [ | ||
| { "matcher": { "id": "byName", "options": "Organization" }, "properties": [{ "id": "custom.width", "value": 260 }] }, |
There was a problem hiding this comment.
Improvement Org filter/groups use COALESCE(o.display_name, o.name).
If two orgs share a display name (or a name changes), series/rows can merge and filtering becomes ambiguous.
Fix: switch the org template var to ids (
SELECT id AS __value, COALESCE(display_name,name) AS __text ...) and filter/group byo.id(while still using the name only for display/legend).
There was a problem hiding this comment.
This is addressed now that the org variable is ID-valued and the panels group by o.id / filter on o.id (vs grouping/filtering by COALESCE(...)).
Two more from review, both the same shape as the fixes in the previous commit: an implicit dependency on the DB session TimeZone, and a value that is not as unique as it looks. period_start is a date, and casting a date to timestamptz reads midnight in the session TimeZone. East of UTC that moves the point into the previous month: under Asia/Tokyo the August bucket plotted at 2026-07-31 15:00 UTC. Casting to timestamp and attaching UTC puts it back on the month boundary it belongs to. The org variable used COALESCE(display_name, name) as its value, and four panels grouped or filtered on the same expression. organization.name has no unique constraint and display_name is free text, so two orgs sharing a display name were silently summed into a single row. The variable now returns ids with names as labels, and grouping is on o.id. Verified with two orgs deliberately sharing a display name: they stay separate rows of 11 and 22 rather than merging into 33. One limit worth knowing: Grafana keys time-series by the metric value, so same-named orgs still share a legend entry in the trend panel. The tables and the filter are correct; only that legend is ambiguous, and making it unique would mean appending an id fragment to every label. Signed-off-by: Harry Cruz <harry@rhesis.ai>
|
Most of the earlier issues are addressed now (org variable keyed on Remaining small improvement: in the trend queries you anchor The |
The plotted time column was anchored to UTC in the previous commit but the range filter still compared the bare `date` column, and Postgres converts a date to timestamptz using the session TimeZone. So the two disagreed at the edges: with a range starting at 2026-08-01T00:00Z under Asia/Tokyo, the August bucket converts to 2026-07-31T15:00Z, falls before the range start, and drops off the chart entirely. I had left this on the grounds that wrapping the column costs an index scan. That was wrong: `usage` has no index on `period_start` alone, only the (organization_id, resource, period_start) unique constraint where it is the third column and unusable for a standalone range scan. There was no scan to preserve. Also spelled out why grafana-viewer gets full-schema read rather than a grant on `usage` and `organization`: default privileges cannot name tables that do not exist yet, and at initdb time none of them do. #2479 carries the question of whether that posture is right across all three envs. Signed-off-by: Harry Cruz <harry@rhesis.ai>
There was a problem hiding this comment.
Most earlier issues are addressed now (org variable keyed on id, month bounds + trend timestamps anchored to UTC, and $__timeFilter applied to the same UTC expression). The dev datasource/secret wiring and initdb role script look consistent with the existing stg/prd pattern.
Ship it.
1 participant
Purpose
Usage accrues per org into the
usagetable, but the only way to look at it is the in-app Usage page, which shows one org its own numbers. There is no way for us to see consumption across every tenant: who is heaviest, what the platform total is, whether a month looks abnormal. This adds that view to the Grafana we already run, on top of the read-only Postgres datasource from #2348.This is the Observability slice of the usage-limits work, redesigned. The original plan called for
prometheus_client, a/metricsendpoint, in-process gauges updated on increment, and a Celery beat task to resync them after restarts. That does not fit this codebase, for four independent reasons:/metricsis already taken byrouters/metric.py; there is no Celery beat process anywhere, so one would mean a new container and deployment; the Celery worker never callsbootstrap_ee, so a beat task would resolve community limits for every org rather than their real ones; and the backend runs several replicas, so gauges updated on increment would each hold a different subset of the state and Prometheus would scrape disagreeing values. Alertmanager is also disabled, so the planned PrometheusRule thresholds would have fired into nothing.Current usage is database state, not process state. Reading it with SQL is both simpler and more accurate, needs no new dependency and no backend code, and sidesteps the unbounded per-organization label cardinality the metrics approach would have introduced.
What Changed
kubernetes/base/grafana-resources/usage-dashboard.yaml(new) — the repo's firstGrafanaDashboard. Six panels plus a notes panel: platform totals and a per-org breakdown for the current billing period, top consumers, a 12-month trend by resource and by org, and a month-over-month table.kubernetes/clusters/dev/grafana-resources/(new datasource + credentials) — Feat/Grafana Dashboard For Rhesis #2348 covered stg and prd only, so dev had nothing for the dashboard to query.charts/rhesis/values-dev.yaml— aninit-grafana-viewer.shinitdb script creating thegrafana-viewerrole. Dev has no CNPG, so it cannot come frommanaged.roles; this mirrors howrhesis-userandrhesis-analytics-userare already created there.infrastructure/config/gsm-secrets.json.example— added thedeventry forGRAFANA_VIEWER_PASSWORD, which stg and prd already had.No backend, frontend, SDK or EE changes. No migration, no new dependency.
Two design notes worth reviewing:
The dashboard lives in
base/even though each env names its datasource differently, because the datasource is chosen by a template variable rather than hardcoded. That variable queries typepostgres, which is what all threeGrafanaDatasourceCRs declare, so it resolves whether or not the running Grafana maps the newergrafana-postgresql-datasourceplugin alias.Every panel joins
organizationand filtersdeleted_aton both sides. Without the join, a deleted org's live usage rows inflate the platform total and make it disagree with the per-org table directly beneath it. My fixture reproduced exactly that:tracing_spansread 19777 instead of 12000.Additional Context
bypassrls: trueongrafana-viewer, without which every cross-org query returns zero rows under theusagetable'sFORCE ROW LEVEL SECURITY).BYPASSRLSon the dev role is required for that same reason, not for convenience.seats,projectsandendpointsare absent too. They are live counts off theuser,projectandendpointtables rather than accrued intousage, so they have no history to chart.terraform/infrastructure/is GCP-only, with nopostgresqlprovider and nopostgresql_role/postgresql_grantresources anywhere, so this follows the existing manifest-and-initdb pattern rather than pioneering a new home for one role.grafana-viewer-credentials.yamlusesecretKey: password, sogsm-secrets-sync.shlooks up.stg.password, which does not exist in the JSON. TheirGRAFANA_VIEWER_PASSWORDentries are unreachable by that script as written, and were presumably set by hand. The dev wiring added here uses a matching name so the sync resolves it. Left the deployed ones alone.Testing
Three prerequisites before any panel shows data. These are environment state, not code:
./infrastructure/config/gsm-secrets-sync.shfor dev once the real password value is filled in. That createsdev-grafana-viewer-passwordand binds ESO access.GRANTrun.bypassrls: truereconciles onto a running cluster viamanaged.roles, but theGRANTlives inpostInitApplicationSQL, which only fires on a fresh initdb. Check withSELECT has_table_privilege('grafana-viewer','public.usage','SELECT'), rolbypassrls FROM pg_roles WHERE rolname = 'grafana-viewer';— both must be true, otherwise panels fail withpermission denied for table usage.CREATE ROLEandGRANTblock run once by hand, same fresh-PVC caveat.What I verified locally:
COALESCE(display_name, name)falling back correctly for an org with no display name, month-over-month deltas correct at +200 and -280.FORCE ROW LEVEL SECURITYand the realtenant_isolationpolicy:grafana-viewerreads across all orgs, writes are refused withpermission denied, and as a control an identically granted role withoutBYPASSRLSsees zero rows. That control is what justifies theBYPASSRLSline rather than assuming it.kubectl kustomize kubernetes/clusters/{dev,stg,prd}/grafana-resourcesall render the dashboard alongside their own datasource, and the embeddedspec.jsonre-parses from each rendered output. Note the render path is thegrafana-resourcesoverlay, not the cluster root, which only renders the wrapper Argo Applications.helm templatewith dev values, confirming the init script and theGRAFANA_VIEWER_PASSWORDenv var.To check after it syncs to dev: the datasource passes Grafana's "Save & test", the dashboard appears in the Rhesis folder, the current-period table lists more than one org, and the 12-month trend is non-empty (the
91607f0dd412and9550c62e80a5backfills already populated trailing months). Then cross-check one org's current-period figure against that org's own Usage page underorganizations/usage. Both read the same rows, so a mismatch means a panel query is wrong.