Skip to content

feat(analytics): report to collect.mindshub.ai and send an identifying agent (ENG-1355) - #611

Open
lucas-koontz wants to merge 1 commit into
stagingfrom
feat/eng-1355-repoint-analytics-collector
Open

feat(analytics): report to collect.mindshub.ai and send an identifying agent (ENG-1355)#611
lucas-koontz wants to merge 1 commit into
stagingfrom
feat/eng-1355-repoint-analytics-collector

Conversation

@lucas-koontz

Copy link
Copy Markdown
Contributor

User story

As a whoever is asked how many people finish the installer, accept terms, or pick BYOK
I want the desktop app's four install events to actually arrive in PostHog
So that the onboarding funnel can be answered from data rather than guessed at

Why this matters

None of the four ANTONAPP_* events has ever reached PostHog. The collector they were pointed at relayed an event only when its action name started with a lowercase anton_ or ds_connect_, and that match is case sensitive. These names are uppercase, so every one of them was discarded. The endpoint answered HTTP 200 while doing it, and sendEvent reads no response, so nothing anywhere reported it.

This was found while fixing the same defect for anton's per-turn cost event, which is the ticket's original subject. The blast radius turned out to include the install funnel.

What happens today

flowchart LR
  I["installer.ts:563<br/>ANTONAPP_INSTALLATION_SUCCESS"] --> S["sendEvent"]
  X["index.ts:1146-1151<br/>TERMS_ACCEPTED, MINDSLLM, BYOK"] --> S
  S --> L{"zoomInfoCollector"}
  L -->|"action starts with lowercase<br/>anton_ or ds_connect_"| P["PostHog 355390"]
  L -->|"ANTONAPP_* is uppercase,<br/>so it never matches"| D["discarded"]
  D --> R["HTTP 200"]
  R --> Q["sendEvent discards the response.<br/>Four events, zero rows, no signal."]
Loading

What should happen

flowchart LR
  I["installer.ts:563<br/>ANTONAPP_INSTALLATION_SUCCESS"] --> S["sendEvent, User-Agent set"]
  X["index.ts:1146-1151<br/>TERMS_ACCEPTED, MINDSLLM, BYOK"] --> S
  S --> L{"collect.mindshub.ai"}
  L -->|"no action filter,<br/>properties minus a denylist"| P["PostHog 355390"]
  L --> M["EventsRelayed counter<br/>plus a Slack alarm"]
Loading

What changed

File Change
src/main/analytics.ts ANALYTICS_URL points at https://collect.mindshub.ai/collect; the request sends User-Agent: cowork-analytics/1.0.
src/main/analytics.test.ts New. This file had no tests. Four: the host, the user agent, action plus extra-property merging, and that a failed request never throws.
README.md New "Anonymous install analytics" section.

The README section exists because these two paths get conflated. The renderer has posthog-js keyed on the signed-in Keycloak user and configured by VITE_POSTHOG_MINDSHUB_MAIN_PROJECT_TOKEN. The main process has this beacon, which identifies nothing and is configured by a constant. Anyone debugging "why is this event missing" needs to know which one they are looking at.

One gap recorded, not fixed here. sendEvent sends no installation fingerprint, unlike anton's send_event, which sends aid. So these four events cannot be attributed to a machine. The collector groups them under one synthetic id and stamps aid_missing on them so the gap is queryable rather than invisible. Sending an install id is a small follow-up and is deliberately outside this change.

Rollout, which is different here

src/main/** has no OTA path, so this constant reaches users only when they download a new installer. That is worth saying out loud because it means the fix lands slowly. It costs nothing, though: these events are already dark for every existing install, so nothing regresses while the change propagates, and every new installer from here reports correctly.

Acceptance criteria

  • A fresh install driven to completion produces one ANTONAPP_INSTALLATION_SUCCESS row in project 355390 within 60 seconds.
  • Accepting terms produces ANTONAPP_TERMS_ACCEPTED; each provider choice produces ANTONAPP_MINDSLLM or ANTONAPP_BYOK.
  • Those rows carry aid_missing set, and a distinct_id that is the collector's synthetic id rather than a real fingerprint.
  • Still true afterwards: sendEvent throws nothing when the endpoint is unreachable or refuses the request.
  • Still true afterwards: no event carries conversation content, a file path, or an email address.

How to test

  1. Merge order matters. mindsdb/mindshub_services#186 and mindsdb/terraform#162 must be deployed first. Confirm with curl -s "https://collect.mindshub.ai/collect?action=ANTONAPP_TERMS_ACCEPTED&aid=qaprobe1355" returning relayed: true. That call alone proves the case-sensitivity fix, since the same request against the old endpoint returns posthogResult: false.
  2. npm run pack, clear install state per the README's fresh-install reset (anton-reset --deep), and run the app through first-run.
  3. Accept terms, then pick MindsLLM or BYOK.
  4. Wait 60 seconds, then in project 355390: select event, properties.aid_missing from events where event like 'ANTONAPP%' order by timestamp desc limit 10. Expect one row per step above.
  5. Confirm the failure path: point ANALYTICS_URL at an unroutable host in a local build and confirm first-run completes with no error surfaced.

Verified locally

Check Result
npm test 1230 passed, 119 files
npm run typecheck clean (main, renderer, test)

Both re-run after rebasing onto current staging, which needed an npm install for a dependency added in the meantime.

Ships with

Refs ENG-1355

…g agent

The four ANTONAPP_* install events have never reached PostHog. The collector they
were pointed at matched a lowercase anton_ prefix on the action name, the match is
case sensitive, and these names are uppercase, so every one of them was discarded
behind an HTTP 200 that sendEvent never reads. Repointing at the new collector,
which applies no action filter, is what makes them arrive.

The request now carries an explicit user agent as well: Cloudflare's bot
protection answers script-shaped agents with 403 on the mindshub.ai zone, and this
function throws its response away, so a blocked event leaves no trace.

src/main/** has no OTA path, so this reaches users only through a new installer
download. Since these events are already dark, nothing regresses in the meantime.

Adds this file's first tests, and a README section separating the main-process
install beacon from the renderer's product analytics, including the missing
installation id that leaves those events unattributable.

Lucas Koontz
ENG-1355 - Analytics collector lambda silently drops turn_completed and its properties

Refs: ENG-1355
@github-actions

Copy link
Copy Markdown

No PR environment for this pull request

Add the deploy label and push to create one. It is torn down when the label is removed or the PR closes, so any URL you saw here earlier is gone.

Updated on every push to this PR.

@entelligence-ai-pr-reviews

Copy link
Copy Markdown

EntelligenceAI PR Summary

Updated main-process install analytics to send ANTONAPP_* events to the maintainable collect.mindshub.ai/collect hostname instead of a raw API Gateway URL, allowing the collector to change without another app release. Requests now include the identifying cowork-analytics/1.0 User-Agent while preserving event properties and fire-and-forget error handling. Added regression tests for the endpoint, headers, query parameters, timestamps, and network failures, plus README documentation explaining the analytics path and its limitations.

flowchart TD
    classDef newBehavior fill:#dcfce7,stroke:#16a34a,color:#14532d;
    App["Desktop app"] --> Analytics["sendEvent"]
    Analytics --> Request["HTTPS GET with event data and User-Agent"]
    Request --> Collector["collect.mindshub.ai/collect"]
    Collector --> PostHog["PostHog install events"]
    class Analytics,Request,Collector newBehavior;
Loading

🟢 Green = new or changed in this PR


Review Scorecard

Dimension Rating Basis
Code Quality ●●●●● 5/5 — Excellent no critical/significant findings — rated Excellent
Blast Radius Low changed symbols are referenced only within their own file(s); no high-impact surface touched, 2 file(s) / ~35 line(s) changed (size only — not a blast signal)
Merge Confidence ●●●●● 5/5 — Safe to Merge code quality 5/5 × Low blast radius

Safe to merge — this is a clean, low-blast-radius analytics change. The install reporting now targets the maintainable collect.mindshub.ai/collect endpoint, adds the explicit cowork-analytics/1.0 User-Agent, preserves event data, and retains fire-and-forget network error handling. Regression coverage addresses the endpoint, headers, query parameters, timestamps, and network failures, with no introduced correctness or robustness issues and no unresolved pre-existing defects identified.

Key Findings:

  • The analytics request construction preserves existing event properties while updating the collector URL and adding the identifying User-Agent.
  • The regression tests cover the changed request contract and failure behavior, including endpoint selection, headers, query parameters, timestamps, and rejected network requests.
Evaluated against
  • 3/3 changed files reviewed
  • criteria: correctness, security & access control, robustness & error handling, concurrency & data integrity, repo conventions / steering docs
  • steering docs: CLAUDE.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant