docs: add homepage hero slogan A/B test with PostHog tracking. #3161
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| "use client"; | ||
|
|
||
| import { useEffect, useState } from "react"; | ||
| import { usePostHog } from "posthog-js/react"; | ||
| import { Heading } from "@/components/ui/heading"; | ||
| import { TextHighlight } from "@/components/ui"; | ||
| import { cn } from "@/lib/utils"; | ||
| import { usePostHogClientCapture } from "@/src/usePostHogClientCapture"; | ||
|
|
||
| const STORAGE_KEY = "langfuse-hero-slogan-variant"; | ||
|
|
||
| type HeroSloganVariant = "ai-engineering" | "agent-evals"; | ||
|
|
||
| function pickVariant(): HeroSloganVariant { | ||
| return Math.random() < 0.5 ? "ai-engineering" : "agent-evals"; | ||
| } | ||
|
|
||
| function readStoredVariant(): HeroSloganVariant | null { | ||
| try { | ||
| const stored = localStorage.getItem(STORAGE_KEY); | ||
| if (stored === "ai-engineering" || stored === "agent-evals") { | ||
| return stored; | ||
| } | ||
| } catch { | ||
| // localStorage unavailable (e.g. private browsing restrictions) | ||
| } | ||
| return null; | ||
| } | ||
|
|
||
| function persistVariant(variant: HeroSloganVariant) { | ||
| try { | ||
| localStorage.setItem(STORAGE_KEY, variant); | ||
| } catch { | ||
| // ignore write failures | ||
| } | ||
| } | ||
|
|
||
| const headingClassName = cn( | ||
| "flex-col items-center gap-0.5 sm:gap-1 md:gap-1.5 text-center font-medium leading-[105%] max-md:max-w-[500px]", | ||
| "[leading-trim:both] [text-edge:cap]", | ||
| ); | ||
|
|
||
| function AiEngineeringSlogan() { | ||
| return ( | ||
| <> | ||
| <TextHighlight | ||
| highlightClassName="mix-blend-multiply" | ||
| className="whitespace-nowrap" | ||
| > | ||
| Open Source<span className="inline max-[499px]:hidden"> </span> | ||
| </TextHighlight> | ||
| <span className="flex min-[500px]:inline"> | ||
| <TextHighlight | ||
| highlightClassName="mix-blend-multiply" | ||
| className="max-[499px]:pr-1.75" | ||
| > | ||
| AI | ||
| </TextHighlight> | ||
| <TextHighlight | ||
| highlightClassName="mix-blend-multiply" | ||
| className="min-[500px]:pr-2" | ||
| > | ||
| Engineering | ||
| </TextHighlight> | ||
| </span> | ||
| <TextHighlight highlightClassName="mix-blend-multiply"> | ||
| Platform | ||
| </TextHighlight> | ||
| </> | ||
| ); | ||
| } | ||
|
|
||
| function AgentEvalsSlogan() { | ||
| const wordGap = "gap-x-1.5 sm:gap-x-2"; | ||
| const phraseGap = "gap-x-3 sm:gap-x-4"; | ||
|
|
||
| return ( | ||
| <> | ||
| <TextHighlight | ||
| highlightClassName="mix-blend-multiply" | ||
| className="whitespace-nowrap" | ||
| > | ||
| Open Source | ||
| </TextHighlight> | ||
| <span | ||
| className={cn( | ||
| "flex flex-wrap justify-center items-center", | ||
| phraseGap, | ||
| )} | ||
| > | ||
| <span className={cn("inline-flex items-center", wordGap)}> | ||
| <TextHighlight highlightClassName="mix-blend-multiply"> | ||
| Agent | ||
| </TextHighlight> | ||
| <TextHighlight highlightClassName="mix-blend-multiply"> | ||
| Evals | ||
| </TextHighlight> | ||
| </span> | ||
| <span className={cn("inline-flex items-center", wordGap)}> | ||
| <TextHighlight highlightClassName="mix-blend-multiply"> | ||
| and | ||
| </TextHighlight> | ||
| <TextHighlight highlightClassName="mix-blend-multiply"> | ||
| Tracing | ||
| </TextHighlight> | ||
| </span> | ||
| </span> | ||
| </> | ||
| ); | ||
| } | ||
|
|
||
| export function HeroSlogan() { | ||
| const posthog = usePostHog(); | ||
| const capture = usePostHogClientCapture(); | ||
| const [variant, setVariant] = useState<HeroSloganVariant>("ai-engineering"); | ||
|
|
||
| useEffect(() => { | ||
| const stored = readStoredVariant(); | ||
| const nextVariant = stored ?? pickVariant(); | ||
| const isNewAssignment = !stored; | ||
|
|
||
| if (!stored) { | ||
| persistVariant(nextVariant); | ||
| } | ||
|
|
||
|
Contributor
There was a problem hiding this comment.
Prompt To Fix With AIThis is a comment left during a code review.
Path: components/home/HeroSlogan.tsx
Line: 125
Comment:
**`posthog.register` may fire before PostHog has finished initialising**
`usePostHog()` returns the PostHog instance synchronously, but the underlying SDK's async bootstrap (feature-flag fetch, session init) may not have completed when this effect runs immediately after mount. Calling `posthog.register()` at this point can silently succeed but the super-property may not propagate to events captured in the same tick. A safer approach is to gate the registration on `posthog.isFeatureEnabled` or listen to the `onFeatureFlags` callback, ensuring the SDK is ready before registering super-properties.
How can I resolve this? If you propose a fix, please make it concise. |
||
| setVariant(nextVariant); | ||
|
|
||
| posthog.register({ hero_slogan_variant: nextVariant }); | ||
| capture("hero_slogan_exposure", { | ||
| variant: nextVariant, | ||
| is_new_assignment: isNewAssignment, | ||
| }); | ||
|
Contributor
There was a problem hiding this comment.
The component hard-codes A lazy state initialiser avoids the re-render on the client side (localStorage access is already guard-wrapped for SSR): Prompt To Fix With AIThis is a comment left during a code review.
Path: components/home/HeroSlogan.tsx
Line: 112-132
Comment:
**Visible heading flash for `agent-evals` users**
The component hard-codes `"ai-engineering"` as the initial state, so every server render and first client paint shows the control variant. The `useEffect` then reads localStorage and may switch to `"agent-evals"`, causing a visible flash of the wrong heading for roughly half of new visitors. For returning users assigned to `"agent-evals"` the flash happens on every page load.
A lazy state initialiser avoids the re-render on the client side (localStorage access is already guard-wrapped for SSR): `useState<HeroSloganVariant>(() => readStoredVariant() ?? pickVariant())`. You'd also want to move `persistVariant` into the same initialiser block so the variant is stored before any render.
How can I resolve this? If you propose a fix, please make it concise. |
||
| // Track once when the hero mounts on the homepage. | ||
| // eslint-disable-next-line react-hooks/exhaustive-deps | ||
| }, []); | ||
|
claude[bot] marked this conversation as resolved.
Outdated
|
||
|
|
||
| return ( | ||
| <Heading as="h1" size="big" className={headingClassName}> | ||
| {variant === "ai-engineering" ? ( | ||
| <AiEngineeringSlogan /> | ||
| ) : ( | ||
| <AgentEvalsSlogan /> | ||
| )} | ||
| </Heading> | ||
| ); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PostHog's experiments best practices require using
getFeatureFlag()or the React equivalentuseFeatureFlagVariantKey()for experiment assignment. Without this, PostHog does not record the built-in exposure event tied to an experiment flag, so those users are excluded from PostHog's experiment results and statistical significance calculations. Thehero_slogan_exposurecustom event captures the exposure for raw analysis, but the stated goal of "conversion analysis" in PostHog's Experiments UI won't work with this approach.Consider migrating variant assignment to a PostHog feature flag with two variants — this provides cohort consistency, proper exposure tracking, and the Experiments results dashboard out of the box. See PostHog Experiments best practices.
Prompt To Fix With AI
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!