[docs-infra] Extend Code Highlighting Syntax

[dav-is]

Summary

Adds extendSyntaxTokens, a post-processor that enriches starry-night HAST output with fine-grained di-* CSS classes for styling individual token categories (numbers, booleans, JSX components, CSS properties, etc.) without fighting starry-night's coarse pl-* classes.

All di-* classes are added additively — existing pl-* classes are never removed — so the base theme still works and consumers can progressively override specific tokens.

Preview Deploy
Downstream Base UI PR

New di-* classes

Class	Meaning	Scope
di-num	Numeric literal	All
di-bool	Boolean literal	All
di-n	Nullish value (null, undefined, "")	All
di-this	this / super	JS/TS
di-bt	Built-in type keyword (string, number, etc.)	TS only
di-jsx	JSX component name	JSX only
di-ht	HTML tag wrapper (brackets + tag name)	Inline code
di-jt	JSX tag wrapper (brackets + tag name)	Inline code
di-ak	HTML/JSX attribute key	HTML/JSX
di-ae	Attribute equals sign	HTML/JSX
di-av	Attribute value	HTML/JSX
di-da	CSS attribute selector name	CSS
di-cp	CSS property name	CSS
di-cv	CSS property value	CSS

Changes

• extendSyntaxTokens (new): Single-pass HAST walker that adds di-* classes based on grammar scope. Language capabilities extracted into shared languageCapabilities.ts.
• enhanceCodeInline: Extended with tag bracket wrapping (di-ht/di-jt), token reclassification (pl-en "function" → pl-k), and built-in type enhancement for inline code (TypeScript only, using shared getLanguageCapabilities).
• parseImportsAndComments: Fixed scoped package parsing (@wooorm/starry-night/style/light was not recognized).
• Docs: Updated parse-source/page.mdx and enhance-code-inline/page.mdx.
• Demos: Updated imports to use centralized syntax.css.

Breaking

• New .di-ht and .di-jt classes need to be added to properly style <div> and <Box> inline

[code-infra-dashboard]

Deploy preview

• docs/app/docs-infra/pipeline/enhance-code-inline/page.mdx
• docs/app/docs-infra/pipeline/page.mdx
• docs/app/docs-infra/pipeline/parse-source/page.mdx

Bundle size

Bundle	Parsed size	Gzip size
@base-ui/react	0B(0.00%)	0B(0.00%)
@mui/x-charts-pro	0B(0.00%)	0B(0.00%)

Details of bundle changes

Performance

Total duration: 16.10 ms +0.37 ms^(+2.3%) | Renders: 4 ⁽⁺⁰⁾ | Paint: 70.20 ms +2.59 ms^(+3.8%)

Test	Duration	Renders
HeavyList mount	9.91 ms 🔺+0.34 ms(+3.6%)	1 (+0)
Counter click	4.22 ms 🔺+0.05 ms(+1.2%)	2 (+0)
DataGrid mount with paint timing	1.97 ms ▼-0.02 ms(-1.0%)	1 (+0)

Details of benchmark changes

---

Check out the code infra dashboard for more information about this PR.

[Copilot]

Pull request overview

Extends the docs-infra code highlighting pipeline by post-processing starry-night HAST output to add fine-grained di-* token classes (while preserving existing pl-* classes), enabling more precise styling for numbers, booleans, nullish values, JSX tags/components, and CSS constructs.

Changes:

• Add extendSyntaxTokens + shared languageCapabilities to enrich highlighted HAST with additive di-* classes, applied in both server (parseSource) and client (CodeProvider) highlighting paths.
• Update inline-code enhancement to wrap HTML/JSX tag brackets with di-ht / di-jt, and add TS-only built-in type enhancement for inline code.
• Fix CSS @import parsing to treat scoped package imports as externals; centralize demo theme + token styling into syntax.css and update docs/demos accordingly.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/docs-infra/src/pipeline/transformHtmlCodeInline/transformHtmlCodeInline.ts	Runs extendSyntaxTokens for inline highlighted code trees.
packages/docs-infra/src/pipeline/parseSource/parseSource.ts	Runs extendSyntaxTokens during block highlighting before gutters.
packages/docs-infra/src/pipeline/parseSource/languageCapabilities.ts	New shared capability model derived from grammar scope.
packages/docs-infra/src/pipeline/parseSource/extendSyntaxTokens.ts	New HAST post-processor adding additive di-* classes.
packages/docs-infra/src/pipeline/parseSource/extendSyntaxTokens.test.ts	Extensive unit coverage for extendSyntaxTokens behaviors.
packages/docs-infra/src/pipeline/loaderUtils/parseImportsAndComments.ts	Treats scoped @scope/... CSS imports as externals.
packages/docs-infra/src/pipeline/loaderUtils/parseImportsAndComments.test.ts	Adds assertions for scoped CSS import handling.
packages/docs-infra/src/pipeline/loadServerTypes/snapshots/highlightTypes.test.ts.snap	Snapshot updates reflecting new di-* classes.
packages/docs-infra/src/pipeline/enhanceCodeTypes/getLanguageCapabilities.ts	Reuses shared LanguageCapabilities type.
packages/docs-infra/src/pipeline/enhanceCodeInline/enhanceCodeInline.ts	Adds di-ht/di-jt wrappers and TS built-in type enhancement.
packages/docs-infra/src/pipeline/enhanceCodeInline/enhanceCodeInline.test.ts	Updates expectations for wrapper spans + built-in type enhancement.
packages/docs-infra/src/pipeline/enhanceCodeEmphasis/enhanceCodeEmphasis.test.ts	Snapshot updates reflecting new di-* classes in emphasized output.
packages/docs-infra/src/CodeProvider/CodeProvider.tsx	Applies extendSyntaxTokens in client-side highlighting path.
docs/components/DemoPerformanceContentLoading/DemoPerformanceContentLoading.tsx	Switches demo styling import to centralized syntax.css.
docs/components/DemoPerformanceContent/DemoPerformanceContent.tsx	Switches demo styling import to centralized syntax.css.
docs/app/global.css	Removes inline .pl-smi override from global stylesheet.
docs/app/docs-infra/pipeline/parse-source/page.mdx	Documents di-* classes and updates sample language block.
docs/app/docs-infra/pipeline/page.mdx	Updates pipeline docs to reflect new inline enhancements + token classes.
docs/app/docs-infra/pipeline/enhance-code-inline/page.mdx	Updates docs for di-ht/di-jt wrappers + TS built-in type enhancement.
docs/app/docs-infra/pipeline/enhance-code-emphasis/demos/IndentContent.tsx	Uses centralized syntax.css for demo styling.
docs/app/docs-infra/pipeline/enhance-code-emphasis/demos/CollapsibleContent.tsx	Uses centralized syntax.css for demo styling.
docs/app/docs-infra/components/code-highlighter/demos/syntax.css	New centralized theme import + di-* styling variables/rules.
docs/app/docs-infra/components/code-highlighter/demos/demo-fallback/DemoContentLoading.tsx	Uses centralized syntax.css.
docs/app/docs-infra/components/code-highlighter/demos/demo-fallback-all-variants/DemoContentLoading.tsx	Uses centralized syntax.css.
docs/app/docs-infra/components/code-highlighter/demos/demo-fallback-all-files/DemoContentLoading.tsx	Uses centralized syntax.css.
docs/app/docs-infra/components/code-highlighter/demos/DemoContent.tsx	Uses centralized syntax.css.
docs/app/docs-infra/components/code-highlighter/demos/CodeContent.tsx	Uses centralized syntax.css.
docs/app/docs-infra/components/code-controller-context/demos/multi-file/MultiFileContent.tsx	Uses centralized syntax.css.
docs/app/docs-infra/components/code-controller-context/demos/demo-live/DemoLiveContent.tsx	Uses centralized syntax.css.
docs/app/docs-infra/components/code-controller-context/demos/code-editor/CodeEditorContent.tsx	Uses centralized syntax.css.
docs/app/bench/docs-infra/components/code-highlighter/demos/CodeContentLoading.tsx	Uses centralized syntax.css.
docs/app/bench/docs-infra/components/code-highlighter/demos/CodeContent.tsx	Uses centralized syntax.css.

[Janpot]

Is starry-night only used to do the syntax highlighting? With the amount of code/complexity introduced to post-process it, almost feels like we could write our own HAST generator from a token stream. e.g. create a hastHighlighter for lezer or similar.

[dav-is]

Is starry-night only used to do the syntax highlighting?

Yes, but it uses the same regex that VSCode uses. Our extension leans on existing HAST annotations that it outputs, checks their contents or siblings, and adds more detail. It doesn't need to read the entire source text again and only makes a single pass over the HAST. It's essentially a rehype plugin, the same API users of docs-infra would have to extend highlighting

we could write our own HAST generator from a token stream

It seems tempting, but I think starry-night has a solid core, written by the maintainer of HAST and MDX. It would be interesting to contribute upstream to it to give us a more performant entrypoint for our extensions