Skip to content

ASCII-safe heading id slugs (baked Unicode -> ASCII translit) - #13

Merged
dereuromark merged 2 commits into
mainfrom
feat/ascii-safe-heading-ids
May 20, 2026
Merged

ASCII-safe heading id slugs (baked Unicode -> ASCII translit)#13
dereuromark merged 2 commits into
mainfrom
feat/ascii-safe-heading-ids

Conversation

@dereuromark

Copy link
Copy Markdown
Contributor

Summary

Ports the deterministic-fallback transliteration from djot-php #183 so auto-generated heading ids survive being shared as URL fragments. Auto-linkers (Slack, email, plain-text aggregators) routinely truncate or mis-encode non-ASCII fragments; ASCII slugs are the only fix that doesn't depend on every downstream tool getting URL detection right.

Behavior

Heading source Slug
Café Notes cafe-notes
Über uns uber-uns
Привет мир privet-mir (Cyrillic mapped)
Bob's Guide (smart quote) bobs-guide
日本語の見出し 日本語の見出し (CJK pass-through)
Καλημέρα καλημέρα (Greek pass-through, deliberately)

Matches carve-php's deterministic-fallback behavior byte-for-byte (both ports of the same djot-php baked map).

Why Greek/CJK/Arabic pass through

The baked map covers Latin / IPA / combining marks / Cyrillic / Latin-Extended-Additional / punctuation / super- and sub-script / currency / letterlike. Greek is deliberately excluded — its ICU transliteration is context-sensitive (αυau but υy) so it can't be baked as a context-free map. CJK / Arabic aren't in the deterministic map either. For those, authors can attach an explicit {#id} for a share-safe slug.

What's in the patch

  • src/translit-map.ts — baked Unicode → ASCII map (~903 entries, mechanical port from carve-php/src/Renderer/ascii_translit_map.php).
  • src/heading-ids.tsslugify NFC-normalizes the input first (so a decomposed résumé slugs identically to its precomposed form — codex round 2), transliterates through the map, then runs the existing normalize pipeline.

Tests

  • New test/heading-id-ascii.test.ts — 8 cases covering Latin diacritics, Cyrillic, smart-punct, digit-prefix, empty fallback, NFC ↔ NFD parity, Greek / CJK / Arabic pass-through.
  • test/heading-ids.test.ts — pre-existing Unicode-preservation expectations updated to ASCII.
  • test/corpus.test.ts19-heading-ids temporarily removed from IMPLEMENTED while the carve spec corpus updates its fixture from Unicode to ASCII expectations. Re-add once the carve corpus PR lands.

172 tests + 1 todo; typecheck clean; codex review clean (two real findings fixed: missing Greek-claim correction in docs, NFD normalization bug).

Coordination

This is the impl side. Companion carve repo PR to follow that updates syntax.md §4.1, the 19-heading-ids fixture, and the README Comparison cell, then bumps the spec submodule and re-vendors carve-lib.

Ports the deterministic-fallback transliteration from djot-php #183 so
auto-generated heading ids survive being shared as URL fragments
(auto-linkers routinely truncate or mis-encode non-ASCII).

- src/translit-map.ts: baked Unicode->ASCII map (~903 entries, mechanical
  port from carve-php's src/Renderer/ascii_translit_map.php which is the
  same ICU "Any-Latin; Latin-ASCII" transform).
- src/heading-ids.ts: slugify NFC-normalizes the input, transliterates
  through the map, then runs the existing normalize pipeline (lowercase,
  drop CSS-unsafe punct, collapse runs, digit-prefix, fallback).

Greek is deliberately excluded (ICU translit is context-sensitive --
`au` vs `y` -- so it can't be baked as a context-free map). Greek/CJK/
Arabic headings pass through unchanged; authors can attach an explicit
`{#id}` for a share-safe slug if needed. Matches carve-php's
deterministic-fallback behavior byte-for-byte.

Tests:
- test/heading-id-ascii.test.ts: 8 new cases (Latin diacritics,
  Cyrillic, smart-punct, digit-prefix, empty fallback, NFC vs NFD,
  Greek/CJK/Arabic pass-through).
- test/heading-ids.test.ts: pre-existing Unicode-preservation
  expectations updated to ASCII.
- test/corpus.test.ts: 19-heading-ids removed from IMPLEMENTED
  pending the carve corpus update (the spec submodule still pins
  Unicode-preserving fixture). To be re-added once the carve corpus
  PR lands.

172 tests + 1 todo; typecheck clean; codex review clean.
@dereuromark dereuromark added the enhancement New feature or request label May 20, 2026
@dereuromark
dereuromark merged commit 37f0c19 into main May 20, 2026
2 checks passed
@dereuromark
dereuromark deleted the feat/ascii-safe-heading-ids branch May 20, 2026 17:42
dereuromark added a commit that referenced this pull request May 20, 2026
…15)

- spec submodule bumped to markup-carve/carve#16 (Adopt ASCII-safe
  heading id slugs spec + corpus + re-vendor). The corpus fixture
  for 19-heading-ids now matches the ASCII slugs the impl emits.
- Un-skips '19-heading-ids' in IMPLEMENTED — the TODO PR #13 left
  pending the corpus bump.
- Adds the 17 new corpus entries (25-41) the spec now ships:
  table doubled alignment marker, fenced-code shorter inner fence,
  blockquote caption after blank line, escaped/code-span pipes in
  table cells, abbreviation/mention/tag word-boundary rules,
  stacked rowspan, reference-link / collapsed / unresolved cases,
  three smart-typography splits, multi-line cell continuation, and
  rowspan with multi-line content. All 17 pass byte-for-byte
  against the current impl.
- Adds vitest.config.ts to scope test discovery to test/**/*.test.ts.
  The bumped spec submodule carries spec/tests/*.test.mjs files
  (node:test runners for the carve repo's own CI) that vitest
  would otherwise try to load and fail on as 'no test suite found'.

Suite: 217/217 (was 187). Typecheck clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant