ASCII-safe heading id slugs (baked Unicode -> ASCII translit) - #13
Merged
Conversation
Ports the deterministic-fallback transliteration from djot-php #183 so auto-generated heading ids survive being shared as URL fragments (auto-linkers routinely truncate or mis-encode non-ASCII). - src/translit-map.ts: baked Unicode->ASCII map (~903 entries, mechanical port from carve-php's src/Renderer/ascii_translit_map.php which is the same ICU "Any-Latin; Latin-ASCII" transform). - src/heading-ids.ts: slugify NFC-normalizes the input, transliterates through the map, then runs the existing normalize pipeline (lowercase, drop CSS-unsafe punct, collapse runs, digit-prefix, fallback). Greek is deliberately excluded (ICU translit is context-sensitive -- `au` vs `y` -- so it can't be baked as a context-free map). Greek/CJK/ Arabic headings pass through unchanged; authors can attach an explicit `{#id}` for a share-safe slug if needed. Matches carve-php's deterministic-fallback behavior byte-for-byte. Tests: - test/heading-id-ascii.test.ts: 8 new cases (Latin diacritics, Cyrillic, smart-punct, digit-prefix, empty fallback, NFC vs NFD, Greek/CJK/Arabic pass-through). - test/heading-ids.test.ts: pre-existing Unicode-preservation expectations updated to ASCII. - test/corpus.test.ts: 19-heading-ids removed from IMPLEMENTED pending the carve corpus update (the spec submodule still pins Unicode-preserving fixture). To be re-added once the carve corpus PR lands. 172 tests + 1 todo; typecheck clean; codex review clean.
…ng-ids # Conflicts: # src/heading-ids.ts
dereuromark
added a commit
that referenced
this pull request
May 20, 2026
…15) - spec submodule bumped to markup-carve/carve#16 (Adopt ASCII-safe heading id slugs spec + corpus + re-vendor). The corpus fixture for 19-heading-ids now matches the ASCII slugs the impl emits. - Un-skips '19-heading-ids' in IMPLEMENTED — the TODO PR #13 left pending the corpus bump. - Adds the 17 new corpus entries (25-41) the spec now ships: table doubled alignment marker, fenced-code shorter inner fence, blockquote caption after blank line, escaped/code-span pipes in table cells, abbreviation/mention/tag word-boundary rules, stacked rowspan, reference-link / collapsed / unresolved cases, three smart-typography splits, multi-line cell continuation, and rowspan with multi-line content. All 17 pass byte-for-byte against the current impl. - Adds vitest.config.ts to scope test discovery to test/**/*.test.ts. The bumped spec submodule carries spec/tests/*.test.mjs files (node:test runners for the carve repo's own CI) that vitest would otherwise try to load and fail on as 'no test suite found'. Suite: 217/217 (was 187). Typecheck clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the deterministic-fallback transliteration from
djot-php#183 so auto-generated heading ids survive being shared as URL fragments. Auto-linkers (Slack, email, plain-text aggregators) routinely truncate or mis-encode non-ASCII fragments; ASCII slugs are the only fix that doesn't depend on every downstream tool getting URL detection right.Behavior
Café Notescafe-notesÜber unsuber-unsПривет мирprivet-mir(Cyrillic mapped)Bob's Guide(smart quote)bobs-guide日本語の見出し日本語の見出し(CJK pass-through)Καλημέρακαλημέρα(Greek pass-through, deliberately)Matches carve-php's deterministic-fallback behavior byte-for-byte (both ports of the same djot-php baked map).
Why Greek/CJK/Arabic pass through
The baked map covers Latin / IPA / combining marks / Cyrillic / Latin-Extended-Additional / punctuation / super- and sub-script / currency / letterlike. Greek is deliberately excluded — its ICU transliteration is context-sensitive (
αυ→aubutυ→y) so it can't be baked as a context-free map. CJK / Arabic aren't in the deterministic map either. For those, authors can attach an explicit{#id}for a share-safe slug.What's in the patch
src/translit-map.ts— baked Unicode → ASCII map (~903 entries, mechanical port fromcarve-php/src/Renderer/ascii_translit_map.php).src/heading-ids.ts—slugifyNFC-normalizes the input first (so a decomposedrésuméslugs identically to its precomposed form — codex round 2), transliterates through the map, then runs the existing normalize pipeline.Tests
test/heading-id-ascii.test.ts— 8 cases covering Latin diacritics, Cyrillic, smart-punct, digit-prefix, empty fallback, NFC ↔ NFD parity, Greek / CJK / Arabic pass-through.test/heading-ids.test.ts— pre-existing Unicode-preservation expectations updated to ASCII.test/corpus.test.ts—19-heading-idstemporarily removed fromIMPLEMENTEDwhile the carve spec corpus updates its fixture from Unicode to ASCII expectations. Re-add once the carve corpus PR lands.172 tests + 1 todo; typecheck clean; codex review clean (two real findings fixed: missing Greek-claim correction in docs, NFD normalization bug).
Coordination
This is the impl side. Companion carve repo PR to follow that updates
syntax.md§4.1, the19-heading-idsfixture, and the README Comparison cell, then bumps the spec submodule and re-vendorscarve-lib.