Keep the source envelope across a load and a save, and pin where a composite figure stands

README.md

@@ -86,6 +86,12 @@ silently discarded.
 - Admonition divs (`:::warning`) and their container class.
 - Footnotes (reference + definition), including their authored labels.
 - Unsupported constructs such as frontmatter through source preservation.
+- Block attributes above a construct the editor models only partly - a
+  `{#fig-x}` over a `:::` fence - through the document's source envelope. Loads
+  go through `setCarveDocument` and saves through `editorToCarve` for that
+  reason: Tiptap's `setContent` replaces the doc's content and leaves the doc
+  node's attributes behind, so the envelope has to be re-attached. See
+  `src/editor.ts`.
 - The language attribute: an imported `{lang="fr"}` span keeps its value on the
   span mark and serializes back as the `{:fr}` sugar, a `<span lang>` in pasted
   HTML parses onto the same mark, and a value that is not a language tag keeps
@@ -100,6 +106,16 @@ silently discarded.
   cannot move it), and that build predates the production: the run stays
   literal text and comes back with the bracket escaped. Authoring the same span
   as `{lang="fr"}` works today and serializes as `{:fr}`.
+- **Composite figures** (`::: figure` with no title and no label, Carve PART 9
+  section 4c) are not modelled as figures. The mapping is the CarveKit schema in
+  `@markup-carve/carve-grammars`, not this app, and the engine that parses the
+  editor's input is the one carve-grammars pins for its own loader - which
+  predates the construct. So today a composite figure is a generic container
+  and round-trips as one; when that engine moves, the group arrives as a single
+  read-only source atom until carve-grammars gives it a schema entry.
+  `tests/composite-figure.test.ts` holds both states and fails when the second
+  one changes, which is the signal to model it here. Tracked as
+  markup-carve/carve-wysiwyg#15.
 - **CriticMarkup containing its own closing delimiter** (`+}` / `-}` inside
   `{+...+}` / `{-...-}`) cannot round-trip - Carve provides no escape for it.
   This is an upstream serializer limitation noted in carve-grammars.

src/editor.ts

@@ -8,6 +8,7 @@
  * change.
  */
 import { Editor } from '@tiptap/core';
+import type { JSONContent } from '@tiptap/core';
 import { CarveKit, serializeToCarve } from '@markup-carve/carve-grammars/tiptap';
 
 export interface CarveEditorOptions {
@@ -16,21 +17,124 @@ export interface CarveEditorOptions {
   onUpdate?: (carve: string) => void;
 }
 
+/** The document-level attribute names CarveKit's source envelope uses. */
+const ENVELOPE_ATTRS = ['carveSource', 'carveFingerprint', 'carveSourceLayout'] as const;
+
+type Envelope = Record<string, unknown>;
+
+/**
+ * The source envelope of the document currently loaded into each editor.
+ *
+ * WHY THIS IS HELD HERE RATHER THAN ON THE DOCUMENT. `carveToProseMirror`
+ * returns a doc carrying `carveSource` / `carveFingerprint` /
+ * `carveSourceLayout` whenever the rich model would be lossy to write back, and
+ * `serializeToCarve` writes that source verbatim for as long as the fingerprint
+ * still matches the document - which is how a construct with no exact editor
+ * model survives load/save instead of being normalized away.
+ *
+ * Tiptap's `setContent` does not carry them. It dispatches
+ * `tr.replaceWith(0, doc.content.size, document)`, which replaces the doc's
+ * CONTENT and leaves the doc NODE - and therefore its attributes - as it was,
+ * so the envelope never reaches `editor.getJSON()`. Measured, not assumed: the
+ * import produces the three attrs and the editor hands back all three as
+ * `null`.
+ *
+ * The visible symptom was a block-attribute line above a fenced div. CarveKit's
+ * `carveDiv` models the class but not the `{#id}`, so the rich document is
+ * lossy, the bridge produced an envelope, the envelope was dropped, and
+ *
+ *   {#fig-x}
+ *   ::: figure
+ *   ![one](a.png)
+ *   ^ (a) One
+ *   :::
+ *
+ * came back out without its `{#fig-x}` on the first save.
+ *
+ * A WeakMap rather than a field on the editor so nothing is retained after an
+ * editor is destroyed. Re-attaching is safe without any staleness check of its
+ * own: the fingerprint is that check, and the serializer falls through to
+ * ordinary serialization the moment the document is edited.
+ */
+const envelopes = new WeakMap<Editor, Envelope>();
+
+/** The envelope attrs of a bridge document, or null when it carries none. */
+function envelopeOf(doc: JSONContent): Envelope | null {
+  const attrs = (doc as { attrs?: Envelope }).attrs;
+  if (!attrs) return null;
+  const kept: Envelope = {};
+  for (const name of ENVELOPE_ATTRS) {
+    if (attrs[name] != null) kept[name] = attrs[name];
+  }
+  return Object.keys(kept).length ? kept : null;
+}
+
+/**
+ * Drop attributes the editor materialized from schema defaults.
+ *
+ * The envelope is guarded by a FINGERPRINT of the document it was taken from,
+ * and the fingerprint is over the bridge's JSON - which carries only the attrs
+ * that were actually set. Tiptap hands back every attribute the schema
+ * declares, so `{"class":"figure"}` returns as
+ * `{"id":null,"keyValues":null,"label":null,"class":"figure","title":null}` and
+ * the two never compare equal. Re-attaching the envelope without this is
+ * therefore inert: the fingerprint check fails every time and the verbatim
+ * source is never used.
+ *
+ * Dropping nulls is not a reinterpretation of the document. The serializer
+ * reads every one of these with optional chaining, so an absent attribute and a
+ * null one already mean the same thing to it - what changes is only whether the
+ * fingerprint can recognize its own document.
+ */
+function pruneDefaults(value: unknown): unknown {
+  if (Array.isArray(value)) return value.map(pruneDefaults);
+  if (!value || typeof value !== 'object') return value;
+  const out: Record<string, unknown> = {};
+  for (const [key, inner] of Object.entries(value)) {
+    if (inner === null) continue;
+    out[key] = pruneDefaults(inner);
+  }
+  const attrs = out['attrs'];
+  if (attrs && typeof attrs === 'object' && !Object.keys(attrs).length) delete out['attrs'];
+  return out;
+}
+
+/** `editor.getJSON()` with the loaded document's envelope put back on it. */
+function withEnvelope(editor: Editor, json: JSONContent): JSONContent {
+  const envelope = envelopes.get(editor);
+  if (!envelope) return json;
+  const pruned = pruneDefaults(json) as JSONContent;
+  return { ...pruned, attrs: { ...(pruned.attrs ?? {}), ...envelope } };
+}
+
 export function createCarveEditor(opts: CarveEditorOptions): Editor {
-  const editor = new Editor({
+  const editor: Editor = new Editor({
     element: opts.element,
     extensions: [CarveKit],
     content: opts.content ?? '',
     onUpdate: ({ editor }) => {
-      opts.onUpdate?.(serializeToCarve(editor.getJSON()));
+      opts.onUpdate?.(editorToCarve(editor));
     },
   });
   return editor;
 }
 
+/**
+ * Load a bridge document into the editor, keeping its source envelope.
+ *
+ * Every load goes through here rather than through `setContent` directly, so
+ * the previous document's envelope cannot outlive it.
+ */
+export function setCarveDocument(editor: Editor, doc: JSONContent): void {
+  const envelope = envelopeOf(doc);
+  if (envelope) envelopes.set(editor, envelope);
+  else envelopes.delete(editor);
+  editor.commands.setContent(doc);
+}
+
 /** Serialize the current editor document to Carve markup. */
 export function editorToCarve(editor: Editor): string {
-  return serializeToCarve(editor.getJSON());
+  return serializeToCarve(withEnvelope(editor, editor.getJSON()));
 }
 
 /** Re-serialize an arbitrary ProseMirror/Tiptap JSON doc. */

src/main.ts

@@ -15,7 +15,7 @@ import { Editor } from '@tiptap/core';
 import type {} from '@tiptap/starter-kit';
 import type {} from '@tiptap/extension-link';
 import type {} from '@tiptap/extension-underline';
-import { createCarveEditor, editorToCarve } from './editor';
+import { createCarveEditor, editorToCarve, setCarveDocument } from './editor';
 import { carveToEditorDocument, carveToHtmlRaw } from './carve-import';
 
 const SAMPLE = `# Carve WYSIWYG
@@ -62,9 +62,10 @@ function refreshOutputs(carve: string): void {
 
 /** Load Carve source into the editor (import direction). */
 function loadCarve(source: string): void {
-  const editorDocument = carveToEditorDocument(source);
+  // setCarveDocument, not setContent: the bridge's source envelope rides on
+  // the document's own attributes, which setContent does not carry (editor.ts).
   // emitUpdate defaults to false; we refresh the outputs explicitly below.
-  editor.commands.setContent(editorDocument);
+  setCarveDocument(editor, carveToEditorDocument(source));
   refreshOutputs(editorToCarve(editor));
 }
 

src/types/carve-grammars.d.ts

@@ -18,6 +18,18 @@ declare module '@markup-carve/carve-grammars/tiptap' {
     options?: CarveLoaderOptions,
   ): JSONContent;
 
+  /**
+   * Convert a Carve AST document to a ProseMirror/Tiptap JSON document.
+   *
+   * The seam the loader itself uses. Declared here because it is the only way
+   * to exercise a node type the INSTALLED engine cannot parse yet - see
+   * tests/composite-figure.test.ts.
+   */
+  export function astToProseMirror(
+    ast: unknown,
+    options?: CarveLoaderOptions & { source?: string },
+  ): JSONContent;
+
   /** Serialize a Tiptap/ProseMirror JSON document to Carve markup. */
   export function serializeToCarve(doc: unknown): string;
 

tests/blockquote-attribution.test.ts

@@ -17,6 +17,7 @@ import { describe, it, expect, beforeAll, afterAll } from 'vitest';
 import { Editor } from '@tiptap/core';
 import { CarveKit, serializeToCarve } from '@markup-carve/carve-grammars/tiptap';
 import { carveToEditorDocument } from '../src/carve-import';
+import { setCarveDocument } from '../src/editor';
 
 let editor: Editor;
 
@@ -32,8 +33,12 @@ afterAll(() => {
 
 /** Load Carve, type somewhere that is NOT the attribution, then serialize. */
 function editElsewhereAndSerialize(source: string): string {
-  editor.commands.setContent(carveToEditorDocument(source));
+  setCarveDocument(editor, carveToEditorDocument(source));
   editor.commands.insertContentAt(1, 'EDITED');
+  // serializeToCarve on the raw editor JSON, deliberately NOT editorToCarve:
+  // the point is what the CONTENT tree holds, and the app's serializer would
+  // re-attach the document's source envelope, which is the very thing this
+  // test must not read.
   return serializeToCarve(editor.getJSON());
 }
 
@@ -65,7 +70,7 @@ describe("a quote's attribution survives an edit", () => {
   });
 
   it('holds the attribution in an editable node, not only the source envelope', () => {
-    editor.commands.setContent(carveToEditorDocument('> Stay hungry, stay foolish.\n^ Steve Jobs\n'));
+    setCarveDocument(editor, carveToEditorDocument('> Stay hungry, stay foolish.\n^ Steve Jobs\n'));
     expect(editableText()).toContain('Steve Jobs');
   });
 
@@ -76,7 +81,7 @@ describe("a quote's attribution survives an edit", () => {
   });
 
   it('control: the probe reads the content tree and not everything', () => {
-    editor.commands.setContent(carveToEditorDocument('> Just a quote\n'));
+    setCarveDocument(editor, carveToEditorDocument('> Just a quote\n'));
     expect(editableText()).toContain('Just a quote');
     expect(editableText()).not.toContain('Steve Jobs');
   });

tests/composite-figure.test.ts

@@ -0,0 +1,129 @@
+/**
+ * Composite figures (Carve PART 9 section 4c, markup-carve/carve#1215) in the editor.
+ *
+ * WHERE THE MAPPING LIVES, because this repository is not it. The
+ * Carve <-> ProseMirror bridge - the CarveKit schema, `carveToProseMirror` and
+ * `serializeToCarve` - is `@markup-carve/carve-grammars`, and the engine that
+ * parses the source on the way in is the one carve-grammars nests for its own
+ * loader, not the `@markup-carve/carve` this app installs for the preview
+ * pane. So a `figure_group` node type reaches the editor only when
+ * carve-grammars ships both an engine that parses it and a schema entry that
+ * models it. Nothing in `src/` can do either.
+ *
+ * What this file does instead is measure that boundary rather than assert it,
+ * in the two states it can be in:
+ *
+ *   1. THE ENGINE THE EDITOR ACTUALLY RUNS TODAY predates section 4c, so a bare
+ *      `::: figure` is a generic container and survives the round trip. That is
+ *      the state, and it is worth a test because "it happens to work" and "it is
+ *      modelled" look identical from the outside.
+ *   2. WHAT ARRIVES WHEN THE ENGINE MOVES, exercised through a `figure_group`
+ *      AST captured from an engine that has the node. It reaches the editor as
+ *      one opaque source atom - lossless, and not editable as a figure.
+ *
+ * The second test is the handshake: it goes red the day carve-grammars gives
+ * the group a real schema entry, which is the signal to wire the editor up to
+ * it. See markup-carve/carve-wysiwyg#15.
+ */
+import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import { Editor } from '@tiptap/core';
+import { CarveKit, serializeToCarve, astToProseMirror } from '@markup-carve/carve-grammars/tiptap';
+import { carveToEditorDocument } from '../src/carve-import';
+import { editorToCarve, setCarveDocument } from '../src/editor';
+import fixture from './fixtures/figure-group.ast.json';
+
+let editor: Editor;
+
+beforeAll(() => {
+  const el = document.createElement('div');
+  document.body.appendChild(el);
+  editor = new Editor({ element: el, extensions: [CarveKit] });
+});
+
+afterAll(() => {
+  editor?.destroy();
+});
+
+/** The app's own import + serialize path, exactly as main.ts drives it. */
+function roundTrip(source: string): string {
+  setCarveDocument(editor, carveToEditorDocument(source));
+  return editorToCarve(editor);
+}
+
+const GROUP = [
+  '{#fig-x}',
+  '::: figure',
+  '{#fig-a}',
+  '![one](a.png)',
+  '^ (a) One',
+  '',
+  '{#fig-b}',
+  '![two](b.png)',
+  '^ (b) Two',
+  ':::',
+  '^ Group caption',
+  '',
+].join('\n');
+
+/**
+ * The CONTROL spelling. Under section 4c an opener carrying a quoted title is
+ * NOT the composite production - it stays a generic container.
+ *
+ * IT DOES NOT DISCRIMINATE ANYTHING YET, and saying so is the point. The engine
+ * the editor runs predates section 4c, so both spellings parse to the same
+ * generic container and this pair proves only that neither is mangled. Reading
+ * a passing pair as evidence that the two are told apart is the exact mistake
+ * this construct invites, so the discrimination is asserted where it can be -
+ * against the captured AST below - and not here.
+ */
+const TITLED = GROUP.replace('::: figure', '::: figure "A titled figure div"');
+
+describe('composite figures through the editor', () => {
+  it('a bare figure container survives the round trip under the engine in use', () => {
+    expect(roundTrip(GROUP)).toBe(GROUP);
+  });
+
+  it('so does the titled spelling, which is a different production', () => {
+    expect(roundTrip(TITLED)).toBe(TITLED);
+  });
+
+  it('the engine in use does not produce the node yet', () => {
+    // The premise the two tests above rest on, checked rather than assumed: if
+    // this ever fails, they stopped describing a pre-section-4c engine and the
+    // group test above may be passing for a different reason than it says.
+    const doc = carveToEditorDocument(GROUP) as { content?: Array<{ type?: string }> };
+    expect(doc.content?.[0]?.type).toBe('carveDiv');
+  });
+
+  it('a figure_group reaches the editor as one opaque atom, not as a figure', () => {
+    // The AST an engine WITH section 4c hands the bridge, captured from
+    // fixture.capturedFrom - the installed engine cannot produce it, so it is
+    // supplied rather than parsed.
+    //
+    // WHEN THIS FAILS, carve-grammars has given the group a schema entry.
+    // That is the moment to model it here: drop this expectation, assert the
+    // rich shape, and give the preview pane styles for
+    // carve-figure-group / carve-figure-panels / carve-figure-panel.
+    const doc = astToProseMirror(fixture.ast, {
+      unsupported: 'preserve',
+      source: fixture.source,
+    }) as { content: Array<{ type: string; attrs?: { carveSource?: string } }> };
+
+    expect(doc.content.map((n) => n.type)).toEqual(['carveUnsupported']);
+
+    // Lossless as SOURCE, which is the guarantee `preserve` actually makes -
+    // the group is not silently dropped and not flattened into its panels.
+    const kept = doc.content[0]!.attrs!.carveSource!;
+    expect(kept).toContain('::: figure');
+    expect(kept).toContain('![one](a.png)');
+    expect(kept).toContain('^ Group caption');
+    expect(serializeToCarve(doc)).toBe(kept);
+
+    // And the loss that IS there, named rather than left to be discovered: a
+    // block's position excludes the block-attribute line above it (every block
+    // type, not just this one), so the slice starts at the opening fence and
+    // the group's own `{#fig-x}` is not in it.
+    expect(kept.startsWith('::: figure')).toBe(true);
+    expect(kept).not.toContain('{#fig-x}');
+  });
+});