Skip to content

📝 fix: Preserve Raw Markdown Formatting on Upload as Text #12734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danny-avila merged 4 commits into from
Apr 20, 2026
+101 −0
Merged

📝 fix: Preserve Raw Markdown Formatting on Upload as Text #12734

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
275b20d
🐛 fix: Preserve Raw Markdown on `Upload as Text`
danny-avila Apr 18, 2026
99f062d
🐛 fix: normalize markdown MIME and accept `text/md`
danny-avila Apr 18, 2026
4d47e80
🧹 chore: Address Code Review Follow-ups on `Upload as Text` fix
danny-avila Apr 18, 2026
3e3917c
🧪 test: Cover `.mdwn` regex branch in Markdown short-circuit
danny-avila Apr 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions packages/api/src/files/text.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,5 +300,73 @@ describe('text', () => {
source: FileSources.text,
});
});

it.each([
{ mimetype: 'text/markdown', originalname: 'notes.md' },
{ mimetype: 'text/x-markdown', originalname: 'notes.md' },
{ mimetype: 'text/md', originalname: 'notes' },
{ mimetype: 'application/markdown', originalname: 'notes.md' },
{ mimetype: 'application/x-markdown', originalname: 'notes.md' },
{ mimetype: 'text/plain', originalname: 'notes.md' },
{ mimetype: 'application/octet-stream', originalname: 'README.md' },
{ mimetype: 'application/octet-stream', originalname: 'GUIDE.MARKDOWN' },
{ mimetype: 'application/octet-stream', originalname: 'post.mdown' },
{ mimetype: 'application/octet-stream', originalname: 'post.mkdn' },
{ mimetype: 'application/octet-stream', originalname: 'post.mkd' },
{ mimetype: 'application/octet-stream', originalname: 'docs.mdwn' },
{ mimetype: 'text/markdown; charset=utf-8', originalname: 'notes' },
{ mimetype: 'TEXT/MARKDOWN', originalname: 'notes' },
{ mimetype: ' text/markdown ; charset=UTF-8 ', originalname: 'notes' },
{ mimetype: '', originalname: 'notes.md' },
])(
Comment on lines +304 to +321
Copy link

Copilot AI Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The markdown short-circuit tests cover several markdown MIME types and extension-only detection, but they don't cover the text/md MIME type which is treated elsewhere in the codebase as valid Markdown. Adding a text/md case (ideally with a non-.md filename to ensure the MIME-type path is what triggers) would prevent regressions where markdown still gets routed through the RAG /text endpoint.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Owner Author

@danny-avila danny-avila Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a text/md test case (with an extension-less filename so the MIME path is what triggers) and three more cases covering the normalization edges: text/markdown; charset=utf-8, uppercase TEXT/MARKDOWN, and whitespace-padded input. See 99f062d2a.

'should short-circuit to native parsing for markdown file (%o)',
async ({ mimetype, originalname }) => {
process.env.RAG_API_URL = 'http://rag-api.test';
const mockText = '# Heading\n\n**bold** text';
const mockBytes = Buffer.byteLength(mockText, 'utf8');

mockedReadFileAsString.mockResolvedValue({
content: mockText,
bytes: mockBytes,
});

const result = await parseText({
req: mockReq,
file: { ...mockFile, mimetype, originalname },
file_id: mockFileId,
});

expect(mockedAxios.get).not.toHaveBeenCalled();
expect(mockedAxios.post).not.toHaveBeenCalled();
expect(mockedReadFileAsString).toHaveBeenCalledWith('/tmp/test.txt', {
fileSize: 100,
});
expect(result).toEqual({
text: mockText,
bytes: mockBytes,
source: FileSources.text,
});
},
);

it('should still call the RAG API for non-markdown text files', async () => {
process.env.RAG_API_URL = 'http://rag-api.test';
const mockText = 'plain text content';

mockedAxios.get.mockResolvedValue({ status: 200, statusText: 'OK' });
mockedAxios.post.mockResolvedValue({ data: { text: mockText } });

await parseText({
req: mockReq,
file: mockFile,
file_id: mockFileId,
});

expect(mockedAxios.post).toHaveBeenCalledWith(
'http://rag-api.test/text',
expect.any(Object),
expect.objectContaining({ timeout: 300000 }),
);
});
});
});
33 changes: 33 additions & 0 deletions packages/api/src/files/text.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,32 @@ import type { ServerRequest } from '~/types';
import { logAxiosError, readFileAsString } from '~/utils';
import { generateShortLivedToken } from '~/crypto/jwt';

const MARKDOWN_MIME_TYPES = new Set([
'text/markdown',
'text/x-markdown',
Copy link

Copilot AI Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isMarkdownFile() checks a fixed set of MIME types but does not include text/md. In the codebase, text/md is treated as a valid Markdown MIME type (e.g., artifacts prompt), and the Upload-as-Text path accepts arbitrary type/subtype values—so a Markdown upload with mimetype text/md (especially if the filename lacks a .md extension) will still go through the RAG /text endpoint and have formatting stripped. Consider adding text/md to the Markdown MIME set (or normalizing MIME types using the same canonicalization used elsewhere) so Markdown is consistently short-circuited.

Suggested change
'text/x-markdown',
'text/x-markdown',
'text/md',

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Owner Author

@danny-avila danny-avila Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — text/md is treated as a valid markdown MIME elsewhere in the codebase (e.g. artifacts prompt). Added it to MARKDOWN_MIME_TYPES in 99f062d2a, plus MIME-type normalization so parameterized variants like text/markdown; charset=utf-8 also short-circuit.

'text/md',
'application/markdown',
'application/x-markdown',
]);

const MARKDOWN_EXTENSIONS_RE = /\.(md|markdown|mdown|mkdn|mkd|mdwn)$/i;

function normalizeMimeType(mimetype: string): string {
if (!mimetype) {
return '';
}
const semi = mimetype.indexOf(';');
const base = semi === -1 ? mimetype : mimetype.slice(0, semi);
return base.trim().toLowerCase();
}

function isMarkdownFile(file: Express.Multer.File): boolean {
if (MARKDOWN_MIME_TYPES.has(normalizeMimeType(file.mimetype))) {
return true;
}
return MARKDOWN_EXTENSIONS_RE.test(file.originalname ?? '');
}

/**
* Attempts to parse text using RAG API, falls back to native text parsing
* @param params - The parameters object
Expand All @@ -29,6 +55,13 @@ export async function parseText({
return parseTextNative(file);
}

if (isMarkdownFile(file)) {
logger.debug(
`[parseText] Markdown file detected (${file.originalname}, ${file.mimetype}), using native parsing to preserve raw formatting`,
);
return parseTextNative(file);
}

const userId = req.user?.id;
if (!userId) {
logger.debug('[parseText] No user ID provided, falling back to native text parsing');
Expand Down
Loading