Skip to content

About

Extract structured data out of any file, using the amazing Xberg library.

Resources

Stars

2 stars

Watchers

1 watching

Forks

Repository files navigation

Extract Anything

Extract Anything

A versatile data extraction plugin for the xyOps Workflow Automation System.

This Plugin uses the amazing open-source Xberg library to perform structured data extractions on files. It can pull text and markdown out of PDFs, Word Docs or Powerpoint slides, cell grids or CSV out of Excel spreadsheets, perform OCR inside images to extract text, you name it!

To use this Plugin in your workflows, it requires one or more files as inputs. Once the data is extracted, it is returned as job output data (which is passed to subsequent jobs in linked chains and workflows).

Requirements

  • docker

Input Formats

This Plugin can extract structured data from many different formats, including:

Office Documents

Format Extensions MIME Type Extraction Method OCR Support Special Features
PDF .pdf application/pdf Native Rust Yes Metadata extraction, image extraction, text layer detection
Excel .xlsx, .xlsm, .xlsb, .xls, .xlam, .xla, .ods Various Excel MIME types Native Rust (calamine) No Multi-sheet support, formula preservation
PowerPoint .pptx, .pptm, .ppsx application/vnd.openxmlformats-officedocument.presentationml.presentation Native Rust (roxmltree) Yes (for embedded images) Slide extraction, image OCR, table detection
Word (Modern) .docx application/vnd.openxmlformats-officedocument.wordprocessingml.document Native Rust No Preserves formatting, extracts metadata
Word (Legacy) .doc application/msword Native Rust No Legacy Word text extraction
PowerPoint (Legacy) .ppt application/vnd.ms-powerpoint Native Rust No Legacy PowerPoint text extraction
OpenDocument Text .odt application/vnd.oasis.opendocument.text Native Rust No Full OpenDocument support
OpenDocument Spreadsheet .ods application/vnd.oasis.opendocument.spreadsheet Native Rust (calamine) No Multi-sheet support

Text & Markup

Format Extensions MIME Type Extraction Method OCR Support Special Features
Plain Text .txt text/plain Native Rust (streaming) No Line/word/character counting, memory-efficient streaming
Markdown .md, .markdown text/markdown, text/x-markdown Native Rust (streaming) No Header extraction, link detection, code block detection
HTML .html, .htm text/html, application/xhtml+xml Native Rust (html-to-markdown-rs) No Converts to Markdown, metadata extraction
XML .xml application/xml, text/xml Native Rust (quick-xml streaming) No Element counting, unique element tracking
SVG .svg image/svg+xml Native Rust (XML parser) No Treated as XML document
reStructuredText .rst text/x-rst Native (rst-parser) No Full reST syntax support
Org Mode .org text/x-org Native (org) No Emacs Org mode support
Rich Text Format .rtf application/rtf, text/rtf Native (rtf-parser) No RTF 1.x support

Structured Data

Format Extensions MIME Type Extraction Method OCR Support Special Features
JSON .json application/json, text/json Native Rust (serde_json) No Field counting, nested structure extraction
YAML .yaml application/x-yaml, text/yaml, text/x-yaml Native Rust (serde_yaml) No Multi-document support, field counting
TOML .toml application/toml, text/toml Native Rust (toml crate) No Configuration file support
CSV .csv text/csv Native Rust No Tabular data extraction
TSV .tsv text/tab-separated-values Native Rust No Tab-separated data extraction

Email

Format Extensions MIME Type Extraction Method OCR Support Special Features
EML .eml message/rfc822 Native Rust (mail-parser) No Header extraction, attachment listing, body text
MSG .msg application/vnd.ms-outlook Native Rust (mail-parser) No Outlook message support, metadata extraction

Images

All image formats support OCR when configured with ocr parameter in ExtractionConfig.

Format Extensions MIME Type Extraction Method OCR Support Special Features
PNG .png image/png Native Rust (image-rs) Yes EXIF metadata extraction
JPEG .jpg, .jpeg image/jpeg, image/jpg Native Rust (image-rs) Yes EXIF metadata extraction
WebP .webp image/webp Native Rust (image-rs) Yes Modern format support
BMP .bmp image/bmp, image/x-bmp, image/x-ms-bmp Native Rust (image-rs) Yes Uncompressed format
TIFF .tiff, .tif image/tiff, image/x-tiff Native Rust (image-rs) Yes Multi-page support
GIF .gif image/gif Native Rust (image-rs) Yes Animation frame extraction
JPEG 2000 .jp2, .jpx, .jpm, .mj2 image/jp2, image/jpx, image/jpm, image/mj2 Native Rust (image-rs) Yes Advanced JPEG format
PNM Family .pnm, .pbm, .pgm, .ppm image/x-portable-anymap, etc. Native Rust (image-rs) Yes NetPBM formats

Archives

Format Extensions MIME Type Extraction Method OCR Support Special Features
ZIP .zip application/zip, application/x-zip-compressed Native Rust (zip crate) No File listing, text content extraction
TAR .tar, .tgz application/x-tar, application/tar, application/x-gtar, application/x-ustar Native Rust (tar crate) No Unix archive support, compression detection
7-Zip .7z application/x-7z-compressed Native Rust (sevenz-rust) No High compression format support
Gzip .gz application/gzip Native Rust No Gzip compression support

Academic & Publishing (Native)

Format Extensions MIME Type Extraction Method OCR Support Special Features
LaTeX .tex, .latex application/x-latex, text/x-tex Native (manual parser) No Full LaTeX document support
EPUB .epub application/epub+zip Native (zip + roxmltree + html-to-markdown-rs) No E-book format, metadata extraction
BibTeX .bib application/x-bibtex, application/x-biblatex Native (biblatex) No Bibliography database support
Typst .typst application/x-typst Native (typst-syntax) No Modern typesetting format
Jupyter Notebook .ipynb application/x-ipynb+json Native (JSON parsing) No Code cells, markdown cells, output extraction
FictionBook - application/x-fictionbook+xml Native (fb2) No XML-based e-book format
DocBook - application/docbook+xml Native (roxmltree) No Technical documentation format
JATS - application/x-jats+xml Native (roxmltree) No Journal article XML format
OPML - application/x-opml+xml Native (roxmltree) No Outline format
RIS - application/x-research-info-systems Native (ris-parser) No Citation format
EndNote XML - application/x-endnote+xml Native (XML parser) No Reference manager format
CSL JSON - application/csl+json Native (JSON parser) No Citation Style Language JSON

Markdown Variants (Native)

Format MIME Type Extraction Method Special Features
CommonMark text/x-commonmark Native (pulldown-cmark) Standard Markdown spec
GitHub Flavored Markdown text/x-gfm Native (pulldown-cmark) GFM extensions (tables, strikethrough, etc.)
MultiMarkdown text/x-multimarkdown Native (pulldown-cmark) MMD extensions
Markdown Extra text/x-markdown-extra Native (pulldown-cmark) PHP Markdown Extra extensions

Other Formats

Format MIME Type Extraction Method Special Features
Man Pages text/x-mdoc Native (mdoc-parser) Unix manual page format
Troff text/troff Native (troff-parser) Unix document format
POD text/x-pod Native (pod-parser) Perl documentation format
DokuWiki text/x-dokuwiki Native (dokuwiki-parser) Wiki markup format

See Xberg Format Support for more details.

Output Format

Your extracted file data will be in the job output data, which can then be used in linked jobs or attached workflow nodes. The plugin unwraps Xberg's result envelope and adds the input filename. Here is an abbreviated PDF example:

{
	"filename": "document.pdf",
	"content": "123 Somewhere St, Melbourne VIC 3000\n\n(03) 1234 5678\n\nInvoice Number: #20130304\n...",
	"mimeType": "application/pdf",
	"metadata": {
		"title": "Sunny Farm Invoice Sample",
		"format": {
			"formatType": "pdf",
			"pdfVersion": "1.5",
			"producer": "Prince 16 (www.princexml.com)",
			"isEncrypted": false,
			"width": 595,
			"height": 842,
			"pageCount": 1
		},
		"outputFormat": "plain",
		"ocrUsed": false
	},
	"extractionMethod": "native",
	"tables": []
}

Additional fields depend on the input and extraction settings. Optional fields may be omitted. See Xberg's Node.js API for the full result types.

When batch mode is enabled, or when multiple files are provided, each file's extractions are pushed onto an extractions array. Example:

{
	"extractions": [
		{
			"filename": "document1.pdf",
			"content": "..."
		},
		{
			"filename": "document2.pdf",
			"content": "..."
		}
	]
}		

If you also check the "Attach Files" checkbox in the plugin parameters, this will attach all text output as files when the job completes.

Upgrading to Version 3

Version 3 uses Xberg, the successor to Kreuzberg. To use the new release, update the image tag in your installed Plugin command to ghcr.io/pixlcore/xyplug-extract:v3.0.0. Your existing extraction configuration can stay in place: the plugin silently upgrades the applicable legacy settings in the job's configuration object every time it runs. It does not rewrite the saved Plugin or event in xyOps.

The runtime conversions include:

Legacy setting Runtime conversion
ocr.language: "eng" or "eng+fra" ocr.language: ["eng"] or ["eng", "fra"]
ocr.tesseractConfig.language as a string An array of language codes
Boolean ocr.tesseractConfig.thresholdingMethod false becomes 0; true becomes 1
chunking.maxChars or chunking.chunkSize chunking.maxCharacters
chunking.maxOverlap or chunking.chunkOverlap chunking.overlap
chunking.enabled: false Removes the runtime chunking block to keep chunking disabled
chunking.sizingType, sizingModel, sizingCacheDir A chunking.sizing object with type, model, and cacheDir
Embedding selectors using modelType and value Xberg's type plus name, modelId, or llm.model
keywords.ngramRange: [1, 3] keywords.ngramRange: { "min": 1, "max": 3 }
layout.preset: "fast" or "accurate" layout.strategy: "always", using Xberg's current layout model

Explicit Xberg settings take precedence over legacy aliases. Conversions are limited to the settings above; removed engines or features without an equivalent are not recreated. Advanced OCR, layout, tokenizer, and embedding configurations may require additional models or runtime libraries. The supplied image includes English Tesseract language data for the default OCR settings.

For new configurations, use Xberg's supported format directly. The default is:

{
	"ocr": {
		"backend": "tesseract",
		"language": ["eng"],
		"tesseractConfig": {
			"psm": 3
		}
	}
}

The xyOps output wrapper is unchanged: one file returns a single extraction unless Batch Mode is enabled, and multiple files return an extractions array. Attachments, progress updates, and per-file error handling are preserved. If some input files fail, their entries contain an error and the job succeeds when at least one file succeeds. If every file fails, the job fails.

Xberg's document output can differ from Kreuzberg's. Text ordering and table detection may change, and format metadata is now nested under metadata.format. For example, workflows reading metadata.pageCount should read metadata.format.pageCount instead. Review custom settings against the Xberg configuration reference and migration guide.

Local Testing

Use Node.js 24 for local testing, matching the Docker image. Install dependencies with npm install, then run npm test for the configuration compatibility and plugin integration checks.

Use this sample JSON to pass a local file into the Plugin. Set cwd to the directory containing the input file:

{
	"xy": 1,
	"cwd": "/path/to/your/files",
	"input": {
		"files": [
			{ "filename": "document.pdf" }
		]
	}
}

Example pipe command:

node -e 'console.log(JSON.stringify({ xy: 1, cwd: process.cwd(), input: { files: [{ filename: "document.pdf" }] } }))' | node index.js

License

MIT

About

Extract structured data out of any file, using the amazing Xberg library.

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages