Skip unsupported file types during source crawl - #303
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
158b794 to
5c95467
Compare
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe change adds supported file-type helpers and a crawl catalog. S3 and Google Drive synchronization now records file outcomes, skips unsupported and oversized files, removes stale catalog entries, and reports skipped unsupported files during reconciliation. ChangesCrawl Catalog and Unsupported File Handling
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant S3BucketSource
participant GDriveSource
participant FileStore
S3BucketSource->>S3BucketSource: classify discovered objects
S3BucketSource->>FileStore: persist CrawlResult records
GDriveSource->>GDriveSource: filter and download Drive files
GDriveSource->>FileStore: persist CrawlResult records
S3BucketSource->>FileStore: remove stale catalog entries
GDriveSource->>FileStore: remove stale catalog entries
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
pkg/unstructured/file_types.go (1)
26-38: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider making the extension set immutable to callers.
SupportedFileExtensionsis an exportedmap[string]bool. Any package that importsunstructuredcan add or remove entries at runtime, for exampleunstructured.SupportedFileExtensions[".txt"] = true. This would change filtering behavior for allS3BucketSourceandGDriveSourceinstances without going throughIsSupportedFileType.Unexport the map and keep only
IsSupportedFileTypeandFileExtensionas the public surface, or expose a copy through an accessor function.♻️ Proposed fix
-// SupportedFileExtensions mirrors Docling defaultFromFormats, excluding image. -// Supported: docx, pptx, html, pdf, asciidoc, md, csv, xlsx. -var SupportedFileExtensions = map[string]bool{ +// supportedFileExtensions mirrors Docling defaultFromFormats, excluding image. +// Supported: docx, pptx, html, pdf, asciidoc, md, csv, xlsx. +var supportedFileExtensions = map[string]bool{ ".docx": true, ".pptx": true, ".html": true, ".htm": true, ".pdf": true, ".adoc": true, ".asciidoc": true, ".md": true, ".markdown": true, ".csv": true, ".xlsx": true, } func IsSupportedFileType(fileName string) bool { - return SupportedFileExtensions[strings.ToLower(path.Ext(fileName))] + return supportedFileExtensions[strings.ToLower(path.Ext(fileName))] +} + +// IsSupportedExtension checks a pre-computed, normalized extension (e.g. an +// extension overridden for Google-native document conversion). +func IsSupportedExtension(ext string) bool { + return supportedFileExtensions[strings.ToLower(ext)] }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/unstructured/file_types.go` around lines 26 - 38, Make the supported extension collection private instead of exposing the mutable SupportedFileExtensions map. Update IsSupportedFileType and FileExtension to use the private collection, preserving their existing public behavior; if callers require access, provide a copy-returning accessor rather than the underlying map.internal/controller/sourcecrawler_controller.go (1)
315-324: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider moving skipped-file reporting onto the
DataSourceinterface.
skippedUnsupportedFromSourcetype-switches on*unstructured.S3BucketSourceand*unstructured.GDriveSource. This works today becauseReconcileonly constructs these two types. If a futureDataSourceimplementation is added and this switch is not updated, its skipped files silently disappear from the status message with no compile-time warning.Add a method to the
DataSourceinterface inpkg/unstructured/source.goinstead, so every implementation must supply its own skipped-file list.♻️ Proposed fix
type DataSource interface { // SyncFilesToFilestore will store all files from the source to the filestore and return the list of file paths SyncFilesToFilestore(ctx context.Context, fs *filestore.FileStore) ([]RawFileMetadata, error) + // SkippedUnsupportedFiles returns file paths skipped during the last sync + // due to an unsupported extension. + SkippedUnsupportedFiles() []string }-func skippedUnsupportedFromSource(source unstructured.DataSource) []string { - switch s := source.(type) { - case *unstructured.S3BucketSource: - return s.SkippedUnsupported - case *unstructured.GDriveSource: - return s.SkippedUnsupported - default: - return nil - } -} +func skippedUnsupportedFromSource(source unstructured.DataSource) []string { + return source.SkippedUnsupportedFiles() +}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/controller/sourcecrawler_controller.go` around lines 315 - 324, Move skipped-file reporting into the DataSource interface by adding a method that returns the source’s skipped unsupported files, and implement it for S3BucketSource and GDriveSource. Update skippedUnsupportedFromSource to call that interface method directly and remove its concrete-type switch, ensuring future DataSource implementations must provide the behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@internal/controller/sourcecrawler_controller.go`:
- Around line 315-324: Move skipped-file reporting into the DataSource interface
by adding a method that returns the source’s skipped unsupported files, and
implement it for S3BucketSource and GDriveSource. Update
skippedUnsupportedFromSource to call that interface method directly and remove
its concrete-type switch, ensuring future DataSource implementations must
provide the behavior.
In `@pkg/unstructured/file_types.go`:
- Around line 26-38: Make the supported extension collection private instead of
exposing the mutable SupportedFileExtensions map. Update IsSupportedFileType and
FileExtension to use the private collection, preserving their existing public
behavior; if callers require access, provide a copy-returning accessor rather
than the underlying map.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Enterprise
Run ID: cbf8ee34-b863-44a0-b8a5-15ccce6fe5e8
📒 Files selected for processing (3)
internal/controller/sourcecrawler_controller.gopkg/unstructured/file_types.gopkg/unstructured/source.go
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/unstructured/source.go`:
- Around line 78-80: Update the catalog identifier generation in the source
discovery flow around rel and fileID so distinct S3 keys always produce
distinct, path-safe identifiers. Replace the slash-to-double-underscore mapping
with a reversible full-key encoding or collision-resistant identifier, and use
the same identifier consistently for catalogIDs and garbage-collection tracking.
- Around line 87-93: Update the reconciliation flow in
pkg/unstructured/source.go so every listed catalog write or deletion failure is
added to the synchronization error collection in addition to being logged, then
return a non-nil reconciliation error after processing all items. Apply this to
lines 87-93, 103-110, 131-136, 362-369, 409-418, 453-460, and 486-495; preserve
mutex protection for the Google Drive download-result path at lines 453-460.
- Around line 147-158: Restrict the catalog GC branch around
CrawlCatalogPath-generated entries so raw objects such as catalog/report.pdf are
not treated as catalog JSON files. Use an isolated catalog namespace or validate
the path against the exact format produced by CrawlCatalogPath before deriving
catalogID and deleting; leave raw files and their sidecars to the normal GC
path.
- Line 357: Update the discovered-file handling around the catalogIDs assignment
to retain the shortcut target’s FileID even when the target is inaccessible, and
persist an error catalog result for that failed file. Ensure the failed entry is
included in catalogIDs so cleanup does not remove its previous catalog record.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Enterprise
Run ID: 59f6e081-21c6-4c99-92cb-16197305cc1f
📒 Files selected for processing (2)
pkg/unstructured/crawl_result.gopkg/unstructured/source.go
|
Sorry, just to clarify why is .txt, .doc not supported for example? We have done work before to support these files inside Docling. @maharora has done work before in Analyze to support these formats inside Docling. There is already code written before to support this. Can we not take that please? |
Yeah sure will update the patch and include the changes, thanks for the review 🤗 |
b2bff5c to
0d54c2f
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/unstructured/crawl_result.go`:
- Line 34: Update the CrawlMetadataDir constant and the related catalog-entry
path construction to use the catalog directory under the crawl output path,
producing stages/<crawl>/catalog/<fileID>.json so reconciliation and
stale-entry cleanup resolve the same location.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Enterprise
Run ID: 89af6d8c-9505-4856-9cd9-6b969e52b0d8
📒 Files selected for processing (5)
internal/controller/sourcecrawler_controller.gopkg/unstructured/crawl_result.gopkg/unstructured/file_types.gopkg/unstructured/source.gotest/e2e/unstructured_test.go
🚧 Files skipped from review as they are similar to previous changes (2)
- internal/controller/sourcecrawler_controller.go
- pkg/unstructured/source.go
0d54c2f to
352eaaa
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
pkg/unstructured/source.go (2)
481-496: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winRemove stale content when a file becomes oversized.
Line 481 adds the file to
currentFilesbefore download. IfstoreFilelater returnserrFileExceedsMaxSize, garbage collection retains the previous raw file, metadata, and permissions. The catalog reportsskipped, but downstream processing can still use stale content.Remove the file ID from
currentFilesundermuwhen this sentinel occurs.Proposed fix
if errors.Is(err, errFileExceedsMaxSize) { + mu.Lock() + delete(currentFiles, record.FileID) + mu.Unlock() logger.Info("WARNING: skipping file exceeding max file size limit", "fileID", record.FileID, "fileName", record.FileName) status, reason = CrawlStatusSkipped, err.Error() }Also applies to: 525-526, 590-593
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/unstructured/source.go` around lines 481 - 496, In the download error handling around storeFile, remove record.FileID from currentFiles while holding mu whenever the error matches errFileExceedsMaxSize. Apply the same cleanup to the corresponding oversized-file branches at the other reported locations, while preserving the existing skipped status and reason handling.
328-331: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winAbort garbage collection after a folder crawl failure.
filterCrawlRecordslogsr.errand removes that folder fromcurrentFilesandcatalogIDs. Reconciliation then runsgarbageCollect, which deletes the stored files, permissions, and catalog entries for the failed folder. It finally reports success.Return the crawl errors to
SyncFilesToFilestore. If any folder crawl fails, return an error before garbage collection.Also applies to: 426-436, 525-526
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/unstructured/source.go` around lines 328 - 331, Update filterCrawlRecords to propagate folder crawl failures instead of only logging and removing the failed folder’s records. Have SyncFilesToFilestore detect and return the crawl error before invoking garbageCollect, preserving the failed folder’s stored files, permissions, and catalog entries and avoiding a successful reconciliation result.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@pkg/unstructured/source.go`:
- Around line 481-496: In the download error handling around storeFile, remove
record.FileID from currentFiles while holding mu whenever the error matches
errFileExceedsMaxSize. Apply the same cleanup to the corresponding
oversized-file branches at the other reported locations, while preserving the
existing skipped status and reason handling.
- Around line 328-331: Update filterCrawlRecords to propagate folder crawl
failures instead of only logging and removing the failed folder’s records. Have
SyncFilesToFilestore detect and return the crawl error before invoking
garbageCollect, preserving the failed folder’s stored files, permissions, and
catalog entries and avoiding a successful reconciliation result.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Enterprise
Run ID: ec26cc73-46c6-40db-97c4-3ffe61feb2b0
📒 Files selected for processing (5)
internal/controller/sourcecrawler_controller.gopkg/unstructured/crawl_result.gopkg/unstructured/file_types.gopkg/unstructured/source.gotest/e2e/unstructured_test.go
🚧 Files skipped from review as they are similar to previous changes (2)
- pkg/unstructured/file_types.go
- internal/controller/sourcecrawler_controller.go
352eaaa to
b79544f
Compare
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/unstructured/source.go (1)
481-496: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winRemove stale content when a Google Drive file becomes oversized.
currentFilesincludes the file beforestoreFilechecks its downloaded size. If an existing file later grows beyond 128 MB,storeFilereturns the sentinel without replacing the old content. Garbage collection then preserves that old content because its file ID remains incurrentFiles.Remove the raw file and metadata for this skipped result, or exclude the file ID from
currentFilesso garbage collection removes them.Also applies to: 525-526
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/unstructured/source.go` around lines 481 - 496, Update the oversized-file branch in the dlGroup callback around storeFile so a CrawlStatusSkipped result for errFileExceedsMaxSize does not remain represented in currentFiles. Remove the existing raw file and associated metadata, or exclude record.FileID from currentFiles before garbage collection, while preserving the current warning and skipped status behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/unstructured/source.go`:
- Around line 319-322: Update GDriveSource.filterCrawlRecords to return the
encountered folder crawl errors alongside the filtered records and inventory,
rather than only logging them. Propagate those errors to the caller at the
reconciliation logic around line 526, and when any folder crawl fails, return a
reconciliation error before garbage collection so missing records are not
treated as deleted.
- Around line 84-94: The current S3 size check relies on listing metadata;
enforce the limit in the object-read flow instead. Update the GetObject/read
handling to cap reads at maxFileSize+1, detect content exceeding maxFileSize,
and return errFileExceedsMaxSize before invoking fs.Store; retain the existing
crawl-result handling for skipped oversized files.
In `@test/e2e/unstructured_test.go`:
- Around line 493-494: Update the unstructured destination test around
foundCount to derive expected files from supported, non-oversized inputs, then
assert each expected file has a destination result instead of only checking that
foundCount is nonzero. Also validate skipped files using their crawl catalog
entries, reusing the outcome data recorded by the production source flow.
- Around line 503-509: Update the hash-test cleanup in the surrounding
unstructured test to delete the bucket entry keyed by pdflatex-4-pages.pdf
instead of removing filesInBucket[0]. Preserve pdflatex-outline.pdf so the later
upload exercises a new file without overwriting the wrong fixture.
---
Outside diff comments:
In `@pkg/unstructured/source.go`:
- Around line 481-496: Update the oversized-file branch in the dlGroup callback
around storeFile so a CrawlStatusSkipped result for errFileExceedsMaxSize does
not remain represented in currentFiles. Remove the existing raw file and
associated metadata, or exclude record.FileID from currentFiles before garbage
collection, while preserving the current warning and skipped status behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Enterprise
Run ID: a2d5a0a9-4db5-47d7-ada5-f8aeaf32c3b6
📒 Files selected for processing (5)
internal/controller/sourcecrawler_controller.gopkg/unstructured/crawl_result.gopkg/unstructured/file_types.gopkg/unstructured/source.gotest/e2e/unstructured_test.go
🚧 Files skipped from review as they are similar to previous changes (2)
- pkg/unstructured/file_types.go
- internal/controller/sourcecrawler_controller.go
| if object.Size != nil && *object.Size > maxFileSize { | ||
| logger.Info("WARNING: skipping file exceeding max file size limit", | ||
| "key", *object.Key, "sizeMB", *object.Size/(1<<20)) | ||
| if err := storeCrawlResult(ctx, fs, outputPrefix, CrawlResult{ | ||
| FileID: fileID, FileName: baseName, SourcePath: sourcePath, | ||
| Status: CrawlStatusSkipped, Reason: "file exceeds max size limit (128 MB)", | ||
| SourceType: "s3", | ||
| }); err != nil { | ||
| logger.Error(err, "failed to store crawl catalog result", "fileID", fileID) | ||
| } | ||
| continue |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Enforce the S3 size limit while reading the object.
The listing size is not a reliable enforcement point. The object can change between ListObjectsInPrefix and GetObject, and object.Size can be absent.
Limit the read to maxFileSize+1. Return errFileExceedsMaxSize before fs.Store if the downloaded content exceeds the limit.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/unstructured/source.go` around lines 84 - 94, The current S3 size check
relies on listing metadata; enforce the limit in the object-read flow instead.
Update the GetObject/read handling to cap reads at maxFileSize+1, detect content
exceeding maxFileSize, and return errFileExceedsMaxSize before invoking
fs.Store; retain the existing crawl-result handling for skipped oversized files.
| func (g *GDriveSource) filterCrawlRecords( | ||
| ctx context.Context, fs *filestore.FileStore, | ||
| results []folderResult, | ||
| ) ([]gdrive.CrawlRecord, map[string]bool) { |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Do not run garbage collection after a partial folder crawl.
filterCrawlRecords logs folderResult.err and returns a partial inventory. Line 526 then treats missing records as deleted source files and removes their stored files, permissions, and catalog entries.
Return the folder crawl errors from filterCrawlRecords. If any folder crawl fails, return a reconciliation error and skip garbage collection.
Also applies to: 436-436, 525-526
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/unstructured/source.go` around lines 319 - 322, Update
GDriveSource.filterCrawlRecords to return the encountered folder crawl errors
alongside the filtered records and inventory, rather than only logging them.
Propagate those errors to the caller at the reconciliation logic around line
526, and when any folder crawl fails, return a reconciliation error before
garbage collection so missing records are not treated as deleted.
| file2Content, err := os.ReadFile(filepath.Join(unstructuredFilesDirectory, "pdflatex-4-pages.pdf")) | ||
| if err != nil { | ||
| t.Fatalf("read file2 test PDF: %v", err) | ||
| } | ||
|
|
||
| file1 := "pdflatex-4-pages.pdf" | ||
| hashTestFile2Key := fmt.Sprintf("%s/pdflatex-outline.pdf", dataPipelineCRName) | ||
| file1 := "pdflatex-outline.pdf" | ||
| hashTestFile2Key := fmt.Sprintf("%s/pdflatex-4-pages.pdf", dataPipelineCRName) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
mapfile -t names < <(
fd --type f --max-depth 1 . test/resources/unstructured/unstructured-files |
sed 's#^.*/##' |
sort
)
printf '%s\n' "${names[@]}"
[[ "${names[0]:-}" == "pdflatex-4-pages.pdf" ]] || {
echo "The first fixture is not pdflatex-4-pages.pdf" >&2
exit 1
}
printf '%s\n' "${names[@]}" | grep -Fxq "pdflatex-outline.pdf"Repository: redhat-data-and-ai/unstructured-data-controller
Length of output: 237
🏁 Script executed:
set -euo pipefail
printf '%s\n' '--- fixture files ---'
python3 - <<'PY'
from pathlib import Path
root = Path("test/resources/unstructured/unstructured-files")
for p in sorted(root.iterdir(), key=lambda p: p.name):
if p.is_file():
print(p.name)
PY
printf '%s\n' '--- test context ---'
sed -n '450,590p' test/e2e/unstructured_test.goRepository: redhat-data-and-ai/unstructured-data-controller
Length of output: 5298
🏁 Script executed:
set -euo pipefail
printf '%s\n' '--- filesInBucket references ---'
rg -n -C 8 'filesInBucket|unstructuredFilesDirectory|PutObject' test/e2e/unstructured_test.go
printf '%s\n' '--- fixture construction and deletion context ---'
sed -n '250,455p' test/e2e/unstructured_test.goRepository: redhat-data-and-ai/unstructured-data-controller
Length of output: 14808
Delete the hash-test file explicitly.
The fixture set includes minimal-document.pdf, pdflatex-4-pages.pdf, and pdflatex-outline.pdf. Deleting filesInBucket[0] removes minimal-document.pdf, so the later upload overwrites an existing file instead of testing a new file. Delete pdflatex-4-pages.pdf by key and retain pdflatex-outline.pdf.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/e2e/unstructured_test.go` around lines 503 - 509, Update the hash-test
cleanup in the surrounding unstructured test to delete the bucket entry keyed by
pdflatex-4-pages.pdf instead of removing filesInBucket[0]. Preserve
pdflatex-outline.pdf so the later upload exercises a new file without
overwriting the wrong fixture.
| return ctrl.Result{}, r.handleError(ctx, sourceCrawlerCR, fmt.Errorf("unsupported source type: %s", sourceCrawlerConfig.Type)) | ||
| } | ||
|
|
||
| storedFiles, err := source.SyncFilesToFilestore(ctx, r.fileStore) |
There was a problem hiding this comment.
one approach is that we give this information to the source from source crawler because this is a property of the source crawler and not of the individual source like Google Drive or S3
Only sync Docling-compatible document extensions from S3/GDrive and surface skipped files in the SourceCrawler Ready status message.
Persist per-file crawl outcomes under stages/<crawl>/catalog/ so success, skipped, and error results can be synced and queried later.
b79544f to
6a88cf1
Compare
4 participants
Summary
from_formats, excludingimage)Truewhen skips occur; real store/API failures still fail reconcilestages/<crawl>/catalog/<file_id>.json(upserted each reconcile; folders are not catalogued)Supported file types
.pdf.md/.markdown.docx.pptx.html/.htm.csv.xlsx.adoc/.asciidoc.txt.doc(legacy Word)Not supported yet (skipped)
.png,.jpg,.jpeg,.gif,.bmp,.tif,.tiff,.webp) — Doclingimageformat intentionally excluded for now.ppt/.xls(legacy Office).odt/.ods/.odp.rtf.zipand other archivesGoogle Workspace native docs (
application/vnd.google-apps.*) are exported as PDF and treated as supported.Example SourceCrawler status
Crawl catalog output
Path:
Successful file:
{ "file_id": "1Fen8XcdmzOVM8pY3wG8b-rTc-llm39z0", "file_name": "Data and AI Sprint Review ... Notes by Gemini.pdf", "media_type": "application/pdf", "extension": "pdf", "status": "successful", "source_type": "googleDrive", "crawled_at": "2026-08-03T06:14:32Z" }Skipped unsupported type:
{ "file_id": "...", "file_name": "notes.txt", "extension": "txt", "status": "skipped", "reason": "unsupported file type \".txt\"", "source_type": "s3", "crawled_at": "2026-08-03T06:14:32Z" }statusreasonsuccessfulskippedunsupported file type ".txt"skippedfile exceeds max size limit (128 MB)errorSummary by CodeRabbit
New Features
Bug Fixes