Surface gdrive folder status in SourceCrawler CR status by gshikhar2021 · Pull Request #318 · redhat-data-and-ai/unstructured-data-controller

gshikhar2021 · 2026-08-04T14:05:14Z

gdriveStatus:
  - url: https://drive.google.com/drive/u/0/folders/1RfWZyqz7H2b8pu441rcb1hUgtPa1Kx
  - error: 'failed to get root folder metadata for 1or5z3gVLPuhZMiMnYiitZ6JQ38n5I:
      failed to get metadata for file 1or5z3gVLPuhZDMiMnYiitE5Z6JQ3n5I: googleapi:
      Error 404: File not found: 1or5z3gVLPuhZDMiMnYiitE5Z6JQ3n5I., notFound'
    url: https://drive.google.com/drive/u/0/folders/1or5z3gVLPuhZDMiMnitE5Z6JQ38n5I
  lastAppliedGeneration: 1

Summary by CodeRabbit

New Features
- Added per-folder Google Drive synchronization status, including folder URLs and accessibility information.
- Error details are now available for folders that cannot be accessed.
- Status information is retained for both successful and failed synchronization attempts.
Bug Fixes
- Synchronization now continues when some configured folders are inaccessible.
- An error is reported when all configured root folders are inaccessible.

gemini-code-assist · 2026-08-04T14:05:17Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

coderabbitai · 2026-08-04T14:05:43Z

📝 Walkthrough

Walkthrough

This PR records Google Drive root-folder failures. The controller maps failures to GDriveFolderStatus records and persists them in SourceCrawlerStatus. The CRD schema and deep-copy methods support the new status field. The e2e test now retries destination verification.

Changes

Google Drive folder status tracking

Layer / File(s)	Summary
Track failed root folder crawls `pkg/unstructured/source.go`	Adds `FailedRootFolder` and `GDriveSource.FailedRootFolders`. Records failed crawls with folder IDs and errors. Returns an error when all configured root folders are inaccessible.
Define folder status contract `api/v1alpha1/sourcecrawler_types.go`, `api/v1alpha1/zz_generated.deepcopy.go`, `config/crd/bases/operator.dataverse.redhat.com_sourcecrawlers.yaml`	Adds `GDriveFolderStatus` and `SourceCrawlerStatus.GDriveStatus`. Adds deep-copy support and the `gdriveStatus` CRD schema.
Build and persist controller status `internal/controller/sourcecrawler_controller.go`	Maps failed folders to configured folder URLs and persists `GDriveStatus` during successful and failed synchronization paths.
Retry destination verification `test/e2e/unstructured_test.go`	Polls destination listings for up to five minutes, logs listing errors and incomplete results, and reports expected and observed counts on timeout.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Reconciler
  participant GDriveSource
  participant buildGDriveStatus
  participant SourceCrawlerStatus

  Reconciler->>GDriveSource: synchronize configured root folders
  GDriveSource-->>Reconciler: return FailedRootFolders and sync result
  Reconciler->>buildGDriveStatus: map failures to folder status records
  buildGDriveStatus-->>Reconciler: return GDriveFolderStatus list
  Reconciler->>SourceCrawlerStatus: persist GDriveStatus

Possibly related PRs

redhat-data-and-ai/unstructured-data-controller#294: Extends related Google Drive failure tracking and SourceCrawler status reporting.
redhat-data-and-ai/unstructured-data-controller#304: Modifies the same failure-tracking structures, reconciliation logic, and CRD status schema.
redhat-data-and-ai/unstructured-data-controller#306: Adds related Google Drive e2e testing and controller behavior.

Suggested reviewers: concaf

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: exposing Google Drive folder status in the SourceCrawler custom resource status.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/unstructured/source.go`:
- Around line 320-322: Update the reconciliation flow around the
FailedRootFolders check so Phase 3 garbage collection is skipped whenever any
configured root folder fails, not only when all folders fail. Preserve the
existing all-failed error return, and ensure partial failures retain prior files
and permissions until a complete crawl succeeds.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Enterprise

Run ID: dc4f760e-8f86-4a58-b59b-faa384c455b1

📥 Commits

Reviewing files that changed from the base of the PR and between 7e35584 and ed840f3.

📒 Files selected for processing (5)

api/v1alpha1/sourcecrawler_types.go
api/v1alpha1/zz_generated.deepcopy.go
config/crd/bases/operator.dataverse.redhat.com_sourcecrawlers.yaml
internal/controller/sourcecrawler_controller.go
pkg/unstructured/source.go

coderabbitai · 2026-08-05T06:55:10Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@config/crd/bases/operator.dataverse.redhat.com_sourcecrawlers.yaml`:
- Around line 179-189: Bound the gdriveStatus payload produced by
buildGDriveStatus: add appropriate maxItems and maxLength constraints for folder
entries, URLs, and error strings in the CRD and source configuration, and
enforce the same limits when constructing status so existing resources cannot
exceed them. Preserve the current status shape while truncating or capping
values before writing the Kubernetes object.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Enterprise

Run ID: 5867c34e-d0bb-4433-824a-8ccb318bc272

📥 Commits

Reviewing files that changed from the base of the PR and between 56e6b34 and 41f98fe.

📒 Files selected for processing (3)

api/v1alpha1/sourcecrawler_types.go
config/crd/bases/operator.dataverse.redhat.com_sourcecrawlers.yaml
internal/controller/sourcecrawler_controller.go

🚧 Files skipped from review as they are similar to previous changes (1)

internal/controller/sourcecrawler_controller.go

coderabbitai · 2026-08-05T09:12:04Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/unstructured_test.go`:
- Around line 499-500: Update the PollUntilContextTimeout call in the polling
flow to pass the existing ctx instead of context.Background(), ensuring
assessment cancellation and deadlines stop further S3 list requests.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Enterprise

Run ID: c70227e1-81b5-498b-9c22-08055a5b081b

📥 Commits

Reviewing files that changed from the base of the PR and between 7e35584 and f3234ed.

📒 Files selected for processing (6)

api/v1alpha1/sourcecrawler_types.go
api/v1alpha1/zz_generated.deepcopy.go
config/crd/bases/operator.dataverse.redhat.com_sourcecrawlers.yaml
internal/controller/sourcecrawler_controller.go
pkg/unstructured/source.go
test/e2e/unstructured_test.go

🚧 Files skipped from review as they are similar to previous changes (5)

api/v1alpha1/zz_generated.deepcopy.go
config/crd/bases/operator.dataverse.redhat.com_sourcecrawlers.yaml
api/v1alpha1/sourcecrawler_types.go
pkg/unstructured/source.go
internal/controller/sourcecrawler_controller.go

coderabbitai Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread pkg/unstructured/source.go

gshikhar2021 force-pushed the failed-root branch from ed840f3 to 56e6b34 Compare August 5, 2026 06:54

PuneetPunamiya previously approved these changes Aug 5, 2026

View reviewed changes

concaf reviewed Aug 5, 2026

View reviewed changes

Comment thread api/v1alpha1/sourcecrawler_types.go Outdated

gshikhar2021 dismissed PuneetPunamiya’s stale review via 41f98fe August 5, 2026 08:40

coderabbitai Bot reviewed Aug 5, 2026

View reviewed changes

Comment thread config/crd/bases/operator.dataverse.redhat.com_sourcecrawlers.yaml

Shikhar Gupta added 2 commits August 5, 2026 14:20

Surface gdrive folder status in SourceCrawler CR status

f2fd4a7

Address comments

f3234ed

gshikhar2021 force-pushed the failed-root branch from 41f98fe to f3234ed Compare August 5, 2026 09:11

coderabbitai Bot reviewed Aug 5, 2026

View reviewed changes

Comment thread test/e2e/unstructured_test.go

PuneetPunamiya approved these changes Aug 5, 2026

View reviewed changes

concaf requested changes Aug 5, 2026

View reviewed changes

PuneetPunamiya enabled auto-merge (squash) August 5, 2026 09:22

PuneetPunamiya approved these changes Aug 5, 2026

View reviewed changes

concaf approved these changes Aug 5, 2026

View reviewed changes

PuneetPunamiya merged commit ae7afd3 into redhat-data-and-ai:main Aug 5, 2026
9 checks passed

Navigation Menu

Uh oh!

Surface gdrive folder status in SourceCrawler CR status - #318

Surface gdrive folder status in SourceCrawler CR status#318
PuneetPunamiya merged 2 commits into
redhat-data-and-ai:mainfrom
gshikhar2021:failed-root

gshikhar2021 commented Aug 4, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

gemini-code-assist Bot commented Aug 4, 2026

Uh oh!

coderabbitai Bot commented Aug 4, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 5, 2026

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 5, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

gshikhar2021 commented Aug 4, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Summary by CodeRabbit

Uh oh!

gemini-code-assist Bot commented Aug 4, 2026

Uh oh!

coderabbitai Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 5, 2026

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 5, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gshikhar2021 commented Aug 4, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 4, 2026 •

edited

Loading