Skip to content

feat(reconciler): add per-record auto-apply policy hook (OBS-2988)#539

Open
unlisted wants to merge 4 commits into
developfrom
OBS-2988-cf-auto-apply
Open

feat(reconciler): add per-record auto-apply policy hook (OBS-2988)#539
unlisted wants to merge 4 commits into
developfrom
OBS-2988-cf-auto-apply

Conversation

@unlisted

Copy link
Copy Markdown

Summary

  • Add AutoApplyPolicy type and WithAutoApplyPolicy(...) functional option on IngestionLogProcessor. Attaches a per-record predicate that fast-paths a planned change set past the OPEN review queue, layered on top of the existing tenant-level AutoApplyChangesets config — widens auto-apply for matches, never restricts.
  • Add ApplyChangeSetForLog to IngestionProcessorOps. OSS impl delegates to BulkPlanApply with a single-item batch (re-plans before applying; one wasted round-trip per matched record). Acceptable for current volume — a dedicated /apply endpoint on the NetBox plugin can replace the re-plan later.
  • Dispatch in processBatch: after BulkPlan, items with a non-nil change set are passed to the policy; matches go through ApplyChangeSetForLog, non-matches stay in OPEN.

This PR ships only the hook. No policy is attached at the OSS call site (cmd/reconciler/main.go is unchanged) — the concrete predicate and wiring are the diode-pro side of OBS-2988 and will follow as a separate PR:

  • ShouldAutoApply predicate matching pure-create extras.customfield change sets
  • WithAutoApplyPolicy(ShouldAutoApply) attached at the diode-pro NewIngestionLogProcessor call site
  • Carries a replace directive against this branch while iterating; that'll be removed and the require pin bumped once this lands

Relates to OBS-2988.

Test plan

  • go build ./... clean
  • go test ./reconciler/... green — including new ingestion_log_processor_autoapply_test.go covering:
    • policy match → ApplyChangeSetForLog called + apply-success metric recorded
    • policy no-match → ApplyChangeSetForLog NOT called, row left in OPEN

@github-actions

github-actions Bot commented May 26, 2026

Copy link
Copy Markdown

Vulnerability Scan: Passed — diode-auth

Image: diode-auth:scan

Source Library CVE Severity Installed Fixed Title
usr/bin/hydra github.com/docker/docker CVE-2026-34040 🟠 HIGH v28.3.3+incompatible 29.3.1 Moby: Moby: Authorization bypass vulnerability
usr/bin/hydra github.com/docker/docker CVE-2026-33997 🟡 MEDIUM v28.3.3+incompatible 29.3.1 moby: docker: github.com/moby/moby: Moby: Privilege validation bypass during plu
usr/bin/hydra github.com/go-jose/go-jose/v3 CVE-2026-34986 🟠 HIGH v3.0.4 3.0.5 github.com/go-jose/go-jose/v3: github.com/go-jose/go-jose/v4: Go JOSE: Denial of
usr/bin/hydra github.com/jackc/pgx/v5 CVE-2026-33816 🔴 CRITICAL v5.7.5 5.9.0 github.com/jackc/pgx/v5: github.com/jackc/pgx: Memory-safety vulnerability
usr/bin/hydra github.com/jackc/pgx/v5 CVE-2026-41889 ⚪ LOW v5.7.5 5.9.2 github.com/jackc/pgx: golang: pgx: SQL injection via specific SQL query conditio
usr/bin/hydra go.opentelemetry.io/otel CVE-2026-29181 🟠 HIGH v1.40.0 1.41.0 github.com/open-telemetry/opentelemetry-go: OpenTelemetry-Go: Denial of Service
usr/bin/hydra go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp CVE-2026-39882 🟡 MEDIUM v1.37.0 1.43.0 OpenTelemetry-Go is the Go implementation of OpenTelemetry. Prior to 1 ...
usr/bin/hydra go.opentelemetry.io/otel/sdk CVE-2026-39883 🟠 HIGH v1.40.0 1.43.0 opentelemetry-go: BSD kenv command not using absolute path enables PATH hijackin
usr/bin/hydra stdlib CVE-2026-25679 🟠 HIGH v1.26.0 1.25.8, 1.26.1 net/url: Incorrect parsing of IPv6 host literals in net/url
usr/bin/hydra stdlib CVE-2026-27137 🟠 HIGH v1.26.0 1.26.1 crypto/x509: Incorrect enforcement of email constraints in crypto/x509
usr/bin/hydra stdlib CVE-2026-32280 🟠 HIGH v1.26.0 1.25.9, 1.26.2 crypto/x509: crypto/tls: golang: Go: Denial of Service vulnerability in certific
usr/bin/hydra stdlib CVE-2026-32281 🟠 HIGH v1.26.0 1.25.9, 1.26.2 crypto/x509: golang: Go crypto/x509: Denial of Service via inefficient certifica
usr/bin/hydra stdlib CVE-2026-32283 🟠 HIGH v1.26.0 1.25.9, 1.26.2 crypto/tls: golang: Go crypto/tls: Denial of Service via multiple TLS 1.3 key up
usr/bin/hydra stdlib CVE-2026-33810 🟠 HIGH v1.26.0 1.26.2 crypto/x509: golang: Go crypto/x509: Certificate validation bypass due to incorr
usr/bin/hydra stdlib CVE-2026-33811 🟠 HIGH v1.26.0 1.25.10, 1.26.3 net: golang: Go net package: Denial of Service via long CNAME response in Lookup
usr/bin/hydra stdlib CVE-2026-33814 🟠 HIGH v1.26.0 1.25.10, 1.26.3 When processing HTTP/2 SETTINGS frames, transport will enter an infini ...
usr/bin/hydra stdlib CVE-2026-39820 🟠 HIGH v1.26.0 1.25.10, 1.26.3 Well-crafted inputs reaching ParseAddress, ParseAddressList, and Parse ...
usr/bin/hydra stdlib CVE-2026-39823 🟠 HIGH v1.26.0 1.25.10, 1.26.3 CVE-2026-27142 fixed a vulnerability in which URLs were not correctly ...
usr/bin/hydra stdlib CVE-2026-39825 🟠 HIGH v1.26.0 1.25.10, 1.26.3 ReverseProxy can forward queries containing parameters not visible to ...
usr/bin/hydra stdlib CVE-2026-39826 🟠 HIGH v1.26.0 1.25.10, 1.26.3 If a trusted template author were to write a <script> tag containing a ...
usr/bin/hydra stdlib CVE-2026-39836 🟠 HIGH v1.26.0 1.25.10, 1.26.3 ELSA-2026-22112: go-toolset:ol8 security update (IMPORTANT)
usr/bin/hydra stdlib CVE-2026-42499 🟠 HIGH v1.26.0 1.25.10, 1.26.3 Pathological inputs could cause DoS through consumePhrase when parsing ...
usr/bin/hydra stdlib CVE-2026-42504 🟠 HIGH v1.26.0 1.25.11, 1.26.4 Decoding a maliciously-crafted MIME header containing many invalid enc ...
usr/bin/hydra stdlib CVE-2026-27142 🟡 MEDIUM v1.26.0 1.25.8, 1.26.1 html/template: URLs in meta content attribute actions are not escaped in html/te
usr/bin/hydra stdlib CVE-2026-27145 🟡 MEDIUM v1.26.0 1.25.11, 1.26.4 *x509.Certificate).VerifyHostname previously called matchHostnames in ...
usr/bin/hydra stdlib CVE-2026-32282 🟡 MEDIUM v1.26.0 1.25.9, 1.26.2 golang: internal/syscall/unix: Root.Chmod can follow symlinks out of the root
usr/bin/hydra stdlib CVE-2026-32288 🟡 MEDIUM v1.26.0 1.25.9, 1.26.2 archive/tar: golang: Go's archive/tar package: Denial of Service via maliciously
usr/bin/hydra stdlib CVE-2026-32289 🟡 MEDIUM v1.26.0 1.25.9, 1.26.2 html/template: golang: html/template: Cross-Site Scripting (XSS) via improper co
usr/bin/hydra stdlib CVE-2026-42507 🟡 MEDIUM v1.26.0 1.25.11, 1.26.4 When returning errors, functions in the net/textproto package would in ...
usr/bin/hydra stdlib CVE-2026-27138 ⚪ LOW v1.26.0 1.26.1 crypto/x509: Panic in name constraint checking for malformed certificates in cry
usr/bin/hydra stdlib CVE-2026-27139 ⚪ LOW v1.26.0 1.25.8, 1.26.1 os: FileInfo can escape from a Root in golang os module

Commit: f114219

@github-actions

github-actions Bot commented May 26, 2026

Copy link
Copy Markdown

Vulnerability Scan: Passed — diode-ingester

Image: diode-ingester:scan

No vulnerabilities found.

Commit: f114219

@github-actions

github-actions Bot commented May 26, 2026

Copy link
Copy Markdown

Vulnerability Scan: Passed — diode-reconciler

Image: diode-reconciler:scan

No vulnerabilities found.

Commit: f114219

@github-actions

github-actions Bot commented May 26, 2026

Copy link
Copy Markdown

Go test coverage

STATUS ELAPSED PACKAGE COVER PASS FAIL SKIP
🟢 PASS 1.48s github.com/netboxlabs/diode/diode-server/auth 44.7% 42 0 0
🟢 PASS 0.93s github.com/netboxlabs/diode/diode-server/auth/cli 0.0% 0 0 0
🟢 PASS 1.02s github.com/netboxlabs/diode/diode-server/authutil 82.8% 5 0 0
🟢 PASS 0.17s github.com/netboxlabs/diode/diode-server/dbstore/postgres 0.0% 0 0 0
🟢 PASS 1.12s github.com/netboxlabs/diode/diode-server/entityhash 79.2% 13 0 0
🟢 PASS 1.11s github.com/netboxlabs/diode/diode-server/entitymatcher 82.8% 97 0 0
🟢 PASS 0.09s github.com/netboxlabs/diode/diode-server/errors 0.0% 0 0 0
🟢 PASS 1.18s github.com/netboxlabs/diode/diode-server/graph 52.0% 81 0 0
🟢 PASS 1.02s github.com/netboxlabs/diode/diode-server/grpckeepalive 100.0% 1 0 0
🟢 PASS 1.43s github.com/netboxlabs/diode/diode-server/ingester 85.4% 66 0 0
🟢 PASS 1.12s github.com/netboxlabs/diode/diode-server/matching 94.1% 66 0 0
🟢 PASS 1.07s github.com/netboxlabs/diode/diode-server/migrator 70.4% 4 0 0
🟢 PASS 3.12s github.com/netboxlabs/diode/diode-server/netboxdiodeplugin 45.4% 23 0 0
🟢 PASS 0.16s github.com/netboxlabs/diode/diode-server/pprof 0.0% 0 0 0
🟢 PASS 5.10s github.com/netboxlabs/diode/diode-server/reconciler 62.5% 94 0 0
🟢 PASS 0.11s github.com/netboxlabs/diode/diode-server/reconciler/changeset 0.0% 0 0 0
🟢 PASS 1.06s github.com/netboxlabs/diode/diode-server/reconciler/differ 49.3% 23 0 0
🟢 PASS 1.02s github.com/netboxlabs/diode/diode-server/server 85.7% 14 0 0
🟢 PASS 1.01s github.com/netboxlabs/diode/diode-server/strcase 100.0% 24 0 0
🟢 PASS 1.02s github.com/netboxlabs/diode/diode-server/telemetry 28.0% 26 0 0
🟢 PASS 1.02s github.com/netboxlabs/diode/diode-server/telemetry/otel 90.2% 25 0 0
🟢 PASS 0.10s github.com/netboxlabs/diode/diode-server/tls 0.0% 0 0 0
🟢 PASS 1.01s github.com/netboxlabs/diode/diode-server/version 100.0% 2 0 0

Total coverage: 52.7%

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a per-record auto-apply hook to the reconciler’s plan-only ingestion log processing flow, allowing specific planned change sets to bypass the OPEN review queue and be applied immediately (without changing the existing tenant-level AUTO_APPLY_CHANGESETS processor selection).

Changes:

  • Add AutoApplyPolicy + WithAutoApplyPolicy(...) functional option to IngestionLogProcessor, and invoke it after BulkPlan.
  • Extend IngestionProcessorOps with ApplyChangeSetForLog, with an OSS implementation delegating to BulkPlanApply for single items.
  • Add tests covering the policy match/no-match paths and ensuring apply + metrics behavior.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
diode-server/reconciler/ops.go Adds ApplyChangeSetForLog delegating to BulkPlanApply for single-item apply.
diode-server/reconciler/mocks/ingestionprocessorops.go Updates mock ops interface to include ApplyChangeSetForLog.
diode-server/reconciler/ingestion_processor.go Extends IngestionProcessorOps interface with ApplyChangeSetForLog.
diode-server/reconciler/ingestion_log_processor.go Adds AutoApplyPolicy option + invokes apply path on policy match.
diode-server/reconciler/ingestion_log_processor_autoapply_test.go Adds unit tests verifying apply invocation + metric recording behavior.
Files not reviewed (1)
  • diode-server/reconciler/mocks/ingestionprocessorops.go: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diode-server/reconciler/ops.go Outdated
Comment thread diode-server/reconciler/ops.go Outdated
Comment thread diode-server/reconciler/ingestion_processor.go Outdated
Comment thread diode-server/reconciler/ingestion_log_processor.go Outdated
Comment thread diode-server/reconciler/ingestion_log_processor.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f86d4cd0e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread diode-server/reconciler/ops.go Outdated

@mfiedorowicz mfiedorowicz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the auto-apply hook. Structure is clean — AutoApplyPolicy + WithAutoApplyPolicy functional option are idiomatic, and "ship the hook only, wire the predicate on the diode-pro side" is a good way to land it. One main change plus a safety caveat. (Overlaps the Copilot/Codex threads; consolidating into a single recommendation.)

1. Drop ApplyChangeSetForLog entirely; call BulkPlanApply with the matched item(s) in processBatch. The method is a thin pass-through to BulkPlanApply([]{item}) that collapses the result into error/nil — and that collapse is what produces the metric bug below (a NO_CHANGES outcome returns nil and gets counted as a successful apply of N changes). AutoApplyProcessor.processBatch already consumes BulkPlanApply results directly with the right PlanErr / ApplyErr / changes > 0 switch — the plan-only processor can do the same. That removes the misleading name, the ignored *changeset.ChangeSet param, the extra IngestionProcessorOps method, and its mock, and fixes the metric for free. Worth extracting the shared result→metrics handling into one helper both processors call. (Simplest: BulkPlanApply([]{item}) per matched record; even better: collect the matched items and issue one BulkPlanApply after the loop.) A dedicated single-item method earns its place only once a real apply-only endpoint (no re-plan) exists.

2. The re-plan divergence is a safety caveat, not just perf. The policy is evaluated against the BulkPlan change set, but the apply re-plans and applies a possibly-different set. For an auto-apply gate, a divergent re-plan can apply something the policy never approved. Either document this as an accepted interim limitation or close it with the deferred apply-only endpoint (there's no interception point while BulkPlanApply does plan+apply in one server call).

Nit: the tests sync with time.Sleep(200ms) after go p.Start — flaky under load; a channel fired from a mock would be deterministic.

Comment thread diode-server/reconciler/ops.go Outdated
Comment thread diode-server/reconciler/ingestion_log_processor.go Outdated
unlisted added a commit that referenced this pull request Jun 3, 2026
@unlisted unlisted requested a review from Copilot June 3, 2026 16:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment on lines +291 to +294
if p.autoApplyPolicy != nil && result.ChangeSet != nil &&
p.autoApplyPolicy(item.IngestionLog, result.ChangeSet) {
applyResults := p.ops.BulkPlanApply(ctx, []ops.QueuedIngestionLog{item}, branchID)
if len(applyResults) == 0 {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — this is valid. The consequences are housekeeping rather than correctness: the ingestion log state is overwritten to its terminal value by the second persist, and readers use latest-wins (ORDER BY cs.id DESC LIMIT 1), so the superseded plan change_set is shadowed — it just occupies a retention slot. This can't be resolved from the caller side, since the policy needs the plan output and BulkPlan persists internally; properly fixing it (along with the re-plan/policy-divergence threads) requires a dedicated apply-only netbox-plugin endpoint that applies the already-persisted change set without re-planning.

@unlisted unlisted left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mfiedorowicz addressed your review across df449d2, 6d740b2, 7c07bdd:

1. Drop ApplyChangeSetForLog — done in df449d2: method, interface entry, and mock removed; processBatch consumes BulkPlanApplyResult directly with the PlanErr / ApplyErr / changes > 0 switch, which also fixes the NO_CHANGES metric miscount. 6d740b2 then takes your "even better" variant: matched items are collected during the plan loop and applied with a single BulkPlanApply call after it — one round-trip per batch instead of one per matched record.

2. Re-plan divergence — documented as an accepted interim limitation at the apply call site in 6d740b2, to be closed properly by the deferred apply-only endpoint.

Nit (test sync)7c07bdd drops the time.Sleep(200ms) in favor of channels closed from the final expected mock call in each flow (RecordChangeSetApply / RecordChangeSetCreate); verified stable with -count=5 -race.

@unlisted unlisted requested a review from mfiedorowicz June 3, 2026 19:29
@unlisted unlisted force-pushed the OBS-2988-cf-auto-apply branch from 7c07bdd to 39425ca Compare June 8, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants