Skip to content

rpk: add OAUTHBEARER SASL mechanism support#30169

Merged
david-yu merged 19 commits intodevfrom
rpk-oauthbearer-support
Apr 22, 2026
Merged

rpk: add OAUTHBEARER SASL mechanism support#30169
david-yu merged 19 commits intodevfrom
rpk-oauthbearer-support

Conversation

@david-yu
Copy link
Copy Markdown
Contributor

@david-yu david-yu commented Apr 15, 2026

Summary

  • Add OAUTHBEARER as a supported SASL mechanism for rpk, enabling OIDC-based authentication with Kafka brokers
  • The token is passed via --password (supports both raw token and token:<TOKEN> format)
  • OAUTHBEARER support added to Kafka client, admin API, and schema registry clients
  • Unit tests for token extraction, admin API auth, and Kafka client error paths
  • Updated -X help / -X list and profile docs to document all four SASL mechanisms

Ref: https://docs.redpanda.com/current/manage/security/authentication/#oidc
Ref: https://github.com/david-yu/redpanda-operator-mtls-oidc-listeners

Usage Examples

Obtain an OIDC token

# Example: get a token from Dex via password grant
export OIDC_TOKEN=$(curl -s -X POST "https://dex.example.com/dex/token" \
  -d "grant_type=password&username=user@example.com&password=password" \
  -d "client_id=redpanda&client_secret=redpanda-secret&scope=openid email" \
  | jq -r '.access_token')

Topic operations

# List topics
rpk topic list \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

# Create a topic
rpk topic create oidc-test \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

# Produce messages
echo "hello from OIDC client" | rpk topic produce oidc-test \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

# Consume messages
rpk topic consume oidc-test --num 1 \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

# Describe a topic
rpk topic describe oidc-test \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

# Delete a topic
rpk topic delete oidc-test \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

Consumer group operations

# List consumer groups
rpk group list \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

# Describe a consumer group
rpk group describe my-group \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

Cluster operations

# Cluster info
rpk cluster info \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

# Cluster health
rpk cluster health \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

ACL operations

# List ACLs
rpk security acl list \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

# Create an ACL
rpk security acl create --allow-principal User:user@example.com \
  --operation all --topic oidc-test \
  --brokers localhost:31094 \
  --tls-enabled \
  --tls-truststore certs/ca.crt \
  --sasl-mechanism OAUTHBEARER \
  --password "token:${OIDC_TOKEN}"

Using a profile (avoids repeating flags)

rpk profile create oidc-profile \
  --set kafka_api.brokers=localhost:31094 \
  --set kafka_api.tls.ca_file=certs/ca.crt \
  --set kafka_api.sasl.mechanism=OAUTHBEARER \
  --set kafka_api.sasl.password="token:${OIDC_TOKEN}"

# Now commands use the profile automatically
rpk topic list
rpk topic create oidc-test
rpk cluster info

Test plan

Unit tests (all passing)

  • oauthBearerToken helper — token prefix stripping, raw tokens, empty inputs, case sensitivity
  • GetAuth with OAUTHBEARER — bearer token construction, case-insensitivity, empty token error, token:-only error
  • GetAuth regression — SCRAM-SHA-256 still returns BasicAuth, no-SASL still returns NopAuth
  • NewFranzClient error paths — OAUTHBEARER with empty token, token:-only prefix, unknown mechanism

E2E tested locally on Kind + Dex OIDC provider

Tested with a local rpk build against a Kind cluster running Redpanda with Dex as the OIDC provider
(setup).
Redpanda configured with oidc_discovery_url, oidc_token_audience, oidc_principal_mapping,
and an OIDC listener with sasl_mechanisms_overrides: [OAUTHBEARER] on NodePort 31094.

# Test Result
1 rpk topic list Listed topics (_schemas, mtls-test)
2 rpk topic create oidc-rpk-test Created OK
3 rpk topic produce oidc-rpk-test Produced to partition 0, offset 0
4 rpk topic consume oidc-rpk-test --num 1 Consumed message
5 rpk topic describe oidc-rpk-test Full topic config returned
6 rpk group list Listed groups
7 rpk cluster info Cluster ID, broker, topics returned
8 rpk topic delete oidc-rpk-test Deleted OK
9 Raw token (no token: prefix) Worked
10 No token provided Error: "OAUTHBEARER requires a token"
11 Empty token: prefix Error: "OAUTHBEARER requires a token"

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v26.1.x
  • v25.3.x
  • v25.2.x

Release Notes

Features

  • Add OAUTHBEARER SASL mechanism support to rpk, enabling OIDC-based authentication for the Kafka client, admin API, and schema registry. Pass the token via --password (raw value or token:<TOKEN> format) with --sasl-mechanism OAUTHBEARER.

Generated with Claude Code

Allow rpk to authenticate with Kafka brokers using the OAUTHBEARER SASL
mechanism. This enables OIDC-based authentication workflows where users
pass an OAuth token via --sasl-password (with optional "token:" prefix).

The token is used as an OAuth bearer token for Kafka connections, admin
API requests, and schema registry access.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Apr 15, 2026

CLA assistant check
All committers have signed the CLA.

@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Test the oauthBearerToken helper (token: prefix stripping, raw tokens,
empty inputs) and GetAuth with OAUTHBEARER profiles (bearer token
construction, case-insensitivity, error paths).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
david-yu added a commit to david-yu/redpanda-operator-mtls-oidc-listeners that referenced this pull request Apr 15, 2026
Key finding: sasl_mechanisms_overrides works through the CRD bootstrap
config passthrough when using the correct Redpanda list-of-objects format:
  - listener: oidc
    sasl_mechanisms:
      - OAUTHBEARER

The previous map format ({oidc: [OAUTHBEARER]}) was the wrong format
for this Redpanda property. No chart/operator code change needed.

Updated:
- manifests/redpanda-cr.yaml: correct list-of-objects format, removed
  the "must set post-deploy via rpk" comment
- README: CRD listener config snippets, rpk OAUTHBEARER tracking PR
  (redpanda-data/redpanda#30169), corrected test results showing
  overrides work via CRD, removed workaround instructions
- scripts/oidc-test-pod.yaml: cleaned up test script

Validated end-to-end on Kind:
- sasl_mechanisms_overrides correctly applied via CRD bootstrap config
- OIDC token acquisition from Dex: PASS
- SASL/OAUTHBEARER authentication: PASS
- mTLS create/produce/consume: PASS
Comment thread src/go/rpk/pkg/config/profile_doc.go Outdated
david-yu and others added 5 commits April 15, 2026 10:07
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The sasl.mechanism help in both -X help and -X list was missing PLAIN
and OAUTHBEARER. Update to list all four supported mechanisms and
document the token: password prefix for OAUTHBEARER.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Test that NewFranzClient returns the correct errors for OAUTHBEARER
with empty token, token:-only prefix, and unknown SASL mechanisms.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@david-yu david-yu marked this pull request as ready for review April 16, 2026 18:35
@david-yu david-yu requested review from a team, kbatuigas and r-vasquez as code owners April 16, 2026 18:35
david-yu and others added 2 commits April 16, 2026 12:11
Previous build failed in ci:pandatriage:parse-results due to a
--fetch-pt-analysis argument parsing bug in the CI tool, unrelated
to this PR's changes.
@david-yu
Copy link
Copy Markdown
Contributor Author

Not exactly sure why the tests keep failing does not seem related to this PR.

Copy link
Copy Markdown
Contributor

@c-julin c-julin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, got codex to run a review and it found the debug bundle drops the auth error silently. Just a few nits otherwise

Comment thread src/go/rpk/pkg/adminapi/admin.go Outdated
Comment thread src/go/rpk/pkg/adminapi/admin.go
Comment thread src/go/rpk/pkg/kafka/client_franz.go Outdated
Comment thread src/go/rpk/pkg/schemaregistry/client.go Outdated
Comment thread src/go/rpk/pkg/kafka/client_franz_test.go Outdated
@c-julin
Copy link
Copy Markdown
Contributor

c-julin commented Apr 17, 2026

Expanding on the "debug bundle drops auth silently" note from the review body, plus a couple of items that didn't make it in:

Remote debug bundle — details

src/go/rpk/pkg/cli/debug/remotebundle/start.go:253 is unchanged by this PR:

if p.HasSASLCredentials() {
    s := p.KafkaAPI.SASL
    opts = append(opts, rpadmin.WithSCRAMAuthentication(s.User, s.Password, s.Mechanism))
}

HasSASLCredentials() requires both User != "" and Password != "". OAUTHBEARER profiles typically have no username, so this branch is skipped and the request goes out with no auth at all. On a secured cluster this fails in a confusing way. The downstream rpadmin debug-bundle request type only has SCRAM/OIDC payloads, so a bearer fallback doesn't exist today.

Options:

  • Add explicit OAUTHBEARER handling end-to-end (requires broker-side support for bearer in the debug bundle API)
  • Or reject OAUTHBEARER profiles up front from this command with a clear "not yet supported" error, so users don't get a silent failure

Missing schema registry tests

pkg/schemaregistry/client.go gained an OAUTHBEARER branch but has zero tests. adminapi and kafka both got table tests — schema registry should too.

Ordering dependency in schemaregistry/client.go

The OAUTHBEARER case at line 64 must come before the HasSASLCredentials() case at line 70. HasSASLCredentials() returns false for OAUTHBEARER profiles today (no username), so the current ordering happens to work, but if a future refactor reordered the cases or loosened that check, OAUTHBEARER tokens would get sent as BasicAuth passwords. Worth a one-line comment documenting the ordering dependency.

Move oauthBearerToken to adminapi.OAuthBearerToken and share it across
the kafka and schemaregistry clients (was previously duplicated in all
three packages).

Fix the "requires a token" error to reference the actual --password
flag (not --sasl-password, which does not exist) and the profile field.
Use errors.New where no format verbs were used.

Drop the duplicate oauthBearerToken test from client_franz_test.go;
the require-based version in adminapi/auth_test.go is canonical and
gains the case-sensitivity case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@david-yu david-yu requested a review from c-julin April 17, 2026 17:12
@david-yu
Copy link
Copy Markdown
Contributor Author

@c-julin Thanks for the review! All 5 comments addressed in 7dd8eb0:

  1. adminapi/admin.go:52 — error message now references --password (or kafka_api.sasl.password in the profile) instead of the non-existent --sasl-password.
  2. adminapi/admin.go:89oauthBearerToken is now exported as adminapi.OAuthBearerToken and reused by the kafka and schemaregistry clients (removed the two duplicate copies).
  3. kafka/client_franz.go:139 — same error-text fix; also switched fmt.Errorferrors.New since there were no format verbs.
  4. schemaregistry/client.go:67 — same error-text fix.
  5. kafka/client_franz_test.go:20 — removed the duplicate Test_oauthBearerToken that used raw t.Errorf. The require.Equal-based version in adminapi/auth_test.go is now the single canonical test, and I moved over the Token:abc case-sensitivity case so no coverage is lost.

david-yu and others added 3 commits April 17, 2026 11:07
The remote debug bundle forwards SASL credentials to the broker so rpk
running on the broker can authenticate to Kafka. The broker's admin API
currently only accepts a SCRAM-shaped payload (username, password,
mechanism), and rpk previously gated credential forwarding behind
HasSASLCredentials(), which returns false for OAUTHBEARER profiles
because they have no username.

That meant a remote debug bundle started with an OAUTHBEARER profile
would silently go out with no auth and fail in a confusing way once it
reached Kafka. Until the broker/rpadmin gain a bearer payload, reject
up front with a clear "not yet supported" error pointing users at
SCRAM.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a table test covering OAUTHBEARER in schemaregistry.NewClient: the
token-prefix path, raw token, case-insensitive mechanism match, and
the two empty-token error paths. The adminapi and kafka clients
already had this coverage; the schema registry client did not.

Also document the ordering dependency in the switch: the OAUTHBEARER
case must match before the HasSASLCredentials() case. OAUTHBEARER
profiles have no username, so HasSASLCredentials() returns false today
and the ordering is fine, but a future loosening of that predicate or
a re-ordering of the cases would route the bearer token into a
BasicAuth password.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Point future readers at redpanda#30222 which tracks the broker-side
and rpadmin-side work needed to lift the OAUTHBEARER rejection in
rpk debug remote-bundle start.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@david-yu
Copy link
Copy Markdown
Contributor Author

Fixed a few build errors and should now pass tests. Waiting on #30225 to see if it merges to enable the oidc remote debug bundle sub command.

@david-yu david-yu requested a review from c-julin April 22, 2026 17:26
@david-yu
Copy link
Copy Markdown
Contributor Author

@c-julin will go ahead merge as is once I get your approval and backport. Then re-do when #30225 lands since that will take some time to review.

@david-yu david-yu merged commit c21df75 into dev Apr 22, 2026
26 checks passed
@david-yu david-yu deleted the rpk-oauthbearer-support branch April 22, 2026 17:52
@vbotbuildovich
Copy link
Copy Markdown
Collaborator

/backport v26.1.x

@vbotbuildovich
Copy link
Copy Markdown
Collaborator

/backport v25.3.x

@vbotbuildovich
Copy link
Copy Markdown
Collaborator

Failed to create a backport PR to v26.1.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-30169-v26.1.x-343 remotes/upstream/v26.1.x
git cherry-pick -x a1a4bd271c 4b3f7896d3 8d87e83c5b 03d7388110 2d5f797eb5 d8ac8463b3 319275c233 bcf5806864 eeab01cd1c 7dd8eb0160 33b24f100f c4e9f70aaa f47f4c18f4 2fa957954e 2e13ac5baa 4dcd842991 ce7880aa84 7a003d6222 9a73bbbd0a

Workflow run logs.

@vbotbuildovich
Copy link
Copy Markdown
Collaborator

Failed to create a backport PR to v25.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-30169-v25.3.x-972 remotes/upstream/v25.3.x
git cherry-pick -x a1a4bd271c 4b3f7896d3 8d87e83c5b 03d7388110 2d5f797eb5 d8ac8463b3 319275c233 bcf5806864 eeab01cd1c 7dd8eb0160 33b24f100f c4e9f70aaa f47f4c18f4 2fa957954e 2e13ac5baa 4dcd842991 ce7880aa84 7a003d6222 9a73bbbd0a

Workflow run logs.

@david-yu
Copy link
Copy Markdown
Contributor Author

david-yu commented Apr 27, 2026

Will try to backport just this PR back to 26.1.x, 25.3.x, and 25.2.x

@david-yu
Copy link
Copy Markdown
Contributor Author

/backport v26.1.x

@vbotbuildovich
Copy link
Copy Markdown
Collaborator

Failed to create a backport PR to v26.1.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-30169-v26.1.x-897 remotes/upstream/v26.1.x
git cherry-pick -x a1a4bd271c 4b3f7896d3 8d87e83c5b 03d7388110 2d5f797eb5 d8ac8463b3 319275c233 bcf5806864 eeab01cd1c 7dd8eb0160 33b24f100f c4e9f70aaa f47f4c18f4 2fa957954e 2e13ac5baa 4dcd842991 ce7880aa84 7a003d6222 9a73bbbd0a

Workflow run logs.

david-yu added a commit that referenced this pull request Apr 27, 2026
Add OAUTHBEARER as a supported SASL mechanism in rpk, alongside the
existing SCRAM-SHA-256 and SCRAM-SHA-512 mechanisms.

- toSASLConfig and NewFranzClient now dispatch on OAUTHBEARER to set
  up kgo.SASL with the bearer token from the profile's sasl.password
- toRpadminOptions similarly calls WithOAuthBearerAuthentication for
  the admin client
- OAUTHBEARER is rejected early in remote debug bundle (follow-up
  issue referenced in the guard comment)
- Update -X help text and profile docs to list all SASL mechanisms
  including PLAIN and OAUTHBEARER
- Add unit tests for the SASL dispatch paths in adminapi, franz client,
  and schema registry client; fix $HOME-unset failures in those tests
- Regenerate BUILD files for new test files

(cherry picked from commits in PR #30169)
@david-yu
Copy link
Copy Markdown
Contributor Author

/backport v25.3.x

@david-yu
Copy link
Copy Markdown
Contributor Author

/backport v25.2.x

@vbotbuildovich
Copy link
Copy Markdown
Collaborator

Failed to create a backport PR to v25.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-30169-v25.2.x-876 remotes/upstream/v25.2.x
git cherry-pick -x a1a4bd271c 4b3f7896d3 8d87e83c5b 03d7388110 2d5f797eb5 d8ac8463b3 319275c233 bcf5806864 eeab01cd1c 7dd8eb0160 33b24f100f c4e9f70aaa f47f4c18f4 2fa957954e 2e13ac5baa 4dcd842991 ce7880aa84 7a003d6222 9a73bbbd0a

Workflow run logs.

@vbotbuildovich
Copy link
Copy Markdown
Collaborator

Failed to create a backport PR to v25.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-30169-v25.3.x-730 remotes/upstream/v25.3.x
git cherry-pick -x a1a4bd271c 4b3f7896d3 8d87e83c5b 03d7388110 2d5f797eb5 d8ac8463b3 319275c233 bcf5806864 eeab01cd1c 7dd8eb0160 33b24f100f c4e9f70aaa f47f4c18f4 2fa957954e 2e13ac5baa 4dcd842991 ce7880aa84 7a003d6222 9a73bbbd0a

Workflow run logs.

david-yu added a commit that referenced this pull request Apr 28, 2026
Add OAUTHBEARER as a supported SASL mechanism in rpk, alongside the
existing SCRAM-SHA-256 and SCRAM-SHA-512 mechanisms.

- toSASLConfig and NewFranzClient now dispatch on OAUTHBEARER to set
  up kgo.SASL with the bearer token from the profile's sasl.password
- toRpadminOptions similarly calls WithOAuthBearerAuthentication for
  the admin client
- OAUTHBEARER is rejected early in remote debug bundle (follow-up
  issue referenced in the guard comment)
- Update -X help text and profile docs to list all SASL mechanisms
  including PLAIN and OAUTHBEARER
- Add unit tests for the SASL dispatch paths in adminapi, franz client,
  and schema registry client; fix $HOME-unset failures in those tests
- Regenerate BUILD files for new test files

(cherry picked from commits in PR #30169)
david-yu added a commit that referenced this pull request Apr 28, 2026
Add OAUTHBEARER as a supported SASL mechanism in rpk, alongside the
existing SCRAM-SHA-256 and SCRAM-SHA-512 mechanisms.

- toSASLConfig and NewFranzClient now dispatch on OAUTHBEARER to set
  up kgo.SASL with the bearer token from the profile's sasl.password
- toRpadminOptions similarly calls WithOAuthBearerAuthentication for
  the admin client
- OAUTHBEARER is rejected early in remote debug bundle (follow-up
  issue referenced in the guard comment)
- Update -X help text and profile docs to list all SASL mechanisms
  including PLAIN and OAUTHBEARER
- Add unit tests for the SASL dispatch paths in adminapi, franz client,
  and schema registry client; fix $HOME-unset failures in those tests
- Regenerate BUILD files for new test files

(cherry picked from commits in PR #30169)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants