-
Notifications
You must be signed in to change notification settings - Fork 184
Add Redpanda Kafka-compatible hosting integration #1410
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
slang25
wants to merge
19
commits into
CommunityToolkit:main
Choose a base branch
from
slang25:slang25/redpanda-hosting-package
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 6 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
2a45fbe
Add Redpanda hosting integration
slang25 ac76e85
Add configurable broker CPU/memory options and bind mount test
slang25 74c5aaf
Add WithKafkaUI for Kafbat-based Kafka management UI
slang25 14e039d
Align example AppHost SDK with main (Aspire 13.4.3)
slang25 03a93c0
Merge branch 'main' into slang25/redpanda-hosting-package
slang25 73f6caf
Merge branch 'main' into slang25/redpanda-hosting-package
aaronpowell c895d9d
Address review feedback
slang25 197a635
Add TypeScript app host and a Redpanda consumer sample
slang25 980694d
Merge branch 'main' into slang25/redpanda-hosting-package
slang25 139bd86
Add integration tests for Redpanda Console and Kafka UI
slang25 9250897
Merge branch 'main' into slang25/redpanda-hosting-package
slang25 a2199dd
Merge branch 'main' into slang25/redpanda-hosting-package
slang25 a79fda0
Fix Redpanda broker connectivity and add consumer health check
slang25 537c70c
Add Kafka broker health check to Redpanda resource
slang25 596fb29
Merge branch 'main' into slang25/redpanda-hosting-package
slang25 77b860e
Merge branch 'main' into slang25/redpanda-hosting-package
slang25 9e4db7b
Nest Redpanda UIs under broker and cache Kafka health check
slang25 69225f3
Merge branch 'main' into slang25/redpanda-hosting-package
jmezach 798bc66
Allow multiple Redpanda resources by keeping the Kafka endpoint proxied
slang25 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
15 changes: 15 additions & 0 deletions
15
...t.Aspire.Hosting.RedPanda.AppHost/CommunityToolkit.Aspire.Hosting.RedPanda.AppHost.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| <Project Sdk="Aspire.AppHost.Sdk/13.4.3"> | ||
|
|
||
| <PropertyGroup> | ||
| <OutputType>Exe</OutputType> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <IsAspireHost>true</IsAspireHost> | ||
| <UserSecretsId>3dc32fad-1d56-45b5-bea8-b96fa74a9628</UserSecretsId> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\..\..\src\CommunityToolkit.Aspire.Hosting.RedPanda\CommunityToolkit.Aspire.Hosting.RedPanda.csproj" IsAspireProjectResource="false" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
6 changes: 6 additions & 0 deletions
6
examples/redpanda/CommunityToolkit.Aspire.Hosting.RedPanda.AppHost/Program.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| var builder = DistributedApplication.CreateBuilder(args); | ||
|
|
||
| builder.AddRedPanda("redpanda") | ||
| .WithConsole(); | ||
|
|
||
| builder.Build().Run(); |
29 changes: 29 additions & 0 deletions
29
.../redpanda/CommunityToolkit.Aspire.Hosting.RedPanda.AppHost/Properties/launchSettings.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| { | ||
| "$schema": "https://json.schemastore.org/launchsettings.json", | ||
| "profiles": { | ||
| "https": { | ||
| "commandName": "Project", | ||
| "dotnetRunMessages": true, | ||
| "launchBrowser": true, | ||
| "applicationUrl": "https://localhost:17277;http://localhost:15247", | ||
| "environmentVariables": { | ||
| "ASPNETCORE_ENVIRONMENT": "Development", | ||
| "DOTNET_ENVIRONMENT": "Development", | ||
| "DOTNET_DASHBOARD_OTLP_ENDPOINT_URL": "https://localhost:21270", | ||
| "DOTNET_RESOURCE_SERVICE_ENDPOINT_URL": "https://localhost:22001" | ||
| } | ||
| }, | ||
| "http": { | ||
| "commandName": "Project", | ||
| "dotnetRunMessages": true, | ||
| "launchBrowser": true, | ||
| "applicationUrl": "http://localhost:15247", | ||
| "environmentVariables": { | ||
| "ASPNETCORE_ENVIRONMENT": "Development", | ||
| "DOTNET_ENVIRONMENT": "Development", | ||
| "DOTNET_DASHBOARD_OTLP_ENDPOINT_URL": "http://localhost:19100", | ||
| "DOTNET_RESOURCE_SERVICE_ENDPOINT_URL": "http://localhost:20013" | ||
| } | ||
| } | ||
| } | ||
| } |
9 changes: 9 additions & 0 deletions
9
examples/redpanda/CommunityToolkit.Aspire.Hosting.RedPanda.AppHost/appsettings.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "Logging": { | ||
| "LogLevel": { | ||
| "Default": "Information", | ||
| "Microsoft.AspNetCore": "Warning", | ||
| "Aspire.Hosting.Dcp": "Warning" | ||
| } | ||
| } | ||
| } |
16 changes: 16 additions & 0 deletions
16
src/CommunityToolkit.Aspire.Hosting.RedPanda/CommunityToolkit.Aspire.Hosting.RedPanda.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <Description>An Aspire hosting package for the Redpanda Kafka-compatible streaming platform.</Description> | ||
| <AdditionalPackageTags>hosting redpanda kafka streaming messaging</AdditionalPackageTags> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="Aspire.Hosting" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <InternalsVisibleTo Include="CommunityToolkit.Aspire.Hosting.RedPanda.Tests" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| # CommunityToolkit.Aspire.Hosting.RedPanda | ||
|
|
||
| ## Overview | ||
|
|
||
| This Aspire hosting integration runs [Redpanda](https://www.redpanda.com/) in a container. Redpanda is a Kafka API compatible streaming platform, so the resource can be referenced by any Kafka client integration (for example the official `Aspire.Confluent.Kafka` client integration). The integration exposes the Kafka API, the Schema Registry, and the Admin API, and can optionally run the Redpanda Console web UI. | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Example 1: Add a Redpanda broker | ||
|
|
||
| ```csharp | ||
| var builder = DistributedApplication.CreateBuilder(args); | ||
|
|
||
| var redpanda = builder.AddRedPanda("redpanda"); | ||
|
|
||
| builder.AddProject<Projects.MyApp>("myapp") | ||
| .WithReference(redpanda) | ||
| .WaitFor(redpanda); | ||
|
|
||
| builder.Build().Run(); | ||
| ``` | ||
|
|
||
| The connection string injected into the referencing resource is the Kafka bootstrap server address in the form `host:port`. | ||
|
|
||
| ### Example 2: Pin the Kafka host port | ||
|
|
||
| ```csharp | ||
| var redpanda = builder.AddRedPanda("redpanda", port: 9092); | ||
| ``` | ||
|
|
||
| ### Example 3: Tune the broker CPU and memory | ||
|
|
||
| ```csharp | ||
| var redpanda = builder.AddRedPanda("redpanda", options => | ||
| { | ||
| options.CpuCount = 2; // Redpanda --smp, defaults to 1 | ||
| options.Memory = "2G"; // Redpanda --memory, defaults to "1G" | ||
| }); | ||
| ``` | ||
|
|
||
| ### Example 4: Persist data with a volume or bind mount | ||
|
|
||
| ```csharp | ||
| var redpanda = builder.AddRedPanda("redpanda") | ||
| .WithDataVolume(); | ||
|
|
||
| // or | ||
|
|
||
| var redpanda = builder.AddRedPanda("redpanda") | ||
| .WithDataBindMount("./redpanda-data"); | ||
| ``` | ||
|
|
||
| ### Example 5: Add the Redpanda Console web UI | ||
|
|
||
| ```csharp | ||
| var redpanda = builder.AddRedPanda("redpanda") | ||
| .WithConsole(console => console.WithHostPort(8080)); | ||
| ``` | ||
|
|
||
| The console is configured automatically to connect to the broker's Kafka API, Schema Registry, and Admin API. | ||
|
|
||
| ### Example 6: Add the Kafka UI web UI | ||
|
|
||
| ```csharp | ||
| var redpanda = builder.AddRedPanda("redpanda") | ||
| .WithKafkaUI(kafkaUi => kafkaUi.WithHostPort(9000)); | ||
| ``` | ||
|
|
||
| `WithKafkaUI` runs the same Kafka management UI (the `kafbat/kafka-ui` image) used by the official Aspire Kafka integration. It is wired automatically to the broker's Kafka API and Schema Registry, and can be used as an alternative to the Redpanda Console. | ||
|
|
||
| ## Endpoints | ||
|
|
||
| | Name | Description | | ||
| | ---------------- | -------------------------------------------- | | ||
| | `kafka` | Kafka API (host accessible) | | ||
| | `internal` | Kafka API (container-to-container) | | ||
| | `schemaregistry` | Schema Registry HTTP API | | ||
| | `admin` | Admin API HTTP (used for the health check) | | ||
|
|
||
| ## Upstream Image | ||
|
|
||
| This integration pins the `redpandadata/redpanda` image (from `docker.redpanda.com`) to a specific version tag (`v26.1.10`) rather than a floating tag, and the optional console uses `redpandadata/console` pinned to `v3.7.4`. Redpanda publishes immutable, fully-versioned tags (`vYY.M.P`); update the pinned tags to adopt newer releases. The optional Kafka UI uses `kafbat/kafka-ui` (from `docker.io`) pinned to `v1.5.0`. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.