-
Notifications
You must be signed in to change notification settings - Fork 131
Add a cassandra-backed storage option #246
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
shs96c
wants to merge
6
commits into
buildbarn:main
Choose a base branch
from
shs96c:cassandra
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
f453e1f
Add a cassandra-backed storage option
shs96c 47b945a
Also suggest timeouts for other tables
shs96c 2698fe1
Make sure the test compiles
shs96c 9a2f628
Respond to review comments
shs96c d91fca0
Use a non-blocking put on a channel to avoid having too many go routines
shs96c bb26592
Use a semaphore rather than an limit count for the errgroup
shs96c File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| load("@rules_go//go:def.bzl", "go_library", "go_test") | ||
|
|
||
| go_library( | ||
| name = "cassandra", | ||
| srcs = [ | ||
| "cassandra_blob_access.go", | ||
| "format.go", | ||
| ], | ||
| importpath = "github.com/buildbarn/bb-storage/pkg/blobstore/cassandra", | ||
| visibility = ["//visibility:public"], | ||
| deps = [ | ||
| "//pkg/blobstore", | ||
| "//pkg/blobstore/buffer", | ||
| "//pkg/blobstore/slicing", | ||
| "//pkg/capabilities", | ||
| "//pkg/digest", | ||
| "//pkg/util", | ||
| "@bazel_remote_apis//build/bazel/remote/execution/v2:remote_execution_go_proto", | ||
| "@com_github_gocql_gocql//:gocql", | ||
| "@com_github_prometheus_client_golang//prometheus", | ||
| "@org_golang_google_grpc//codes", | ||
| "@org_golang_google_grpc//status", | ||
| "@org_golang_x_sync//errgroup", | ||
| ], | ||
| ) | ||
|
|
||
| go_test( | ||
| name = "cassandra_test", | ||
| srcs = [ | ||
| "cassandra_blob_access_test.go", | ||
| "format_test.go", | ||
| ], | ||
| embed = [":cassandra"], | ||
| deps = [ | ||
| "@com_github_gocql_gocql//:gocql", | ||
| "@com_github_stretchr_testify//require", | ||
| ], | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| # Cassandra | ||
|
|
||
| ## Required setup | ||
|
|
||
| `bb-storage` should not be configured to use the admin user by default in | ||
| production. Instead, you should be using a regular user which cannot create | ||
| tables or keyspaces. | ||
|
|
||
| The tables that are required can be set up using command similar to the | ||
| below in a `cqlsh` session: | ||
|
|
||
| ```shell | ||
| CREATE KEYSPACE IF NOT EXISTS buildbarn_storage WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor': 3 }; | ||
|
|
||
| USE buildbarn_storage; | ||
|
|
||
| CREATE TABLE IF NOT EXISTS prod_metadata ( | ||
| digest_function TEXT, | ||
| digest_hash BLOB, | ||
| digest_size_bytes BIGINT, | ||
| digest_instance_name TEXT, | ||
| blob_id ASCII, | ||
| last_access TIMESTAMP, | ||
| segment_count INT, | ||
| segment_size INT, | ||
| PRIMARY KEY ((digest_function, digest_hash, digest_size_bytes, digest_instance_name))) | ||
| WITH gc_grace_seconds = 86400; | ||
|
|
||
| CREATE TABLE IF NOT EXISTS prod_content ( | ||
| blob_id ASCII, | ||
| segment INT, | ||
| content BLOB, | ||
| PRIMARY KEY ((blob_id, segment))) | ||
| WITH gc_grace_seconds = 86400; | ||
|
|
||
| CREATE TABLE IF NOT EXISTS prod_orphaned_content ( | ||
| blob_id ASCII, | ||
| digest_instance_name TEXT, | ||
| digest_function TEXT, | ||
| digest_hash BLOB, | ||
| digest_size_bytes BIGINT, | ||
| segment_count INT, | ||
| timestamp DATE, | ||
| PRIMARY KEY (blob_id, digest_instance_name)) | ||
| WITH WITH gc_grace_seconds = 86400; | ||
| ``` | ||
|
|
||
| Note that you will need a table prefix (`prod` in this case), which allow | ||
| multiple different environments to run in the same keyspace. | ||
|
|
||
| The `prod_orphaned_content` table is used to track items that may have been | ||
| orphaned. Essentially, as blobs begin to be streamed into cassandra, a note | ||
| is taken of the blob and the estimated number of segments it will require. | ||
| Users are expected to have their own processes in place for using this data | ||
| to "reap" orphan content from the table. | ||
|
|
||
| The `prod_orphaned_content` table will contain many Cassandra tombstones | ||
| (most rows are deleted soon after they are inserted). For that reason, you | ||
| should reduce the `gc_grace_seconds` to 1 day. This reduces the minimum | ||
| amount of time tombstones will be kept around. This can cause some rows to | ||
| resurrect if a Cassandra node is down for more than a day, but it is not a | ||
| problem at the application level: they will just need to be reaped again. | ||
|
|
||
| In the long term, this might still create many tombstones, causing non-critical | ||
| errors when reaping old orphan rows. Consider setting a scheduled compaction | ||
| (e.g. every 2h) to help ensure that tombstones are regularly reviewed for | ||
| garbage collection. | ||
|
|
||
| The `prod_metadata` and `prod_content` tables also contain many tombstones | ||
| so the `gc_grace_seconds` attribute is also set to 1 day (86400s). | ||
|
|
||
| ### Configuring `bb-storage` | ||
|
|
||
| An example snippet of jsonnet config for setting up Cassandra is: | ||
|
|
||
| ```jsonnet | ||
| { | ||
| cassandra: { | ||
| hosts: ["cassandra.mycorp.com:1234"], | ||
| context: "example/bazel", | ||
| keyspace: "buildbarn_storage", | ||
| tablePrefix: "prod", | ||
| segmentSize: 524288, | ||
| username: "cassandra_user", | ||
| password: "hunter2", | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The segment size is the size in bytes before a new segment should be created. | ||
| It is a balancing act between being able to store as many items as possible | ||
| in a single segment and the maximum row size allowed in Cassandra (1MB). In | ||
| this example, 512kb was chosen, which should allow the majority of writes | ||
| seen by Buildbarn to be written in a single row. | ||
|
|
||
| ### Using the nearest DC | ||
|
|
||
| To minimise latency, you can set a `preferredDC` in the `cassandra` | ||
| configuration. This works on the assumption that physical proximity will | ||
| lead to lower latencies, so the preferred DC is the one that is closest to the | ||
| particular cluster the Buildbarn is installed in. To find the nearest data | ||
| centre, talk to someone familiar with your network topology, and select from | ||
| the datacenters listed in your `NetworkTopologyStrategy` (if you're using | ||
| one) when creating the Cassandra keyspace. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/shell/sql/ or something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Checking the supported languages, I think sql is probably right.