From bb19f991b92699f725e30e27199b33f78941c5bb Mon Sep 17 00:00:00 2001 From: Whit Waldo Date: Sat, 23 Aug 2025 18:04:25 -0500 Subject: [PATCH 1/9] Initial file store proposal Signed-off-by: Whit Waldo --- 20250823-BCS-file-store.md | 306 +++++++++++++++++++++++++++++++++++++ 1 file changed, 306 insertions(+) create mode 100644 20250823-BCS-file-store.md diff --git a/20250823-BCS-file-store.md b/20250823-BCS-file-store.md new file mode 100644 index 0000000..90ec535 --- /dev/null +++ b/20250823-BCS-file-store.md @@ -0,0 +1,306 @@ +# File storage building block +- Author(s): @WhitWaldo +- Significantly inspired by: @ItalyPaleAle (https://github.com/dapr/proposals/pull/18) +- Updated August 23, 2025 + +## Overview +This is a design proposal for building a new specialized state store for managing named files in state providers. I +intentionally use the work file as opposed to blob or object as to avoid any of the overloaded meaning associated with +common object store or blob store providers. I think there's every possibility that it'll make sense to introduce +a proper "blob store" or "object store" building block in the future. However, for now I think it's best to keep the +scope of this proposal focused on files more generally and avoid the complexities of either of those specialty stores. +Over time, the distinction between this and another Blob Store or Object Store blobk should be made clearer by the +enhanced capabilities of either of those APIs (e.g. list by prefix or handling metadata), but this proposal should +be understood to be a bare-boned implementation designed to strictly store and retrieve (potentially) large serialized +files. + +Files are any contiguous sequences of unstructured data comprised of bytes that should be assumed to span lengths as +few as a handful of bytes to several gigabytes. Examples include images, videos, documents, database backups, +compressed archives and so on. + +Finally, while I also considered calling this the even more ambiguous `DataStore`, I want to be clear that this is +intended to persist data that's been serialized and not some store for arbitrary in-memory data that's not trivially +persisted externally. It should be called out in the documentation that this is not intended to be a replacement for +something like SMB or NFS and is just a general-purpose storage implementation that doesn't get into the Object/Blob +details. + +## Background +Dapr currently implements a state store around a strictly key/value basis. This approach has led to component support +across many different component providers and works for many common use cases. However, decisions around the design +and implementation of the state store feature make it increasingly important to introduce a new generation of +special-purpose state management building blocks instead of having a one-size-fits-all approach to managing state. + +For example, files differ from values in key/value stores in that they can frequently span large amounts of data that +isn't generally compatible with the likes of Redis or Etcd. Because they span such large amounts of data, it's frequently +insufficient to simply store and retrieve them synchronously, mandating a stream-first approach to minimize +resources needed by the runtime to transfer the data. The goal is to allow users to store and retrieve files in a +performant and scalable manner via a common runtime API and enable the Dapr SDKs to complement the implementation to +support additional provider-specific features on a case by case basis. A possible example of this will be given below. + +## Component YAML +This component is expected to have similar attributes to existing state stores with some variation: + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: filestore +spec: + type: state.filestore. + version: v1.0 # Following the API versioning convention proposed in https://github.com/dapr/proposals/pull/87 + metadata: + # Various properties necessary for component registration +``` + +## Implementation Overview +As touched on above, there are a few guiding principles I want to carry through from previous iterations of this +concept: +- All operations should be asynchronously performed with the sidecar +- When persisting or retrieving state, this should be done using an asynchronous stream to maximize SDK and runtime +performance while minimizing resource overhead and preventing memory exhaustion +- While SDKs should allow metadata to be stored alongside the data, if it's not supported by the provider, the +documentation should clearly state this lack of support (e.g. that the runtime will not otherwise persist it). + +### Make Runtime APIs Simpler +I'd like to make the runtime APIs as simple as possible so they're more readily used in a composable way alongside other +Dapr APIs by the SDKs, pluggable components or other developer-written utilities. With that in mind, I do not include +any support for persisting metadata with the files or listing the files in the store in this API. This ensures +broader compatibility with possible stores for file-based data and leaves the possibility of greater sophistication +to the SDKs to handle instead. + +This approach allows for a leaner, more provider-agnostic API, takes work off the plate of the runtime developers and +better positions the SDKs to provide a more comprehensive experience leveraging existing building blocks, improving +and simplifying the developer experience. + +> **NOTE** It would be imperative to call out in the developer documentation for this building block that the metadata and listing +information returned is not actually provided by the state provider and is instead provided based on values provided +by the developer and are not necessarily reflective of the actual state of the store. This could provide a key +distinction between this and an actual object or blob store for those that need that level of detail (with the tradeoff +that fewer providers would be supported in the other implementation). + + +#### Example: C# SDK +For example, in the C# SDK, the developer might specify a key/value store of their choosing to persist file metadata +during the dependency injection registration alongside the name of the file store component: + +```csharp +services.AddDapr() + .WithFileStore(opt => + { + opt.FileStoreName = "my-file-store"; // Defines the file store to use for this injected `DaprFileStore` type + opt.MetadataStoreName = "my-metadata-store"; // Defines the metadata store to persist metadata to + }); +``` + +This would provide a simple approach that allows the developer to use the SDK to persist files that optionally include +metadata with a DI-registered type. This would avoid the current approach of having to pass in the store names each time. + +```csharp +public sealed class MyClass(DaprFileStore fileStore) +{ + public async Task SaveDataAsync(string name, byte[] data) + { + await fileStore.SetAsync(name, data); + } + + public async Task SaveDataWithMetadataAsync(string name, byte[] data, IReadOnlyDictionary metadata ) + { + await fileStore.SetAsync(name, data, metadata); + } +} +``` + +Further, the C# SDK could provide a trivial expansion on this idea to allow named types for saving data to different +places throughout the application. This example uses values from a shared configuration while still relying on Dapr to +manage the type lifecycle: + +```csharp +services.AddDapr() + .WithFileStore("fileStoreA", (serviceProvider, options) => + { + // Populate the file store and metadata store names from a configuration + var configuration = serviceProvider.GetRequiredService(); + var fileStoreName = configuration.GetValue("FileStoreName"); + var metadataStoreName = configuration.GetValue("MetadataStoreName"); + + options.FileStoreName = fileStoreName; + options.MetadataStoreName = metadataStoreName; + }) + .WithFileStore("fileStoreB", options => + { + // Use static values to populate the names in this named `DaprFileStore` client + options.FileStoreName = "my-file-store"; + options.MetadataStoreName = "my-metadata-store"; + }); +``` + +Because I don't include list support in this API either, the SDK can again be relied on to abstract this away for the +developer. Paired with the next-generation key/value store I [have proposed](https://github.com/dapr/dapr/issues/7338), +the developer might query the list of files in the store by instead querying the metadata key/value store for any +keys with the file name prefix they're interested in: + +```csharp +public sealed class MyClass(DaprFileStore fileStore) +{ + public async Task> GetFileNamesAsync(string prefix) + => await fileStore.ListFilesAsync(prefix); +} +``` + +### gRPC APIs +In the Dapr gRPC APIs, we'd either extend the `runtime.v1.Dapr` service to add new methods or create a separate service +specific to the file store. The benefit of taking the latter approach is that it need only be updated when the API +changes and otherwise won't trigger downstream operations monitoring changes to that file. However, this approach +generally falls outside the scope of this proposal and we'll assume that `runtime.v1.Dapr` is simply extended with the +following: + +| Note: Consistent with current practice, APIs will have Alpha1 suffixed to the type names while in preview. + +| Note: Any authentication behaviors are maintained in the component YAML configuration. + +```proto +// Existing Dapr service +service Dapr { + rpc SetFileAlpha1(SetFileRequest) returns (SetFileResponse) {} + rpc GetFileAlpha1(GetFileRequest) returns (GetFileResponse) {} + rpc DeleteFileAlpha1(DeleteFileRequest) returns (DeleteFileResponse) {} +} + +message SetFileRequest { + // Request details. Must be present in the first message only. + SetFileRequestOptions options = 1; + // Chunk of data of arbitrary size + common.v1.StreamPayload payload = 2; +} + +message SetFileRequestOptions { + string component_name = 1 [json_name="componentName"]; + string file_name = 2 [json_name="fileName"]; + bool overwrite = 3 [json_name="overwrite"]; +} + +message SetFileResponse {} + +message GetFileRequest { + string component_name = 1 [json_name="componentName"]; + string file_name = 2 [json_name="fileName"]; +} + +// The response for GetFileRequest +message GetFileResponse { + // Chunk of data + common.v1.StreamPayload payload = 1; +} + +message DeleteFileRequest { + string component_name = 1 [json_name="componentName"]; + string file_name = 2 [json_name="fileName"]; +} + +message DeleteFileResponse {} +``` + +### HTTP APIs + +#### Persist the file data whether it already exists or not +##### Request +PUT `http://localhost:{daprPort}/v1.0-alpha1/state/filestore/{componentName}/{fileName}` + +The request body should contain an array of (nominally UTF-8 encoded) bytes. + +##### Response +| Code | Description | +| ---- | ---- | +| 200 | File data successfully persisted | +| 400 | File store provider not found | +| 500 | Request formatted correctly, but error in Dapr code or underlying component | + + +#### Persist the file data only if it doesn't already exist with the given name +##### Request +POST `http://localhost:{daprPort}/v1.0-alpha1/state/filestore/{componentName}/{fileName}` + +The request body should contain an array of (nominally UTF-8 encoded) bytes. + +##### Response +| Code | Description | +| ---- | ---- | +| 200 | File data successfully persisted | +| 400 | File store provider not found | +| 409 | File already exists | +| 500 | Request formatted correctly, but error in Dapr code or underlying component | + + +#### Retrieve the file data +##### Request +GET `http://localhost:{daprPort}/v1.0-alpha1/state/filestore/{componentName}/{fileName}` + +##### Response +The response to a decryption request will have its content type header set to `application/octet-stream` as it +returns an array of bytes representing the file data. + +| Code | Description | +| ---- | ---- | +| 200 | File data successfully retrieved | +| 400 | File store provider not found | +| 404 | File not found | +| 500 | Request formatted correctly, but error in Dapr code or underlying component | + + +#### Delete the file +##### Request +DELETE `http://localhost:{daprPort}/v1.0-alpha1/state/filestore/{componentName}/{fileName}` + +##### Response +If the file does not exist, returns: +| Code | Description | +| ---- | ---- | +| 204 | Indicates the delete operation completed successfully | +| 400 | File store provider not found | +| 404 | File not found | +| 500 | Request formatted correctly, but error in Dapr code or underlying component | + +## Final Thoughts +This section exists to address some of the concerns shared in previous iterations of this proposal: + +### Why now? +This is a new building block intended to be used in conjunction with the existing state store building blocks. +It's not intended to replace them, but rather to complement them. The existing state management building block is +suitable only storing small amounts of data. Persisting large amounts of data puts large amounts of data in providers +not intended for this purposes and can lead to performance and resource usage issues on both the sidecar and client. + +By providing a new building block, we can provide a longer-term solution for storing large amounts of data while +still allowing the developer to leverage the existing state store building blocks for managing small amounts of +data more suitable to the key/value stores it's built for. + +Further, especially as more SDKs support agentic frameworks built atop Dapr, those tools will need a place to store +data that's larger in scope than a key/value store is necessarily designed to accommodate. Having this performant +alternative to our existing key/value store implementation available for persisting and retrieving that data would +be quite timely. + +### How might it be immediately used in the SDKs? +Once we had support for this in the runtime, I'd write a helper tool for Dapr Workflows in the .NET SDK. When activities +yield any data, this utility would persist it to this file store and return a reference to it that can be returned to the +calling workflow. Future activity invocations can then retrieve the data identified by that reference from the file +store. + +Similarly, such large data could be persisted by other operations and passed into workflows as inputs to be retrieved +by activities in later operations. + +This would improve workflow performance by keeping large data out of the event source log in both the inputs and outputs +of the workflows and their activities. + +### Why doesn't it support [awesome-feature-X]? +This is intended as a provider-agnostic, lean building block that provides a simple mechanism to store data +in a streaming fashion that's generally too large for a key/value store. + +I leave it to the architects of future even more specialized state store building blocks to decide whether they want to +support additional features beyond what's provided beyond this narrow implementation. + + +## Completion Checklist +[] File Store Runtime Implementation +[] Tests added (e2e, unit) +[] Implement File Store providers +[] SDK Support +[] Documentation (supported providers, purpose, set up documentation structure for speciality state stores) From 98d52532155d1bc3d9de93846104bc8ff35ba942 Mon Sep 17 00:00:00 2001 From: Whit Waldo Date: Sat, 23 Aug 2025 18:06:59 -0500 Subject: [PATCH 2/9] Spelling, grammar and formatting check Signed-off-by: Whit Waldo --- 20250823-BCS-file-store.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/20250823-BCS-file-store.md b/20250823-BCS-file-store.md index 90ec535..e1779e2 100644 --- a/20250823-BCS-file-store.md +++ b/20250823-BCS-file-store.md @@ -10,7 +10,7 @@ common object store or blob store providers. I think there's every possibility t a proper "blob store" or "object store" building block in the future. However, for now I think it's best to keep the scope of this proposal focused on files more generally and avoid the complexities of either of those specialty stores. Over time, the distinction between this and another Blob Store or Object Store blobk should be made clearer by the -enhanced capabilities of either of those APIs (e.g. list by prefix or handling metadata), but this proposal should +enhanced capabilities of either of those APIs (e.g., list by prefix or handling metadata). However, this proposal should be understood to be a bare-boned implementation designed to strictly store and retrieve (potentially) large serialized files. @@ -59,7 +59,7 @@ concept: - When persisting or retrieving state, this should be done using an asynchronous stream to maximize SDK and runtime performance while minimizing resource overhead and preventing memory exhaustion - While SDKs should allow metadata to be stored alongside the data, if it's not supported by the provider, the -documentation should clearly state this lack of support (e.g. that the runtime will not otherwise persist it). +documentation should clearly state this lack of support (e.g., that the runtime will not otherwise persist it). ### Make Runtime APIs Simpler I'd like to make the runtime APIs as simple as possible so they're more readily used in a composable way alongside other @@ -72,11 +72,11 @@ This approach allows for a leaner, more provider-agnostic API, takes work off th better positions the SDKs to provide a more comprehensive experience leveraging existing building blocks, improving and simplifying the developer experience. -> **NOTE** It would be imperative to call out in the developer documentation for this building block that the metadata and listing -information returned is not actually provided by the state provider and is instead provided based on values provided -by the developer and are not necessarily reflective of the actual state of the store. This could provide a key -distinction between this and an actual object or blob store for those that need that level of detail (with the tradeoff -that fewer providers would be supported in the other implementation). +> **NOTE** It would be imperative to call out in the developer documentation for this building block that the metadata +> and listing information returned is not provided by the state provider and is instead provided based on values provided +> by the developer and are not necessarily reflective of the actual state of the store. This could provide a key +> distinction between this and an actual object or blob store for those that need that level of detail (with the tradeoff +> that fewer providers would be supported in the other implementation). #### Example: C# SDK @@ -161,9 +161,9 @@ following: ```proto // Existing Dapr service service Dapr { - rpc SetFileAlpha1(SetFileRequest) returns (SetFileResponse) {} - rpc GetFileAlpha1(GetFileRequest) returns (GetFileResponse) {} - rpc DeleteFileAlpha1(DeleteFileRequest) returns (DeleteFileResponse) {} + rpc SetFileAlpha1(stream SetFileRequest) returns (SetFileResponse) {}; + rpc GetFileAlpha1(GetFileRequest) returns (stream GetFileResponse); + rpc DeleteFileAlpha1(DeleteFileRequest) returns (DeleteFileResponse) {}; } message SetFileRequest { From f71bc2a42f26c16309de997ef03bf78434be8429 Mon Sep 17 00:00:00 2001 From: Whit Waldo Date: Sat, 23 Aug 2025 18:18:14 -0500 Subject: [PATCH 3/9] Added reference to data reference issue in .NET SDK Signed-off-by: Whit Waldo --- 20250823-BCS-file-store.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/20250823-BCS-file-store.md b/20250823-BCS-file-store.md index e1779e2..f387deb 100644 --- a/20250823-BCS-file-store.md +++ b/20250823-BCS-file-store.md @@ -279,10 +279,10 @@ alternative to our existing key/value store implementation available for persist be quite timely. ### How might it be immediately used in the SDKs? -Once we had support for this in the runtime, I'd write a helper tool for Dapr Workflows in the .NET SDK. When activities -yield any data, this utility would persist it to this file store and return a reference to it that can be returned to the -calling workflow. Future activity invocations can then retrieve the data identified by that reference from the file -store. +Once we had support for this in the runtime, I'd write a proposed [helper tool](https://github.com/dapr/dotnet-sdk/issues/1533) +for Dapr Workflows in the .NET SDK. When activities yield any data, this utility would persist it to this file store +and return a reference to it that can be returned to the calling workflow. Future activity invocations can then retrieve +the data identified by that reference from the file store. Similarly, such large data could be persisted by other operations and passed into workflows as inputs to be retrieved by activities in later operations. From 061e410b91c09295ec2b5cd2003363d6049099db Mon Sep 17 00:00:00 2001 From: Whit Waldo Date: Sat, 23 Aug 2025 18:24:48 -0500 Subject: [PATCH 4/9] Added another source for inspiration Signed-off-by: Whit Waldo --- 20250823-BCS-file-store.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/20250823-BCS-file-store.md b/20250823-BCS-file-store.md index f387deb..b9de9ac 100644 --- a/20250823-BCS-file-store.md +++ b/20250823-BCS-file-store.md @@ -1,6 +1,6 @@ # File storage building block - Author(s): @WhitWaldo -- Significantly inspired by: @ItalyPaleAle (https://github.com/dapr/proposals/pull/18) +- Significantly inspired by: @ItalyPaleAle (https://github.com/dapr/proposals/pull/18), @artursouza (https://github.com/dapr/dapr/issues/4804) - Updated August 23, 2025 ## Overview From 76986f3d38c6a5fe10b12a5227f9782e7ca58ca4 Mon Sep 17 00:00:00 2001 From: Whit Waldo Date: Sat, 23 Aug 2025 18:26:45 -0500 Subject: [PATCH 5/9] Fixed checkbox formatting at bottom Signed-off-by: Whit Waldo --- 20250823-BCS-file-store.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/20250823-BCS-file-store.md b/20250823-BCS-file-store.md index b9de9ac..780c363 100644 --- a/20250823-BCS-file-store.md +++ b/20250823-BCS-file-store.md @@ -299,8 +299,8 @@ support additional features beyond what's provided beyond this narrow implementa ## Completion Checklist -[] File Store Runtime Implementation -[] Tests added (e2e, unit) -[] Implement File Store providers -[] SDK Support -[] Documentation (supported providers, purpose, set up documentation structure for speciality state stores) +- [ ] File Store Runtime Implementation +- [ ] Tests added (e2e, unit) +- [ ] Implement File Store providers +- [ ] SDK Support +- [ ] Documentation (supported providers, purpose, set up documentation structure for speciality state stores) From 04c671219a7c914fe340954313d039d9553b262c Mon Sep 17 00:00:00 2001 From: Whit Waldo Date: Sat, 23 Aug 2025 18:51:11 -0500 Subject: [PATCH 6/9] Added alternative name 'BinaryStore' to thoughts at the bottom Signed-off-by: Whit Waldo --- 20250823-BCS-file-store.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/20250823-BCS-file-store.md b/20250823-BCS-file-store.md index 780c363..ea416fd 100644 --- a/20250823-BCS-file-store.md +++ b/20250823-BCS-file-store.md @@ -295,8 +295,17 @@ This is intended as a provider-agnostic, lean building block that provides a sim in a streaming fashion that's generally too large for a key/value store. I leave it to the architects of future even more specialized state store building blocks to decide whether they want to -support additional features beyond what's provided beyond this narrow implementation. +support additional features beyond what's provided beyond this narrow implementation. +### What if we called it `BinaryStore` instead of `FileStore`? (thanks @olitomlinson!) +Love it. By calling out that it's a file store, I'm simply trying to avoid any confusion that a developer can pass some +arbitrary in-memory representation and trust that some serialization magic will persist it. By requiring that it be a +file, we're ensuring that it's in a state that's persistable elsewhere. Here, I imagine that the SDK API would require +that the developer provide an array of bytes or a byte stream of already-serialized/encoded data as it needn't strictly +be a location on their file system to a file. + +To that end then, I'm entirely ok with calling this a `BinaryStore` instead as it gives a similar level of +understanding if that makes more sense than `FileStore`. ## Completion Checklist - [ ] File Store Runtime Implementation From 73fdd2ae0f6510930f8c238997182320c9c6dd05 Mon Sep 17 00:00:00 2001 From: Whit Waldo Date: Sat, 23 Aug 2025 19:12:27 -0500 Subject: [PATCH 7/9] Added link back to my specialized store issue in dapr/dapr Signed-off-by: Whit Waldo --- 20250823-BCS-file-store.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/20250823-BCS-file-store.md b/20250823-BCS-file-store.md index ea416fd..1e3ff1a 100644 --- a/20250823-BCS-file-store.md +++ b/20250823-BCS-file-store.md @@ -4,15 +4,15 @@ - Updated August 23, 2025 ## Overview -This is a design proposal for building a new specialized state store for managing named files in state providers. I -intentionally use the work file as opposed to blob or object as to avoid any of the overloaded meaning associated with -common object store or blob store providers. I think there's every possibility that it'll make sense to introduce -a proper "blob store" or "object store" building block in the future. However, for now I think it's best to keep the -scope of this proposal focused on files more generally and avoid the complexities of either of those specialty stores. -Over time, the distinction between this and another Blob Store or Object Store blobk should be made clearer by the -enhanced capabilities of either of those APIs (e.g., list by prefix or handling metadata). However, this proposal should -be understood to be a bare-boned implementation designed to strictly store and retrieve (potentially) large serialized -files. +This is a design proposal for building a new [specialized state store](https://github.com/dapr/dapr/issues/7339) for +managing named files in state providers. I intentionally use the work file as opposed to blob or object as to avoid +any of the overloaded meaning associated with common object store or blob store providers. I think there's every +possibility that it'll make sense to introduce a proper "blob store" or "object store" building block in the future. +However, for now I think it's best to keep the scope of this proposal focused on files more generally and avoid the +complexities of either of those specialty stores. Over time, the distinction between this and another Blob Store or +Object Store block should be made clearer by the enhanced capabilities of either of those APIs (e.g., list by prefix +or handling metadata). However, this proposal should be understood to be a bare-boned implementation designed to +strictly store and retrieve (potentially) large serialized files. Files are any contiguous sequences of unstructured data comprised of bytes that should be assumed to span lengths as few as a handful of bytes to several gigabytes. Examples include images, videos, documents, database backups, From 06a30baf35b12048080f95f6be3798b99aa9d9f3 Mon Sep 17 00:00:00 2001 From: Whit Waldo Date: Sat, 23 Aug 2025 22:45:34 -0500 Subject: [PATCH 8/9] Updated to named this the BinaryStore instead of FileStore, reworded a few details, removed an extraneous bullet and generally cleaned it up some Signed-off-by: Whit Waldo --- 20250823-BCS-file-store.md | 162 ++++++++++++++++++------------------- 1 file changed, 77 insertions(+), 85 deletions(-) diff --git a/20250823-BCS-file-store.md b/20250823-BCS-file-store.md index 1e3ff1a..e350021 100644 --- a/20250823-BCS-file-store.md +++ b/20250823-BCS-file-store.md @@ -1,11 +1,12 @@ -# File storage building block +# Binary storage building block - Author(s): @WhitWaldo - Significantly inspired by: @ItalyPaleAle (https://github.com/dapr/proposals/pull/18), @artursouza (https://github.com/dapr/dapr/issues/4804) +- Name inspired by @olitomlinson - Updated August 23, 2025 ## Overview This is a design proposal for building a new [specialized state store](https://github.com/dapr/dapr/issues/7339) for -managing named files in state providers. I intentionally use the work file as opposed to blob or object as to avoid +managing named files in state providers. I intentionally use the word "file" as opposed to blob or object as to avoid any of the overloaded meaning associated with common object store or blob store providers. I think there's every possibility that it'll make sense to introduce a proper "blob store" or "object store" building block in the future. However, for now I think it's best to keep the scope of this proposal focused on files more generally and avoid the @@ -16,13 +17,12 @@ strictly store and retrieve (potentially) large serialized files. Files are any contiguous sequences of unstructured data comprised of bytes that should be assumed to span lengths as few as a handful of bytes to several gigabytes. Examples include images, videos, documents, database backups, -compressed archives and so on. - -Finally, while I also considered calling this the even more ambiguous `DataStore`, I want to be clear that this is -intended to persist data that's been serialized and not some store for arbitrary in-memory data that's not trivially -persisted externally. It should be called out in the documentation that this is not intended to be a replacement for -something like SMB or NFS and is just a general-purpose storage implementation that doesn't get into the Object/Blob -details. +compressed archives and so on. This original version of this proposal referred to this being a "file store", but after +talking about it with Oli, I've warmed up to instead calling it the "binary store". This is better than even `DataStore` +because the intent is that the data will already have been serialized and encoded so it's ready to persist somewhere. +While this might be from a file, it doesn't necessarily have to be, so much as that it's just an array (or while in +transit, a stream) of bytes. Referring to it as a `BinaryStore` also rules out any confusion about whether it's +abstracting SMB or NFS. ## Background Dapr currently implements a state store around a strictly key/value basis. This approach has led to component support @@ -33,7 +33,7 @@ special-purpose state management building blocks instead of having a one-size-fi For example, files differ from values in key/value stores in that they can frequently span large amounts of data that isn't generally compatible with the likes of Redis or Etcd. Because they span such large amounts of data, it's frequently insufficient to simply store and retrieve them synchronously, mandating a stream-first approach to minimize -resources needed by the runtime to transfer the data. The goal is to allow users to store and retrieve files in a +resources needed by the runtime to transfer the data. The goal is to allow users to store and retrieve binary blobs in a performant and scalable manner via a common runtime API and enable the Dapr SDKs to complement the implementation to support additional provider-specific features on a case by case basis. A possible example of this will be given below. @@ -44,9 +44,9 @@ This component is expected to have similar attributes to existing state stores w apiVersion: dapr.io/v1alpha1 kind: Component metadata: - name: filestore + name: binarystore spec: - type: state.filestore. + type: state.binarystore. version: v1.0 # Following the API versioning convention proposed in https://github.com/dapr/proposals/pull/87 metadata: # Various properties necessary for component registration @@ -58,8 +58,6 @@ concept: - All operations should be asynchronously performed with the sidecar - When persisting or retrieving state, this should be done using an asynchronous stream to maximize SDK and runtime performance while minimizing resource overhead and preventing memory exhaustion -- While SDKs should allow metadata to be stored alongside the data, if it's not supported by the provider, the -documentation should clearly state this lack of support (e.g., that the runtime will not otherwise persist it). ### Make Runtime APIs Simpler I'd like to make the runtime APIs as simple as possible so they're more readily used in a composable way alongside other @@ -72,22 +70,22 @@ This approach allows for a leaner, more provider-agnostic API, takes work off th better positions the SDKs to provide a more comprehensive experience leveraging existing building blocks, improving and simplifying the developer experience. -> **NOTE** It would be imperative to call out in the developer documentation for this building block that the metadata -> and listing information returned is not provided by the state provider and is instead provided based on values provided -> by the developer and are not necessarily reflective of the actual state of the store. This could provide a key -> distinction between this and an actual object or blob store for those that need that level of detail (with the tradeoff -> that fewer providers would be supported in the other implementation). +> **NOTE** It would be important to call out in the developer documentation for this building block that the metadata +> and listing information returned is not provided by the state provider. Rather, it is instead provided based on +> values provided by the developer and are not necessarily reflective of the actual state of the store. This could +> provide a key distinction between this and an actual object or blob store for those that need that level of detail +> (with the tradeoff that fewer providers would be supported in the other implementation). #### Example: C# SDK -For example, in the C# SDK, the developer might specify a key/value store of their choosing to persist file metadata +For example, in the C# SDK, the developer might specify a key/value store of their choosing to persist metadata during the dependency injection registration alongside the name of the file store component: ```csharp services.AddDapr() .WithFileStore(opt => { - opt.FileStoreName = "my-file-store"; // Defines the file store to use for this injected `DaprFileStore` type + opt.BinaryStoreName = "my-file-store"; // Defines the binary store to use for this injected `DaprBinaryStore` type opt.MetadataStoreName = "my-metadata-store"; // Defines the metadata store to persist metadata to }); ``` @@ -96,16 +94,19 @@ This would provide a simple approach that allows the developer to use the SDK to metadata with a DI-registered type. This would avoid the current approach of having to pass in the store names each time. ```csharp -public sealed class MyClass(DaprFileStore fileStore) +public sealed class MyClass(DaprBinaryStore binaryStore) { - public async Task SaveDataAsync(string name, byte[] data) + public async Task SaveAsync(string name, byte[] data) { - await fileStore.SetAsync(name, data); + await binaryStore.SetAsync(name, data); } - public async Task SaveDataWithMetadataAsync(string name, byte[] data, IReadOnlyDictionary metadata ) + public async Task SaveWithMetadataAsync(string name, byte[] data, IReadOnlyDictionary metadata ) { - await fileStore.SetAsync(name, data, metadata); + if (string.IsNullOrWhitespace(binaryStore.MetadataStoreName)) + throw new ArgumentException("Metadata store name must be set in the DaprFileStoreOptions during DI registration"); + + await binaryStore.SetAsync(name, data, metadata); } } ``` @@ -116,20 +117,20 @@ manage the type lifecycle: ```csharp services.AddDapr() - .WithFileStore("fileStoreA", (serviceProvider, options) => + .WithBinaryStore("storeA", (serviceProvider, options) => { // Populate the file store and metadata store names from a configuration var configuration = serviceProvider.GetRequiredService(); - var fileStoreName = configuration.GetValue("FileStoreName"); + var binaryStoreName = configuration.GetValue("BinaryStoreName"); var metadataStoreName = configuration.GetValue("MetadataStoreName"); options.FileStoreName = fileStoreName; options.MetadataStoreName = metadataStoreName; }) - .WithFileStore("fileStoreB", options => + .WithBinaryStore("storeB", options => { // Use static values to populate the names in this named `DaprFileStore` client - options.FileStoreName = "my-file-store"; + options.BinaryStoreName = "my-binary-store"; options.MetadataStoreName = "my-metadata-store"; }); ``` @@ -140,10 +141,10 @@ the developer might query the list of files in the store by instead querying the keys with the file name prefix they're interested in: ```csharp -public sealed class MyClass(DaprFileStore fileStore) +public sealed class MyClass(DaprBinaryStore binaryStore) { public async Task> GetFileNamesAsync(string prefix) - => await fileStore.ListFilesAsync(prefix); + => await binaryStore.ListFilesAsync(prefix); } ``` @@ -161,98 +162,99 @@ following: ```proto // Existing Dapr service service Dapr { - rpc SetFileAlpha1(stream SetFileRequest) returns (SetFileResponse) {}; - rpc GetFileAlpha1(GetFileRequest) returns (stream GetFileResponse); - rpc DeleteFileAlpha1(DeleteFileRequest) returns (DeleteFileResponse) {}; + rpc SetBinaryFileAlpha1(stream SetBinaryFileRequest) returns (SetBinaryFileResponse) {}; + rpc GetBinaryFileAlpha1(GetBinarFileRequest) returns (stream GetBinaryFileResponse); + rpc DeleteBinaryFileAlpha1(DeleteBinaryFileRequest) returns (DeleteBinaryFileResponse) {}; } -message SetFileRequest { +message SetBinaryFileRequest { // Request details. Must be present in the first message only. - SetFileRequestOptions options = 1; + SetBinaryRequestOptions options = 1; // Chunk of data of arbitrary size common.v1.StreamPayload payload = 2; } -message SetFileRequestOptions { +message SetBinaryFileRequestOptions { string component_name = 1 [json_name="componentName"]; string file_name = 2 [json_name="fileName"]; bool overwrite = 3 [json_name="overwrite"]; } -message SetFileResponse {} +// The response for SetBinaryFileAlpha1 +message SetBinaryFileResponse {} -message GetFileRequest { +message GetBinaryFileRequest { string component_name = 1 [json_name="componentName"]; string file_name = 2 [json_name="fileName"]; } -// The response for GetFileRequest -message GetFileResponse { +// The response for GetBinaryFileAlpha1 +message GetBinaryFileResponse { // Chunk of data common.v1.StreamPayload payload = 1; } -message DeleteFileRequest { +message DeleteBinaryFileRequest { string component_name = 1 [json_name="componentName"]; string file_name = 2 [json_name="fileName"]; } -message DeleteFileResponse {} +// The response for DeleteBinaryFileAlpha1 +message DeleteBinaryFileResponse {} ``` ### HTTP APIs -#### Persist the file data whether it already exists or not +#### Persist the binary data whether it already exists or not ##### Request -PUT `http://localhost:{daprPort}/v1.0-alpha1/state/filestore/{componentName}/{fileName}` +PUT `http://localhost:{daprPort}/v1.0-alpha1/state/binarystore/{componentName}/{fileName}` The request body should contain an array of (nominally UTF-8 encoded) bytes. ##### Response -| Code | Description | -| ---- | ---- | -| 200 | File data successfully persisted | -| 400 | File store provider not found | +| Code | Description | +| ---- |-----------------------------------------------------------------------------| +| 200 | Binary data successfully persisted | +| 400 | Binary store provider not found | | 500 | Request formatted correctly, but error in Dapr code or underlying component | -#### Persist the file data only if it doesn't already exist with the given name +#### Persist the binary data only if it doesn't already exist with the given name ##### Request -POST `http://localhost:{daprPort}/v1.0-alpha1/state/filestore/{componentName}/{fileName}` +POST `http://localhost:{daprPort}/v1.0-alpha1/state/binarystore/{componentName}/{fileName}` The request body should contain an array of (nominally UTF-8 encoded) bytes. ##### Response -| Code | Description | -| ---- | ---- | -| 200 | File data successfully persisted | -| 400 | File store provider not found | -| 409 | File already exists | +| Code | Description | +| ---- |-----------------------------------------------------------------------------| +| 200 | Binary data successfully persisted | +| 400 | Binary store provider not found | +| 409 | File already exists | | 500 | Request formatted correctly, but error in Dapr code or underlying component | -#### Retrieve the file data +#### Retrieve the binary data ##### Request -GET `http://localhost:{daprPort}/v1.0-alpha1/state/filestore/{componentName}/{fileName}` +GET `http://localhost:{daprPort}/v1.0-alpha1/state/binarystore/{componentName}/{fileName}` ##### Response -The response to a decryption request will have its content type header set to `application/octet-stream` as it -returns an array of bytes representing the file data. - -| Code | Description | -| ---- | ---- | -| 200 | File data successfully retrieved | -| 400 | File store provider not found | -| 404 | File not found | +The response to a retrieval request will have its content type header set to `application/octet-stream` as it +returns an array of bytes representing the binary file data. + +| Code | Description | +| ---- |-----------------------------------------------------------------------------| +| 200 | Binary data successfully retrieved | +| 400 | Binary store provider not found | +| 404 | File not found | | 500 | Request formatted correctly, but error in Dapr code or underlying component | -#### Delete the file +#### Delete the binary file ##### Request -DELETE `http://localhost:{daprPort}/v1.0-alpha1/state/filestore/{componentName}/{fileName}` +DELETE `http://localhost:{daprPort}/v1.0-alpha1/state/binarystore/{componentName}/{fileName}` ##### Response -If the file does not exist, returns: | Code | Description | | ---- | ---- | | 204 | Indicates the delete operation completed successfully | @@ -264,10 +266,10 @@ If the file does not exist, returns: This section exists to address some of the concerns shared in previous iterations of this proposal: ### Why now? -This is a new building block intended to be used in conjunction with the existing state store building blocks. +This is a new building block intended to be used in conjunction with the existing state store building block. It's not intended to replace them, but rather to complement them. The existing state management building block is suitable only storing small amounts of data. Persisting large amounts of data puts large amounts of data in providers -not intended for this purposes and can lead to performance and resource usage issues on both the sidecar and client. +not intended for this purpose and can lead to performance and resource usage issues on both the sidecar and client. By providing a new building block, we can provide a longer-term solution for storing large amounts of data while still allowing the developer to leverage the existing state store building blocks for managing small amounts of @@ -287,26 +289,16 @@ the data identified by that reference from the file store. Similarly, such large data could be persisted by other operations and passed into workflows as inputs to be retrieved by activities in later operations. -This would improve workflow performance by keeping large data out of the event source log in both the inputs and outputs -of the workflows and their activities. +This will improve workflow performance by keeping large data out of the event source log in both the +inputs and outputs of the workflows and their activities. ### Why doesn't it support [awesome-feature-X]? -This is intended as a provider-agnostic, lean building block that provides a simple mechanism to store data +This is intended as a provider-agnostic, lean building block that provides a simple mechanism to store binary data in a streaming fashion that's generally too large for a key/value store. I leave it to the architects of future even more specialized state store building blocks to decide whether they want to support additional features beyond what's provided beyond this narrow implementation. -### What if we called it `BinaryStore` instead of `FileStore`? (thanks @olitomlinson!) -Love it. By calling out that it's a file store, I'm simply trying to avoid any confusion that a developer can pass some -arbitrary in-memory representation and trust that some serialization magic will persist it. By requiring that it be a -file, we're ensuring that it's in a state that's persistable elsewhere. Here, I imagine that the SDK API would require -that the developer provide an array of bytes or a byte stream of already-serialized/encoded data as it needn't strictly -be a location on their file system to a file. - -To that end then, I'm entirely ok with calling this a `BinaryStore` instead as it gives a similar level of -understanding if that makes more sense than `FileStore`. - ## Completion Checklist - [ ] File Store Runtime Implementation - [ ] Tests added (e2e, unit) From f4f57ccc74f808f83db40e4835def1a1fbe74c84 Mon Sep 17 00:00:00 2001 From: Whit Waldo Date: Sat, 23 Aug 2025 22:47:40 -0500 Subject: [PATCH 9/9] Fixed quote marks Signed-off-by: Whit Waldo --- 20250823-BCS-file-store.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/20250823-BCS-file-store.md b/20250823-BCS-file-store.md index e350021..9f1c57f 100644 --- a/20250823-BCS-file-store.md +++ b/20250823-BCS-file-store.md @@ -17,8 +17,8 @@ strictly store and retrieve (potentially) large serialized files. Files are any contiguous sequences of unstructured data comprised of bytes that should be assumed to span lengths as few as a handful of bytes to several gigabytes. Examples include images, videos, documents, database backups, -compressed archives and so on. This original version of this proposal referred to this being a "file store", but after -talking about it with Oli, I've warmed up to instead calling it the "binary store". This is better than even `DataStore` +compressed archives and so on. This original version of this proposal referred to this being a `FileStore`, but after +talking about it with Oli, I've warmed up to instead calling it the `BinaryStore`. This is better than even `DataStore` because the intent is that the data will already have been serialized and encoded so it's ready to persist somewhere. While this might be from a file, it doesn't necessarily have to be, so much as that it's just an array (or while in transit, a stream) of bytes. Referring to it as a `BinaryStore` also rules out any confusion about whether it's