Proposal: Binary Store Building Block

[WhitWaldo]

Increasingly, while writing applications that use Dapr, I keep running into the need to persist data that's too large to reasonably store using Dapr often because it's too large and will exhaust the memory resources of the sidecar, though frequently because it's likely too large to store in a key/value store.

It doesn't make a ton of sense to rely exclusively on bindings for this when that really just provides a Dapr-hosted alternative to the provider's SDK for something that we should increasingly have broad provider support for. Object and blob stores are really overloaded terms representing all manner of things depending on provider for which I think there's a fine opportunity to tackle in the future - this proposal isn't that.

Here, I propose an API devoid of List and even Metadata operations so it can accommodate the broadest of possible storage providers and instead suggest that we increasingly lean on the SDKs to provide the state management instead of putting all that weight on the runtime and the components. It's a slim implementation that should be pretty easily added, but which would provide immediate benefits for popular Dapr features: Workflows and the new Agentic operations come to mind, but it would be beneficial for Actor and Cryptographic operations as well.

I look forward to your feedback!

[olitomlinson]

I'm massively in support, but how does this differ from the Object Store proposal? (Other than no support for metadata, anything else?)

[WhitWaldo]

I'm massively in support, but how does this differ from the Object Store proposal? (Other than no support for metadata, anything else?)

There are a few differences:

1. This proposal does not anticipate ever supporting a list operation so as to be more readily and broadly supported by those providers without such capability. Most specific object and blob stores that come to mind do offer such a feature. Leaving this feature as a possible differentiator for a future object/blob store API, although this would limit it to a smaller set of matching providers, is a fine trade-off to simply do without altogether here. This is looking to be little more than a provider to store large files in a way the current state store cannot and without all the other current state management add-ons.
2. This does not purport to offer those behaviors that might be more specific to object and blob stores to perform operations on data through signed URLs. Again, that might be a fine feature to use in a future state store that's more narrowly tailed to that sort of operation. This isn't that.
3. As you indicated, object and blob stores often persist and maintain a lot of metadata. In my experience, blob stores mostly just store it, but object stores will often act on it (e.g. checksum validation). No need to deal with any of that here, including several of the points brought up in your linked discussion (e.g. Content-Length, Content-Hash, ETag, and other metadata being used for other extraneous purposes).
4. We talked about my goal here to avoid having the SDKs deal with serialization here. An object or blob store often handles unstructured data in some format or another and I think we should absolutely create more specialized data stores that support operations more suited to one type or another (certainly could be useful from an agentic tooling and pluggable component perspective), but here, in the name of simplification and starting with a low threshold, I would like to put the responsibility on the developer for ensuring that their data can be serialized and encoded and have the API exclusively persist, retrieve and delete that data with no room for any other possibilities.
5. Object and blob store often support hierarchical or operational permissions structures such as append-only writes, write-only permissions (e.g. no deletion via API), etc. That's also intentionally excluded from consideration here.

Put more simply - those other stores anticipate the developer wanting to do both simple and far more advanced operations with their data. I'd certainly like to build more specialized data stores to accommodate such requirements, but this proposal seeks to do away with any complexities and do one thing really well: manage the reading, writing and deletion of large files in a resource-limiting and highly performant manner which is not possible in today's Dapr state management.

[olitomlinson]

Adding for how we might use this in Workflows for storing large activity inputs / outputs

[lindner]

Seems like the existing s3 and other existing bindings could be mapped. Have you tried a PoC?

[WhitWaldo]

Seems like the existing s3 and other existing bindings could be mapped. Have you tried a PoC?

@lindner The first step is proposing the shape of the block (as I've done here) and soliciting public feedback on the API shape and try to discern if anything else seems necessary within the described purpose of the API.

Out of the box, I'd certainly like to target support for Azure Blob Storage and provide an S3-compatible component (as this would facilitate connectivity with S3 itself, but also the many providers that offer S3-compatible APIs).

Next steps are getting tentative maintainer sign-off (no point building a POC if it's not going to be accepted) and then starting development of it - as I indicated in Discord, I intend to build this out as part of the next Dapr release (1.18).

[olitomlinson]

In the context of its usage for storing large activity inputs and outputs in Workflows, I would strongly recommend that this design allows a workflow author to programmatically choose the path/directory to the binary file.

This is to support multi-tenant use-cases where each tenants data MUST be stored in different locations.

/store/tenant-a/

/store/tenant-b/

/store/tenant-c/

Having this location set at the time of scheduling the workflow (not registering the workflow) gives a good level of flexibility.

---

builder.Services.AddDaprWorkflow(options =>
    {
         options.RegisterWorkflow<MyWorkflow>( BinaryStoreName = "my-binary-store");
    }
    
    ...
    var tenantId = "tenant-a";
    var workflowId = "2c0882d7";
    
    await workflowClient.ScheduleNewWorkflowAsync(
            name: nameof(MyWorkflow),
            instanceId: workflowId,
            input: orderInfo,
            InputOutputBinaryStorePath: $"/store/{tenantId}/wf/{workflowId}"
            );

In the example above, assuming we're using an S3 Binary Store, the Activity input / output blobs would be stored in the following location

/store/tenant-a/wf/2c0882d7/activity/{activity-id}/output/
/store/tenant-a/wf/2c0882d7/activity/{activity-id}/input/

There is an assumption that workflows have an implicit Activity Id which uniquely identifies each activity call. We use that Activity Id, in the path above.

---

Building on the above example, the Reference to the blob becomes {app-id}||{binary-store-name}||{location}||{file-id}

myApp||my-binary-store||/store/tenant-a/wf/2c0882d7/activity/123/input/xyz

The Reference is what is encoded in the Workflow History, rather than the blob contents.

The SDK can then dereference the data whenever the user demands it throughout the workflow. It may even be the case that the data is never dereferenced, until end of the Workflow when someone requests the output of the completed workflow, which maybe one (or more) large blobs!

[WhitWaldo]

In the context of its usage for storing large activity inputs and outputs in Workflows, I would strongly recommend that this design allows a workflow author to programmatically choose the path/directory to the binary file.

This is to support multi-tenant use-cases where each tenants data MUST be stored in different locations.

/store/tenant-a/

/store/tenant-b/

/store/tenant-c/

Having this location set at the time of scheduling the workflow (not registering the workflow) gives a good level of flexibility.

builder.Services.AddDaprWorkflow(options =>
    {
         options.RegisterWorkflow<MyWorkflow>( BinaryStoreName = "my-binary-store");
    }
    
    ...
    var tenantId = "tenant-a";
    var workflowId = "2c0882d7";
    
    await workflowClient.ScheduleNewWorkflowAsync(
            name: nameof(MyWorkflow),
            instanceId: workflowId,
            input: orderInfo,
            InputOutputBinaryStorePath: $"/store/{tenantId}/wf/{workflowId}"
            );

In the example above, assuming we're using an S3 Binary Store, the Activity input / output blobs would be stored in the following location

/store/tenant-a/wf/2c0882d7/activity/{activity-id}/output/ /store/tenant-a/wf/2c0882d7/activity/{activity-id}/input/

There is an assumption that workflows have an implicit Activity Id which uniquely identifies each activity call. We use that Activity Id, in the path above.

Building on the above example, the Reference to the blob becomes {app-id}||{binary-store-name}||{location}||{file-id}

myApp||my-binary-store||/store/tenant-a/wf/2c0882d7/activity/123/input/xyz

The Reference is what is encoded in the Workflow History, rather than the blob contents.

The SDK can then dereference the data whenever the user demands it throughout the workflow. It may even be the case that the data is never dereferenced, until end of the Workflow when someone requests the output of the completed workflow, which maybe one (or more) large blobs!

Might this instead be done more like how actors currently stores state in KVs? Set a path on the component at registration time that's used as the root and defer to the workflow to pick an appropriate path to save the reference to relative to the registration path? Presumably the runtime would pick a path referencing the workflow ID and any namespace values itself and then the user needn't figure out how to specify their own paths?

[acroca]

First of all, I really like this proposal and I'd like to see it move forward. The API itself looks right to me: minimal, streaming, no list or metadata. My comments below are only about the workflow use case discussed in this thread, and I know I'm diverging a bit from the direction of that discussion, so sorry in advance. I hope it's still useful.

About using this store for large activity inputs and outputs, with the SDKs storing the data and putting references in the workflow history (the helper tool idea, and the {app-id}||{store}||{location}||{file-id} format @olitomlinson suggested): I think that specific part should be handled transparently by the runtime instead. Those references would live in workflow histories forever, and every SDK would need to understand the format written by the others. From the SDK point of view, big payloads in workflows, activities and events should just flow normally, and the runtime should be the one optimizing to not store them in the state store. (There's some early work in that direction in dapr/dapr#10170.)

That said, I think this binary store can still give workflows something very useful, close to the helper tool @WhitWaldo described, but without references going through the workflow at all: a simple API on the activity context, something like ctx.StoreBinary(data, "myfile") and ctx.GetBinary("myfile"). Activities run only once (no replay), so these are plain calls, nothing special needed in the SDKs. The workflow body doesn't know about the blobs at all: activities that need "myfile" just use the name in their code, and data is only fetched when an activity asks for it.

The sidecar would store these blobs namespaced by workflow instance, similar to what @WhitWaldo suggested above about the runtime picking paths from the workflow ID and namespace. That also means the runtime can empty the namespace when the workflow is purged, so nothing leaks. And since the name is the whole contract, workflows can cross SDKs freely.

This could also cover the scheduling case. If I want to start a workflow with a 2GB file available from the beginning, instead of sending it as the input payload I could do something like .WithBinary("myfile", fileContents) when scheduling it. The file gets uploaded to the workflow's namespace first, and the workflow starts once it's there.

I believe this keeps the multi-tenant requirement covered too: paths derived from namespace and instance ID keep tenant data separated, and components are namespaced anyway. If someone needs a specific blob in a different store or location, they can always call the binary store API directly from an activity.

Two small things this would need:

• A way in this API to delete all files under a prefix, so the runtime can clean up a workflow's namespace on purge. Everything else is already covered by the proposed API.
• For the workflow use, a single binary store component marked in its config, the same way the actor state store works today (actorStateStore: true, only one allowed). The runtime resolves it, so nothing anywhere depends on a component name.

[olitomlinson]

In the context of workflows :

Ultimately, I really don't care which part of dapr is responsible for streaming the data into the binary store, as long as it is efficient, and the data goes where I say it goes :) but in the spirit of how dapr has always worked, I am slightly biased to having the runtime interfacing with the binary store target

and the data goes where I say it goes

I can't understate this enough.

In large Enterprise, folks want control over where their data goes based upon who the tenant is, and/or the sensitivity of the data.

Right now, there is no distinction between workflows Control plane data and workflows data plane data, but I think this is prime opportunity to make that distinction by introducing the InputOutputBinaryStorePath option when scheduling a workflow instance, which gives a clear separation to where all inputs and outputs are stored (not in the workflow history aka the workflow control plane data)

It allows me, the author, ensure that the sensitive bits of a workflow which may contain customer IP and secrets (Inputs and Outputs), are segregated, into some logical, well-known, structure in the binary store.

To go a step further, I would even prefer to be allowed to specify the binary store itself when scheduling a workflow for real physical segration of workflow activity inputs and outputs.

await workflowClient.ScheduleNewWorkflowAsync(
            name: nameof(MyWorkflow),
            instanceId: workflowId,
            input: orderInfo,
            InputOutPutBinaryStoreName: "my-binary-store",
            InputOutputBinaryStorePath: $"/store/{tenantId}/wf/{workflowId}"
            );

---

@acroca

a simple API on the activity context, something like ctx.StoreBinary(data, "myfile") and ctx.GetBinary("myfile"). Activities run only once (no replay), so these are plain calls, nothing special needed in the SDKs.

While I do agree that Property Bag approach this might provide value in some scenarios, this suggestion does not address the original problem which is how to transparently reference and deference large workflow Inputs and Outputs.

[lindner]

Off the cuff idea: use multiple data stores, one per tenant, then have a mechanism to map that via a policy. Having magic paths feels too tied to the storage implementation.

If needed a separate routable storage component could encapsulate multiple lower level storage components with a standard resolver mapping based on input criteria. This may avoid the confused deputy pattern

[WhitWaldo]

@acroca I think the workflow integration and how it works is probably better suited to a separate proposal, but..

I certainly agree it would be more performant if runtime entirely handled the process of identifying a large (potentially configurable) object and automatically saved to the binary store and persisted the data reference into the workflow history for precisely the reasons you said.

However, I think the SDK does need to have some visibility into the data references around how the data is retrieved back into the workflow or activities - this should ideally be loaded lazily at the developer's explicit discretion to avoid unnecessary round trips of reloading data that's constantly discarded between replays.

However, I disagree that the data should be stored by namespace because it feels like the initial use-case is now driving the shape of the parent API. Not all binary store providers support deletion by prefix, so while it might certainly be a helpful convenience, adding that to the API would likely subsequently limit the number of eligible providers.

Rather, I don't see any reason why the runtime, when purging state, can't walk through all the data references in the workflow history and just delete each blob specifically, or maintain a list of references in key/value store automatically and track there.

Further, as this would be a standalone building block, SDKs would be expected to offer API access to store/retrieve these files directly. Whether automatically done by the runtime when a workflow completes and returns or explicitly by the developer via the SDK, there should be a gRPC method available by the runtime (independent of this building block API) that provides for a copy operation between a workflow data reference to a specific path so workflow outputs survive history (and data ref) purges.

--

Again, doesn't really belong in this proposal, but I agree with @lindner that we should get away from the "one component for all things" mindset in the workflow configuration. It potentially hampers performance (different actors and workflows are all required to use the same storage instance across the entire Dapr implementation) and doesn't offer the flexibility I would expect end users to expect (and potentially enterprises to need) which would hamper adoption.

In something relevant to this specific proposal though, I'm certainly not opposed to reimagining the component metadata to reflect something like "availableForWorkflow: true" as a whitelisting option by the infra team and then have the developer specify the name of this component in Workflow options when the workflow is launched. That leaves developers free to use any similarly marked binary store component in different workflows. This speaks to @olitomlinson 's concerns in that part of the binary store block configuration might be an optional "prefix" path that's used when any blobs are written, letting infrastructure dictate where the block stores data (regardless of whether it's written by the SDK directly or the runtime via Workflows).

[olitomlinson]

@lindner

I appreciate your feedback, and yes, I generally agree that magic numbers and magic strings/paths are asking for trouble, but in this case, IIRC the Binary Store component would only ever target large object stores like S3 and Azure Blob Storage, which are path-based.

But if that's not the case, then yes, I fully agree with some kind layer between!

[lindner]

For prior art to look at us the Query api for state stores as a potential way to handle meta data mapping/lookup. A similar capability model could exist for blobs and alignment between these two componen types for naming/api surface would be useful