Add Azure Blob Storage Gen2 (ADLS HNS) support to BlobFileStore by Skrypt · Pull Request #19014 · OrchardCMS/OrchardCore

Skrypt · 2026-03-16T17:34:16Z

Make BlobFileStore adaptive: it now auto-detects whether the storage account has Hierarchical Namespace (HNS) enabled and uses native DataLake APIs for atomic moves, real directories, and efficient listing when available. Falls back to standard flat-namespace blob operations otherwise. HNS detection can be overridden via configuration.

All operations go through separate, unified API endpoints that adapt server-side

The Vue app calls the same endpoints regardless of storage backend:

Method	Gen2 (HNS) path	Gen1 (flat) path
`GetDirectoryInfoAsync`	`DataLakeFileSystemClient.GetDirectoryClient.ExistsAsync`	blob hierarchy listing
`GetDirectoryContentFlatAsync`	reads `hdi_isfolder` metadata	infers dirs from blob paths
`TryCreateDirectoryAsync`	`DataLakeDirectoryClient.CreateIfNotExistsAsync`	creates marker file
`TryDeleteDirectoryAsync`	`DataLakeDirectoryClient.DeleteAsync(recursive: true)`	iterates and deletes blobs individually
`MoveFileAsync`	`DataLakeFileClient.RenameAsync` (atomic)	`CopyFileAsync` + `TryDeleteFileAsync`

TODO

Implement a set of integration tests against BlobFileStore (Gen 1 and Gen2) with Azurite
Azurite needs to support ADLS; see: ADLS Gen 2 Azure/Azurite#2635

Make BlobFileStore adaptive: it now auto-detects whether the storage account has Hierarchical Namespace (HNS) enabled and uses native DataLake APIs for atomic moves, real directories, and efficient listing when available. Falls back to standard flat-namespace blob operations otherwise. HNS detection can be overridden via configuration. - Add IFileStoreCapabilities interface and FileStoreCapabilities impl - Add GetFilesAsync, GetDirectoriesAsync, Capabilities to IFileStore - Add UseHierarchicalNamespace option to BlobStorageOptions - Add Azure.Storage.Files.DataLake package dependency - Add MediaCreatedFileAsync and MediaCopiedFileAsync event hooks - Wire up EnsureCapabilitiesAsync at startup in Media.Azure module

Add integration tests that run against Azurite to verify BlobFileStore behavior in both flat-namespace (Gen1) and hierarchical-namespace (Gen2) modes. Tests cover capabilities detection, file CRUD, directory operations, move/copy, and directory content listing. Gen2 tests that use DataLake APIs will fail until Azurite supports the DFS endpoint (--dfsPort flag). This is intentional to track when a newer Azurite version enables full Gen2 testing. - Add abstract BlobFileStoreTestsBase with 26 test methods - Add BlobFileStoreGen1Tests and BlobFileStoreGen2Tests subclasses - Add AzuriteFactAttribute to skip tests when Azurite is unavailable - Add OrchardCore.FileStorage.AzureBlob reference to test project - Start Azurite in main_ci and pr_ci workflows on ubuntu runners

Add DfsEndpoint option to BlobStorageOptions for local emulators where the DFS endpoint runs on a separate port. Parse storage credentials from the connection string to construct the DataLakeServiceClient when an explicit DFS endpoint is configured. Fix Gen2 code paths to handle 404 RequestFailedException from DataLakePathClient.ExistsAsync, which can throw instead of returning false when the path does not exist.

Reverted both workflows back to docker run steps (instead of services) since that lets us pass --skipApiVersionCheck directly. The command now uses your image with the correct flags:

Piedone · 2026-03-21T02:32:01Z

I'll try to review this over the weekend.

Piedone

Will #14256 use the new APIs for better UX?

…dia startup Move BlobFileStore.EnsureCapabilitiesAsync() from the DI singleton factory (where it blocked via GetAwaiter().GetResult()) into MediaBlobContainerTenantEvents.ActivatingAsync(), which already runs asynchronously at tenant startup. BlobFileStore is now registered as its own singleton so it can be injected. Also remove the unused StorageProvider member from IFileStoreCapabilities and FileStoreCapabilities, and remove the default implementation of IFileStore.Capabilities so every provider must declare capabilities explicitly. FileSystemStore and AwsFileStore updated accordingly.

…s cleanup

Skrypt · 2026-03-29T16:16:34Z

    private readonly ILogger<FileSystemStore> _logger;
    private readonly string _fileSystemPath;

+    public IFileStoreCapabilities Capabilities { get; } = new FileStoreCapabilities(hasHierarchicalNamespace: false, supportsAtomicMove: false);


I'm changing this configuration to report that the local FileStorage uses HierarchicalNamespace and supports atomic move. It technically should not break any custom implementations.

…lesystem behavior FileSystemStore was configured with hasHierarchicalNamespace: false and supportsAtomicMove: false, which does not match its actual implementation. The local filesystem uses real Directory.* APIs for first-class directory operations and File.Move() which maps to an atomic OS rename syscall. Updated both flags to true to align with the implementation and be consistent with how BlobFileStore Gen2 reports the same native behaviors.

…tification Add a StorageName default interface member to IFileStore that returns a human-readable name for the underlying storage backend. Implementations: FileSystemStore ("Local"), BlobFileStore ("Azure Blob (Gen1)"/"Azure Blob (Gen2)" based on detected HNS), AwsFileStore ("Amazon S3"). DefaultMediaFileStore delegates to the inner store. This enables the media UI to display which storage provider is active at runtime, including distinguishing Azure Gen1 from Gen2.

…tegration project Extract BlobFileStore Gen1/Gen2 tests from OrchardCore.Tests into a new OrchardCore.Tests.Integration project. This separates integration tests that require external services (Azurite) from unit tests, avoiding unnecessary Docker container startup and extra execution time on every PR commit. The new project is built and run by a dedicated integration_tests.yml workflow that only triggers on review submission, push to main/release, or manual dispatch — matching the same pattern as functional_all_db.yml. Removed Azurite setup from pr_ci.yml and main_ci.yml.

github-actions · 2026-04-02T06:30:33Z

This pull request has merge conflicts. Please resolve those before requesting a review.

Skrypt · 2026-04-02T06:31:08Z

@Piedone Same here, moved everything to a new project and also new GH workflow.

github-actions · 2026-04-07T21:33:17Z

This pull request has merge conflicts. Please resolve those before requesting a review.

# Conflicts: # .github/workflows/integration_tests.yml # test/OrchardCore.Tests.Integration/OrchardCore.Tests.Integration.csproj

hishamco · 2026-04-17T12:18:31Z

As we discussed in the latest steering meeting to utilize TestContainers, this closed in favor of #19162

sebastienros · 2026-04-30T18:03:17Z

+    /// Default implementation filters <see cref="GetDirectoryContentAsync"/>. Implementations
+    /// may override this to avoid enumerating directories for better performance.
+    /// </remarks>
+    async IAsyncEnumerable<IFileStoreEntry> GetFilesAsync(string path = null)


This is supposed to be optimized in the implementations. For instance if a storage contains 1M files and 1 folder , we can now get the folder without listing the files.

However this is not implemented in concrete implementations.

I added the implementation details that are different in a table on the PR description. It will be easier for you to review that way.

sebastienros · 2026-04-30T18:08:57Z

+    /// Without this override there is no way to use HNS-dependent features (atomic moves, real
+    /// directory operations) in either of those scenarios.
+    /// </remarks>
+    public bool? UseHierarchicalNamespace { get; set; }


This may not be necessary. The BlobStore already checks the API availability and falls back to flat namespace. Is also uses this flag but the user shouldn't care.

it's still needed. The logic is that it is possible that the GetAccountInfo fails to return the storage type "Gen1 or Gen2". Then it is a fallback that will look into the BlobStorageOptions appsettings file to fallback to the value set manually.

The issue would be if the BlobStore would fail to retrieve the Gen type when doing GetAccountInfo because of account security. Then it would fall back to Gen1 for a Gen2 which would be really bad. Either we keep this fallback option here or we remove the automatic discovery of Gen type.

Skrypt · 2026-04-30T18:45:44Z

@copilot can you take a look at @sebastienros review / comments and check why we don't have anything specific about the IFileStoreCapabilities since we don't seem to make use of it in this PR.

Not necessary for this PR.

The Azurite fork now serves DFS requests on the same port as blob, matching how Azure Blob Storage works in production (single endpoint, different hostnames). DataLakeServiceClient is created directly from the connection string, eliminating the need for a separate DfsEndpoint. - Remove DfsEndpoint from BlobStorageOptions and BlobFileStore - Remove ParseCredentialsFromConnectionString (no longer needed) - Update integration tests: auto-detect HNS via GetAccountInfo for Gen2, use UseHierarchicalNamespaceOverride=false only for Gen1 - Update CI: single port per Azurite container, no separate DFS port Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…string Gen1 tests use UseHierarchicalNamespaceOverride=false against an HNS-enabled Azurite, which triggers the expected warning and forces flat-namespace code paths. Gen2 tests rely on auto-detection. No separate port or connection string needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Gen2 tests now create their containers using DataLakeFileSystemClient, which sends x-ms-namespace-enabled: true and stamps the container as an HNS filesystem — matching how Gen2 filesystems are created in production Azure. Gen1 tests continue using BlobContainerClient. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Gen1: TryCreateDirectoryAsync writes a marker blob (OrchardCore.Media.txt) to simulate directories in flat namespace storage. Gen2: TryCreateDirectoryAsync uses the DataLake API to create a real directory object — no marker blob is written. These tests assert the raw blob listing directly, proving Azurite is routing to the correct pipeline for each storage generation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Skrypt · 2026-04-30T23:56:14Z

I updated the Azurite ADLS pull request and the related docker image to clean up this PR.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Skrypt · 2026-05-01T15:01:25Z

@sebastienros I'm done with changes. Mostly needed to fix the Azurite emulator to simplify the code here. The remaining HierarchicalNamespace fields are there as a fallback for when the automatic discovery fails. This can happen when using a connection string to a specific container and that the storage account is protected. The GetAccountInfo() Azure blob storage / Azurite function could return a 403 since the user account would not be allowed to do a GetAccountInfo() to the root of the storage. Something like it ... keeping the fields was suggested by AI for that matter. Maybe it is completely wrong but we would need to do tests with a real Azure Blob Storage and see how it behaves with that use case.

I don't like needing an override from appsettings like this but that's a fallback because we have auto-discovery of the Gen type. I may investigate further when I get time.

Skrypt marked this pull request as draft March 16, 2026 17:54

Skrypt added 2 commits March 16, 2026 15:07

Skrypt mentioned this pull request Mar 17, 2026

ADLS Gen 2 Azure/Azurite#2635

Open

Skrypt added 3 commits March 18, 2026 04:11

Update unit tests after updating Azurite

7be25b8

Update GH workflows with temp Azurite package repo

71ced65

skipApiVersionCheck

256835f

Reverted both workflows back to docker run steps (instead of services) since that lets us pass --skipApiVersionCheck directly. The command now uses your image with the correct flags:

Skrypt requested a review from Piedone March 19, 2026 20:46

Merge branch 'main' into skrypt/media-azure-gen2

121557d

Piedone requested changes Mar 25, 2026

View reviewed changes

Skrypt added 3 commits March 28, 2026 01:34

Merge branch 'main' into skrypt/media-azure-gen2

03c0980

fix: Remove StorageProvider from test broken by IFileStoreCapabilitie…

7d16f4f

…s cleanup

Skrypt commented Mar 29, 2026

View reviewed changes

Skrypt added 5 commits March 29, 2026 12:19

fix build

aa477e6

add documentation on added configuration

56b294a

github-actions Bot added the merge conflict label Apr 2, 2026

Merge branch 'main' into skrypt/media-azure-gen2

7c01e5c

github-actions Bot added merge conflict and removed merge conflict labels Apr 2, 2026

Merge branch 'main' into skrypt/media-azure-gen2

3a44b79

# Conflicts: # .github/workflows/integration_tests.yml # test/OrchardCore.Tests.Integration/OrchardCore.Tests.Integration.csproj

github-actions Bot removed the merge conflict label Apr 14, 2026

hishamco mentioned this pull request Apr 15, 2026

Use TestContainers for AWS Local Stack #19157

Open

hishamco closed this Apr 17, 2026

Skrypt reopened this Apr 17, 2026

Skrypt marked this pull request as ready for review April 30, 2026 16:01

sebastienros reviewed Apr 30, 2026

View reviewed changes

Skrypt and others added 7 commits April 30, 2026 15:11

Remove IFileStorageCapabilities

c0c186b

Not necessary for this PR.

Fix CA2254: split ternary log message into separate LogInformation calls

fb6bcb6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge branch 'main' into skrypt/media-azure-gen2

d91f49b

Fix dotnet test syntax: use --project flag

f572bd2

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Conversation

Skrypt commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

All operations go through separate, unified API endpoints that adapt server-side

TODO

Uh oh!

Piedone commented Mar 21, 2026

Uh oh!

Piedone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Skrypt Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 2, 2026

Uh oh!

Skrypt commented Apr 2, 2026

Uh oh!

github-actions Bot commented Apr 7, 2026

Uh oh!

hishamco commented Apr 17, 2026

Uh oh!

Uh oh!

Uh oh!

sebastienros Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Skrypt Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sebastienros Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Skrypt Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Skrypt Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Skrypt commented Apr 30, 2026

Uh oh!

Skrypt commented Apr 30, 2026

Uh oh!

Skrypt commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Skrypt commented Mar 16, 2026 •

edited

Loading

Skrypt Mar 29, 2026 •

edited

Loading

Skrypt commented May 1, 2026 •

edited

Loading