Use connection pool for faster download#287
Draft
hagenw wants to merge 5 commits into
Draft
Conversation
Contributor
Reviewer's GuideIntroduces configurable HTTP connection pooling for Artifactory and Minio backends (with sensible defaults, validation, and tests) to speed up bulk parallel downloads while keeping behavior robust for invalid or missing config. Sequence diagram for pooled connection setup and file download (Artifactory backend)sequenceDiagram
actor Worker
participant BackendArtifactory as Artifactory
participant Config as ArtifactoryConfigFile
participant Requests as RequestsSession
participant Adapter as HTTPAdapter
participant Server as ArtifactoryServer
Worker->>BackendArtifactory: create_backend(host, repository)
activate BackendArtifactory
BackendArtifactory->>Config: get_config(host)
alt config_file_exists_and_section_present
Config-->>BackendArtifactory: {pool_connections, pool_maxsize, max_retries}
else missing_or_invalid_config
Config-->>BackendArtifactory: {}
end
BackendArtifactory->>BackendArtifactory: _parse_int(pool_connections)
BackendArtifactory->>BackendArtifactory: _parse_int(pool_maxsize)
BackendArtifactory->>BackendArtifactory: _parse_int(max_retries)
BackendArtifactory->>Requests: Session()
Requests-->>BackendArtifactory: session
BackendArtifactory->>Adapter: HTTPAdapter(pool_connections, pool_maxsize, max_retries)
Adapter-->>BackendArtifactory: adapter
BackendArtifactory->>Requests: mount("https://", adapter)
BackendArtifactory->>Requests: mount("http://", adapter)
BackendArtifactory-->>Worker: backend_ready_with_pooled_session
deactivate BackendArtifactory
par multiple_workers_download_in_parallel
Worker->>BackendArtifactory: download_file(path)
BackendArtifactory->>Requests: GET path (reuses pool connection)
Requests->>Server: HTTPS request
Server-->>Requests: response
Requests-->>BackendArtifactory: response
BackendArtifactory-->>Worker: file_content
and
Worker->>BackendArtifactory: download_file(other_path)
BackendArtifactory->>Requests: GET other_path (reuses pool connection)
Requests->>Server: HTTPS request
Server-->>Requests: response
Requests-->>BackendArtifactory: response
BackendArtifactory-->>Worker: file_content
end
Class diagram for updated Artifactory and Minio backends with connection poolingclassDiagram
class Base {
}
class Artifactory {
- str host
- str repository
- requests.Session _session
- repo _repo
+ get_authentication(host: str) str str
+ get_config(host: str) dict
- _open() None
- _checksum(path: str, checksum: str, label: str) str
- _upload_file(path: str, file: str, label: str) None
- _remove_file(path: str, label: str) None
}
class Minio {
- str host
- str bucket
- minio.Minio _client
+ __init__(host: str, bucket: str, secure: bool, kwargs) None
+ get_config(host: str) dict
- _upload_file(path: str, file: str, label: str) None
- _remove_file(path: str, label: str) None
}
class HTTPAdapter {
+ HTTPAdapter(pool_connections: int, pool_maxsize: int, max_retries: int)
}
class PoolManager {
+ PoolManager(timeout, num_pools: int, maxsize: int, block: bool)
}
class _parse_int_artifactory {
+ _parse_int(value: str|int, name: str, default: int) int
}
class _parse_int_minio {
+ _parse_int(value: str|int, name: str, default: int) int
}
class _parse_bool {
+ _parse_bool(value: str|bool, name: str, default: bool) bool
}
Base <|-- Artifactory
Base <|-- Minio
Artifactory --> HTTPAdapter : uses
Artifactory --> _parse_int_artifactory : uses
Minio --> PoolManager : uses
Minio --> _parse_int_minio : uses
Minio --> _parse_bool : uses
Architecture diagram for backends using HTTP connection poolsflowchart LR
App["Application<br>(audb client, download workers)"] -->|uses| BackendArtifactory["Artifactory backend"]
App -->|uses| BackendMinio["Minio backend"]
BackendArtifactory -->|config via get_config| ArtifactoryConfig["Artifactory config file<br>~/.config/audbackend/artifactory.cfg"]
BackendMinio -->|config via get_config| MinioConfig["Minio config file<br>~/.config/audbackend/minio.cfg"]
BackendArtifactory -->|creates| RequestsSession["requests.Session<br>with HTTPAdapter(pool_connections, pool_maxsize, max_retries)"]
BackendMinio -->|creates| PoolManager["urllib3.PoolManager<br>(timeout, num_pools, maxsize, block)"]
RequestsSession -->|reused HTTP connections| ArtifactoryService["Artifactory server"]
PoolManager -->|reused HTTP connections| MinioService["MinIO server"]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
hagenw
commented
Feb 23, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
🚀 New features to boost your workflow:
|
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Member
Author
|
This has not shown a big boost in performance, so I will set it to draft for now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses audeering/audb#546
This uses the proposal from audeering/audb#546 3b and uses pooled connections to avoid having to establish a HTTPS connection for every single file that it will download. Currently, max connection is set to 10 and can be changed in the config file (maybe we should increase default to 100?).
I did a benchmark on a dataset with ~1,500,000 files and the remaining time after 5 minutes into the download with 6 workers is reduced from 175:40:24 hours to 167:05:28 hours.
I will provide another benchmark with 48 workers at a later stage.A first test with 48 workers and max connection of 100 did not show any improvement. In both cases the time was ~155 hours.It also adds the
get_config()method toaudbackend.backend.Artifactoryto be in line withaudbackend.backend.Minio. The config file is still stored at a different place, but this will be handled in a different pull request (#288).