Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/hub/storage-buckets-integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ from datasets import load_dataset
ds = load_dataset("buckets/username/my-bucket", data_files=["data.parquet"])
```

## Transformers

The [`Trainer`](https://huggingface.co/docs/transformers/trainer_recipes) can push and resume training checkpoints directly to a bucket, so a run can resume on a fresh machine without keeping checkpoints in a Git repo. See the [Trainer checkpointing docs](https://huggingface.co/docs/transformers/trainer_recipes) for setup.

## Filesystem operations

For direct file operations, `huggingface_hub` exposes a pre-instantiated [filesystem object](/docs/huggingface_hub/guides/hf_file_system), `hffs`:
Expand Down
3 changes: 3 additions & 0 deletions docs/hub/storage-buckets.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,9 @@ hf sync ./checkpoints hf://buckets/my-org/training-run-42/checkpoints

Because buckets are built on [Xet](./xet/index), successive checkpoints where large parts of the model are frozen benefit from chunk-level deduplication. Only the changed chunks are uploaded.

> [!TIP]
> Transformers' [`Trainer`](https://huggingface.co/docs/transformers/trainer_recipes) can push and resume training checkpoints directly to a bucket — no manual `sync` step needed. See the Trainer docs for setup.

### Data processing pipelines

Buckets serve as staging areas for data processing workflows. Process raw data, write intermediate outputs to a bucket, then promote the final artifact to a versioned [Dataset](./datasets) repository when the pipeline completes. This keeps your versioned repo clean while giving your pipeline fast mutable storage.
Expand Down
Loading