Skip to content

Warn users about file carving size limits #47376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
noahtalerman wants to merge 9 commits into
base: main
Choose a base branch
from
+38 −39
Draft

Warn users about file carving size limits #47376

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
cf63e57
Update rest-api.md
noahtalerman Jun 10, 2026
53d0afa
Create file-carving.md
noahtalerman Jun 10, 2026
1d05837
Apply suggestion from @noahtalerman
noahtalerman Jun 10, 2026
04f9768
Update file-carving.md
noahtalerman Jun 10, 2026
fc9a50d
Apply suggestion from @noahtalerman
noahtalerman Jun 10, 2026
4e6a711
Apply suggestion from @noahtalerman
noahtalerman Jun 10, 2026
ad97563
Delete docs/Contributing/product-groups/orchestration/file-carving.md
noahtalerman Jun 10, 2026
08178fc
Apply suggestion from @noahtalerman
noahtalerman Jun 10, 2026
7003ef2
Update file-carving.md
noahtalerman Jun 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 33 additions & 34 deletions ...duct-groups/orchestration/file-carving.md → articles/file-carving.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,57 +1,50 @@
## File carving
# File carving

Fleet supports osquery's file carving functionality as of Fleet 3.3.0. This allows the Fleet server to request files (and sets of files) from Fleet's agent (fleetd) returning the full contents to Fleet.
Fleet supports file carving which allows you to request files (and sets of files) and their full contents from hosts.

File carving data can be either stored in Fleet's database or to an external S3 bucket. For information on how to configure the latter, consult the [configuration docs](https://fleetdm.com/docs/deploying/configuration#s-3-file-carving-backend).

### Configuration
## Setup

Given a working flagfile for connecting fleetd to Fleet, add the following flags to enable carving:
In your agent configuration, add the following [command line flags](https://fleetdm.com/docs/configuration/agent-configuration#options-and-command-line-flags) to enable carving:

```sh
--disable_carver=false
--carver_disable_function=false
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block
--carver_block_size=8000000
```yaml
command_line_flags:
disable_carver=false
carver_disable_function=false
carver_start_endpoint=/api/v1/osquery/carve/begin
carver_continue_endpoint=/api/v1/osquery/carve/block
carver_block_size=8000000
```

The default flagfile provided in the "Add new host" dialog also includes this configuration.

#### Carver block size

The `carver_block_size` flag should be configured in osquery.

For the (default) MySQL Backend, the configured value must be less than the value of
For the (default) MySQL Backend, the configured `carver_block_size` must be less than the value of
`max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for [MySQL 8](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_allowed_packet) it is 64MB.

For the S3-compatible backend, this value must be set to at least 5MiB (`5242880`) due to the
For the S3-compatible backend, `carver_block_size` must be set to at least 5MiB (`5242880`) due to the
[constraints of S3's multipart
uploads](https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html).

#### Compression

Compression of the carve contents can be enabled with the `carver_compression` flag in osquery. When used, the carve results will be compressed with [Zstandard](https://facebook.github.io/zstd/) compression.
Compression of the carve contents can be enabled with the `carver_compression` flag. When used, the carve results will be compressed with [Zstandard](https://facebook.github.io/zstd/) compression.

### Usage
## Create carves

@noahtalerman noahtalerman Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> TODO warning about max carve size and number of carves requested at once to avoid performance issues

@noahtalerman noahtalerman Jun 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rfairburn when you get the chance, can you please help me come up w/ a good warning here?

So shackleton and other customers know the best practice max: https://fleetdm.slack.com/archives/C094ZAWGDHC/p1779392158155149

cc @zayhanlon

@rfairburn rfairburn Jun 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case you refer to there were a couple of things that at first glance happened that didn't seem intentional

  • all hosts got targeted for the carve instead of a narrow target set
  • the paths carved had wildcard paths in places such as /tmp or user homes that can sometime have many large files

The net effect was a load both in bandwidth and db usage that came from an X * Y * Z * A situation where:

X = number of hosts
Y = number of paths requests
Z = number of files in the paths
A = avg file size of any individual file

Any one of these can be bigger in scope usually without there being a huge issue for example:

  • you can target a large number of hosts for a single small file
  • you can look in several paths if you are trying to find a file/filename that only exists in 1-2 of said paths
  • there can be a higher-quantiy group of small files in a single path for a single or handul of hosts
  • a single large file (up to 8gb I believe) can easily be qeried

However, when all four of the dials end up being turned up (on accident in this case from the sounds of it), you suddenly end up with the db trying to write millions of new rows of metatada, while fleet is trying to populate 10s of terrabytes of data in the s3 carves bucket, while all hosts are both responding to the distributed query that does the SELECT to trigger the carve and push all of this data over the carves endpoints.

Another gotcha that did not occur in this specific case is the carves in question were only a one-off. There wasn't a scheduled query/automations=on situation, which can also compound.

The only nuance is that a particular carve select with multiple files would get combined into a single tar archive before being transferred, but we would still be subject to the 8GB size, block size of 256mb (row and s3 object per block), and the 4th-power load amplification if all factors get scaled at once.

File carves are initiated with osquery queries. Issue a query to the `carves` table, providing `carve = 1` along with the desired path(s) as constraints.
File carves are initiated with live reports. Run live report using the `carves` table, providing `carve = 1` along with the desired path(s) as constraints.

For example, to extract the `/etc/hosts` file on a host with hostname `mac-workstation`:

```sh
fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path = "/etc/hosts"'
```

The standard osquery file globbing syntax is also supported to carve entire directories or more:
The globbing syntax is also supported to carve entire directories or more:

```sh
fleetctl query --hosts mac-workstation --query 'SELECT * FROM carves WHERE carve = 1 AND path LIKE "/etc/%%"'
```

#### Retrieving carves
## Retrieve carves

List the non-expired (see below) carves with `fleetctl get carves`. Note that carves will not be available through this command until osquery checks in to the Fleet server with the first of the carve contents. This can take some time from initiation of the carve.
List the non-expired (see below) carves with `fleetctl get carves`. Note that carves will not be available through this command until Fleet's agent (fleetd) checks in to the Fleet server with the first of the carve contents. This can take some time from initiation of the carve.

To also retrieve expired carves, use `fleetctl get carves --expired`.

Expand All @@ -69,13 +62,13 @@ It can also be useful to pipe the results directly into the tar command for unar
fleetctl get carve --stdout 3 | tar -x
```

#### Expiration
## Expiration

Carve contents remain available for 24 hours after the first data is provided from the osquery client. After this time, the carve contents are cleaned from the database and the carve is marked as "expired".
Carve contents remain available for 24 hours after the first data is provided from the Fleet's agent (fleetd). After this time, the carve contents are cleaned from the database and the carve is marked as "expired".

The same is not true if S3 is used as the storage backend. In that scenario, it is suggested to setup a [bucket lifecycle configuration](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) to avoid retaining data in excess. Fleet, in an "eventual consistent" manner (i.e. by periodically performing comparisons), will keep the metadata relative to the files carves in sync with what it is actually available in the bucket.

### Alternative carving backends
## Alternative carving backends

#### RustFS

Expand All @@ -89,11 +82,11 @@ Configure the following:

If you're testing file carving locally with the docker-compose environment, the `--dev` flag on Fleet server will automatically point carves to the local RustFS container and write to the `carves-dev` bucket (created automatically) without needing to set additional configuration.

### Troubleshooting
## Troubleshooting

#### Check carve status in osquery
### Check carve status

Osquery can report on the status of carves through queries to the `carves` table.
You can report on the status of carves through queries to the `carves` table.

The details provided by

Expand All @@ -103,9 +96,9 @@ fleetctl query --labels 'All Hosts' --query 'SELECT * FROM carves'

can be helpful to debug carving problems.

#### Ensure `carver_block_size` is set appropriately
### Ensure `carver_block_size` is set appropriately

`carver_block_size` is an osquery flag that sets the size of each part of a file carve that osquery
`carver_block_size` is an option that sets the size of each part of a file carve that Fleet's agent (fleetd)
sends to the Fleet server.

When using the MySQL backend (default), this value must be less than the `max_allowed_packet`
Expand All @@ -121,3 +114,9 @@ The value must be small enough that HTTP requests do not time out.
Start with a default of 2MiB for MySQL (2097152 bytes), and 5MiB for S3 (5242880 bytes).

<meta name="pageOrderInSection" value="2100">

<meta name="articleTitle" value="File carving in Fleet">
<meta name="authorFullName" value="Noah Talerman">
<meta name="authorGitHubUsername" value="noahtalerman">
<meta name="publishedOn" value="2026-06-10">
<meta name="description" value="Learn how to TODO">
10 changes: 6 additions & 4 deletions docs/REST API/rest-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1311,13 +1311,15 @@ None.
- [Get carve](#get-carve)
- [Get carve block](#get-carve-block)

Fleet supports osquery's file carving functionality as of Fleet 3.3.0. This allows the Fleet server to request files (and sets of files) from Fleet's agent (fleetd), returning the full contents to Fleet.
> TODO: Warning on size of carve
Comment thread
noahtalerman marked this conversation as resolved.
Outdated

To initiate a file carve using the Fleet API, you can use the [live query](#run-live-query) endpoint to run a query against the `carves` table.
File carving allows you to request files (and sets of files) and their full contents from hosts.

Keep in mind that any failure when uploading a file block (like a network error) will result on a failed carved file. Starting in osquery v5.22.1, block uploads will be retried up to three times before failing.
To initiate a file carve using the Fleet API, use the [live report](#run-live-report) endpoint to run a query against the `carves` table.

For more information on executing a file carve in Fleet, go to the [File carving with Fleet docs](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/product-groups/orchestration/file-carving.md).
Any failure when uploading a file block (like a network error) will result on a failed carved file. Block uploads are retried up to three times before failing.

To learn more about executing a file carve in Fleet, head to the [file carving guide](https://fleetdm.com/guides/file-carving).

### List carves

Expand Down
Loading