Add article: How to Handle Bulk Uploads in Qdrant#2489
Conversation
New production-ops article covering bulk upload best practices in Qdrant: on-disk vector storage, payload indexes, quantization, sparse-index on-disk storage, batching, parallelization, and sharding — for both dense and sparse vectors. Includes: - content/articles/bulk-uploads-in-qdrant.md - Cover / preview / social images (preview/) - Seven flow diagrams (one per option) illustrating the recommended vs default path for each strategy Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
✅ Deploy Preview for condescending-goldwasser-91acf0 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
@jkupchanko thanks for creating the article preview images, however, let me do these in the future so I can ensure they're on-brand with the others. I've updated those images and will start reviewing the article content now. |
kanungle
left a comment
kanungle
left a comment
There was a problem hiding this comment.
Nice work @jkupchanko. Good first draft.
I've left some requested changes. Images should not be embedded as html, and code snippets can be more concise. Some other small tweaks as well.
@generall is this aligned with your expectations of our revised advice?
| Before we get into the best practices, it's important to remember that not all vectors behave the same way during ingestion. Dense and sparse vectors use different indexing approaches, which means they can create different performance considerations during bulk uploads. | ||
|
|
||
| Let's quickly break down the difference before moving into the recommended upload strategies. |
There was a problem hiding this comment.
move this to the beginning of next section
|
|
||
| Sparse vectors use a separate indexing approach, and the sparse index is updated as points are written. This means sparse vector ingestion should not be treated the same way as dense HNSW indexing. | ||
|
|
||
| This difference matters because the right bulk upload strategy depends on the type of vectors being uploaded and the indexing work Qdrant has to handle during ingestion. |
|
|
||
| ## Choosing the Right Bulk Upload Strategy | ||
|
|
||
| Before we go through the best practices, understand there is no single configuration that works best for every bulk upload. The right approach depends on what you are trying to improve: upload speed, memory usage, search availability, or a balance of all three. |
There was a problem hiding this comment.
there is no single configuration that works best fore every bulk upload
make bold
| from qdrant_client import QdrantClient, models | ||
|
|
||
| collection_name = "my_collection" | ||
|
|
||
| client = QdrantClient(url="http://localhost:6333") |
There was a problem hiding this comment.
this part of code not needed
| from qdrant_client import QdrantClient, models | ||
|
|
||
| collection_name = "my_collection" | ||
|
|
||
| client = QdrantClient(url="http://localhost:6333") | ||
|
|
There was a problem hiding this comment.
This part of code not needed
| from qdrant_client import QdrantClient, models | ||
|
|
||
| collection_name = "my_collection" | ||
|
|
||
| client = QdrantClient(url="http://localhost:6333") | ||
|
|
||
| client.create_collection( | ||
| collection_name=collection_name, | ||
| vectors_config=models.VectorParams( | ||
| size=768, | ||
| distance=models.Distance.COSINE, | ||
| on_disk=True, | ||
| ), | ||
| ) | ||
|
|
||
| points = [ | ||
| models.PointStruct( | ||
| id=i, | ||
| vector=[0.1] * 768, | ||
| payload={"category": "example"}, | ||
| ) | ||
| for i in range(1000) | ||
| ] |
There was a problem hiding this comment.
This part of code not needed
| from qdrant_client import QdrantClient, models | ||
|
|
||
| collection_name = "my_collection" | ||
|
|
||
| client = QdrantClient(url="http://localhost:6333") | ||
|
|
There was a problem hiding this comment.
This part of code not needed
| from qdrant_client import QdrantClient, models | ||
|
|
||
| collection_name = "my_collection" | ||
|
|
||
| client = QdrantClient(url="http://localhost:6333") | ||
|
|
There was a problem hiding this comment.
This part of code not needed
|
|
||
| The safest approach is to choose the right strategy for the workload instead of relying on one universal configuration. Dense vectors, sparse vectors, and hybrid setups can all create different performance considerations during ingestion. | ||
|
|
||
| By designing the collection and upload process before ingestion starts, you can make bulk uploads more efficient, more stable, and easier to scale as your dataset grows. For help sizing your deployment, see the [Capacity Planning guide](/documentation/capacity-planning/). |
There was a problem hiding this comment.
For help sizing your deployment, see the Capacity Planning guide.
Don't reference this. It kind of sucks. Instead point to https://sizing.qdrant.tech
|
|
||
| <strong style="color:#ff9800;">⚠ Watch for:</strong> More shards are not always better. Each shard adds overhead, so the shard count should match the size of the deployment and the amount of write parallelism you actually need. | ||
|
|
||
| ## Choosing the Right Mix |
There was a problem hiding this comment.
This is a good start for offering clear guidance and takeaways, but I feel it's still not solving my needs in figuring out what exactly to do. Maybe we point to Qdrant Skills here?
- Convert all diagram embeds from HTML <img> to markdown image syntax - Trim code snippets to the focal call (drop client/collection boilerplate; drop the points-generation block in the batching example) - Move the "break down the difference" line to open the Vector Type section - Remove the redundant closing sentence in Why Vector Type Matters - Bold "there is no single configuration that works best for every bulk upload" - Replace the Capacity Planning link with the Qdrant sizing calculator (sizing.qdrant.tech) - Add a pointer to Qdrant Skills for scenario-specific guidance Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Thanks for the review @kanungle! Addressed everything in
On the guidance point (comparison-table section): I added a pointer to Qdrant Skills. Happy to go further if it's still not concrete enough — e.g. an explicit "if you're unsure, start here" default recommendation. Let me know what you'd prefer. Ready for another look when you have a chance. |
kanungle
left a comment
kanungle
left a comment
There was a problem hiding this comment.
A couple very minor changes. I would also give @abdonpijpelink and @generall a chance to take a look. Let's plan on merging monday/tuesday
|
|
||
| ## Why Vector Type Matters | ||
|
|
||
| Let's quickly break down the difference before moving into the recommended upload strategies. |
There was a problem hiding this comment.
Sorry, go ahead and move this line back to where you had it. Or you move "Before we get into..." sentence into this section too (which was my original intended feedback)
| | One upload worker is not enough | Option 6: Parallelize Uploads | Option 7: Multiple Shards | Too much parallelism can create pressure on CPU, memory, disk I/O, and network resources. | | ||
| | Very large dataset with high write volume | Option 7: Multiple Shards | Option 5: Batch Uploads, Option 6: Parallelize Uploads | More shards add overhead, so shard count should match the deployment size and write parallelism needed. | | ||
|
|
||
| Still deciding exactly what to configure for your workload? [Qdrant Skills](https://skills.qdrant.tech) provides hands-on, scenario-based guidance that walks you through the specific settings for your situation. |
There was a problem hiding this comment.
Point to "Qdrant's Agent Skills" and this link: https://qdrant.tech/documentation/skills/
|
|
||
|  | ||
|
|
||
| <div style="margin:1.5rem 0; padding:16px 18px; border:1px solid rgba(38,166,154,0.35); border-left:3px solid #26a69a; border-radius:8px; background:rgba(38,166,154,0.08);"><div style="font-size:11px; font-weight:700; letter-spacing:0.08em; text-transform:uppercase; color:#4db6ac; margin-bottom:6px;">📊 Benchmark</div>In testing with 10,000 768-dim vectors, batching at <strong>64 points per request</strong> was <strong style="color:#4db6ac;">~5× faster</strong> than uploading one point at a time.</div> |
There was a problem hiding this comment.
I'm not sure we can use this styling. @abdonpijpelink can you opine?
There was a problem hiding this comment.
In the end it's up to DevRel how you style the articles, but it seems inconsistent with the other articles (and docs). Also, I'm not sure these colors are on-brand?
|
@abdonpijpelink @generall if you have time or preference for reviewing, I've added you to the PR. It looks good to me though and will plan on merging on monday/tuesday |
There was a problem hiding this comment.
Left a couple of comments.
A couple of more general remarks:
- The diagrams do not follow the diagram best practices (see Figma). Most importantly, elements in diagrams should use colors of the same shade, e.g. if box background is blue - text should be dark blue. Here, the diagrams use black text on a colored background. I'm also not sure they're using the Qdrant color palette?
- I feel the options are presented in a somewhat random order? While quantization is a useful tool to reduce memory pressure, should it come before parallelization and batching in an article about bulk uploading?
|
|
||
|  | ||
|
|
||
| <div style="margin:1.5rem 0; padding:16px 18px; border:1px solid rgba(38,166,154,0.35); border-left:3px solid #26a69a; border-radius:8px; background:rgba(38,166,154,0.08);"><div style="font-size:11px; font-weight:700; letter-spacing:0.08em; text-transform:uppercase; color:#4db6ac; margin-bottom:6px;">📊 Benchmark</div>In testing with 10,000 768-dim vectors, batching at <strong>64 points per request</strong> was <strong style="color:#4db6ac;">~5× faster</strong> than uploading one point at a time.</div> |
There was a problem hiding this comment.
In the end it's up to DevRel how you style the articles, but it seems inconsistent with the other articles (and docs). Also, I'm not sure these colors are on-brand?
|
|
||
|  | ||
|
|
||
| In Python, scalar quantization can be configured when creating the collection: |
There was a problem hiding this comment.
Maybe use TurboQuant instead of scalar?
- Move the "Before we get into..." sentence into the Why Vector Type Matters section (per Neil's clarified intent) - Replace all custom-styled inline HTML (vector-type badges, best-fit/ watch-for callouts, benchmark boxes) with standard markdown for consistency with other articles and docs - Point to "Qdrant's Agent Skills" at qdrant.tech/documentation/skills/ - Switch Option 3 example from scalar quantization to TurboQuant Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
I believe Thierry previously used this formatting which was approved, so I think it's okay. That being said, @jkupchanko you can do a quick update on colors with this reference
I did not intend to be that detailed on the diagrams for font colors. Thought black should be okay enough. That being said, @jkupchanko please feed these diagrams to ChatGPT then use it to restyle your current images. Should take 5-10 mins. If those new diagrams still come out wrong, try feeding this as a source style.
Yes, absolutely. This is the feedback I was looking for. I feel overall this article has a lot of methods but no strong prescription on when and how to use them. I think we need more guidance here from the solutions architect team, but in the absence of that, we can do our own tests (@jkupchanko to run). My other thought was to use skills to advise on different scenarios and how to configure bulk uploads, but in order to do that, we first need to understand the right content to go into skills, which, again, likely require our own tests. Ideally, if this knowledge exists within qdrant, we would love to get a download of that information into our brains before running the tests. |
|
Overall, we're trying to get as close as we can to providing some kind of guidance, even if we're not 100% prescriptive. I'm not sure if tests alone can solve for this or if we just need to think about logically orienting the structure of this article better. In particular, this section leaves a lot to be desired for me. @jkupchanko let's discuss |
Swap the "Choosing the Right Mix" comparison table for a top-down decision-tree image that walks readers from vector type to a concrete storage/quantization/sharding configuration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… practices Replace all seven strategy diagrams with updated versions, and group batching, parallelization, and sharding under "Best Practices for Every Upload" since they apply to any bulk upload rather than being situational options. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@kanungle updated since your last review:
Ready for another look when you get a chance — thanks! |
There was a problem hiding this comment.
Good round of fixes on the images, code trimming, and the sizing calculator link — thanks. A few things before this ships:
-
Unpublishing
indexing-optimization.mdneeds a redirect. Setting it todraft: truewill 404/articles/indexing-optimization/and drop its backlinks. Have Claude Code update the redirect file links. -
Frontmatter fixes to match other articles:
dateshould be a full timestamp (2026-07-14T00:00:00.000Z, see indexing-optimization.md), and you're missingauthor_linkandkeywords. Description is 159 chars; keep it under 140. Try: "Plan bulk uploads in Qdrant: batching, parallelization, sharding, payload indexes, quantization, and on-disk storage." -
Decision-tree diagram (
choosing-the-right-mix.png): the logic is right, but every box is rendering empty — each label floats outside its blank box. Moving the existing text inside the boxes fixes it:- Root box: "Uploading Bulk Data"
- Branch heads: "Dense Vectors" / "Sparse Vectors" / "Both (Sparse + Dense)" inside the three top boxes
- Decision boxes get their questions: "Memory issues?", "Keep full precision?", "How aggressive?", "Still too large?", "Heavy writes?", "Multi-node?"
- Outcome boxes get their results: "In RAM (on_disk=False)", "On disk (on_disk=True)", "TurboQuant 4-bit (~8x smaller)", "TurboQuant 1-bit (up to 32x smaller)", "Index in RAM", "Index on disk", "On disk + fp16 (datatype=float16)", "One shard (shard_number=1)", "Pre-shard (shard_number >= 2)", "Shard on cluster + replication"
While you're in there: keep one Yes/No legend (there's a chip version top-right and a "Legend:" row at the bottom), and crop out the "…" button baked into the top-right corner.
-
Three images need compression: option5-batching.png (1.1MB), option1-memory.png (1.0MB), and choosing-the-right-mix.png (940KB). Run them through compressor.io — target under 250KB each. The other five are fine.
-
Benchmark callouts need context. Add environment and Qdrant version to both. 10k vectors is small for an article about bulk scale; if re-running at 1M isn't practical, qualify that gains grow with scale instead.
-
TurboQuant snippet: show
bits=since the decision tree recommends 4-bit vs 1-bit; right now the code doesn't match the diagram. -
Two additions would round it out: a short note on
prefer_grpc=True(gRPC is meaningfully faster for bulk upload) and a line on checking collection status is green before serving traffic. Also drop the repeated "In Python, this can look like:" lead-in; it's verbatim five times.
After the redirect + frontmatter fixes this is good to merge.
3 participants
🔗 Preview this change
View the article on the deploy preview →
https://deploy-preview-2489--condescending-goldwasser-91acf0.netlify.app/articles/bulk-uploads-in-qdrant/
PREVIEW
Summary
New production-ops article: How to Handle Bulk Uploads in Qdrant.
It walks through best practices for ingesting large datasets into Qdrant — on-disk vector storage, payload indexes, quantization, on-disk sparse indexes, batching, parallelization, and sharding — for both dense and sparse vectors, with a "Choosing the Right Mix" comparison table at the end.
What's included
content/articles/bulk-uploads-in-qdrant.mdstatic/articles_data/bulk-uploads-in-qdrant/preview/)Review
👋 @kanungle — this is ready for your review. The Netlify Deploy Preview on this PR will render the full article on the site so you can read it in context (cover, diagrams, code, and table).
Notes
qdrant-clientAPIs.preview_dirconvention.🤖 Generated with Claude Code