diff --git a/qdrant-landing/content/articles/vector-search-filtering.md b/qdrant-landing/content/articles/vector-search-filtering.md index 7a329e03f0..052b256714 100644 --- a/qdrant-landing/content/articles/vector-search-filtering.md +++ b/qdrant-landing/content/articles/vector-search-filtering.md @@ -1,26 +1,37 @@ --- title: "A Complete Guide to Filtering in Vector Search" -short_description: "Merging different search methods to improve the search quality was never easier" -description: "Learn everything about filtering in Qdrant. Discover key tricks and best practices to boost semantic search performance and reduce Qdrant's resource usage." +short_description: "How Qdrant combines semantic search with payload-level constraints to deliver precise, efficient results" +description: "How filtering works in Qdrant: the filterable HNSW index, payload indexing, cardinality, and the conditions behind precise search." preview_dir: /articles_data/vector-search-filtering/preview social_preview_image: /articles_data/vector-search-filtering/social-preview.png weight: 70 -author: Sabrina Aquino, David Myriel -author_link: -date: 2024-09-10T00:00:00.000Z +author: Sabrina Aquino, David Myriel, Ewa Szyszka +author_link: https://github.com/sabrinaaquino +date: 2026-07-02T10:00:00+02:00 +draft: false category: mastering-search +tags: + - vector search + - filtering + - payload indexing + - HNSW --- -Imagine you sell computer hardware. To help shoppers easily find products on your website, you need to have a **user-friendly [search engine](https://qdrant.tech)**. -![vector-search-ecommerce](/articles_data/vector-search-filtering/vector-search-ecommerce.png) +Vector search is a ranking system, not a filter system. Every point in your collection is a candidate, which based on the semantic similarity to the query would get surfaced. That works well when you want "what's most relevant." It becomes insufficient when the result must also satisfy hard constraints: exact values, ranges, categorical membership, or payload-level conditions that similarity alone cannot enforce. - If you’re selling computers and have extensive data on laptops, desktops, and accessories, your search feature should guide customers to the exact device they want - or at least a **very similar** match. +Filtering is Qdrant's mechanism for combining semantic retrieval with predicate-based constraints. This guide covers how it works, when to use it, and how to configure indexing to keep filtered queries fast and accurate. -When storing data in Qdrant, each product is a point, consisting of an `id`, a `vector` and `payload`: +Take an e-commerce scenario where a customer searches for a budget computer. Vector search alone surfaces the most semantically similar results, but similarity has no concept of price. Without a payload filter, a $1,299 laptop ranks second simply because its embedding is close to the query. Add a `price ≤ $1,000` filter and Qdrant evaluates the constraint before scoring: over-budget candidates are excluded entirely, never ranked, never returned. The result set is both relevant and correct. + +![A budget query where an over-budget laptop is excluded by a price filter before scoring](/articles_data/vector-search-filtering/budget-query-example.png) + +## Data Model: Points, Vectors, and Payloads + +When storing data in Qdrant, each product is a point, consisting of an id, a vector and payload: ```json { - "id": 1, + "id": 1, "vector": [0.1, 0.2, 0.3, 0.4], "payload": { "price": 899.99, @@ -28,20 +39,22 @@ When storing data in Qdrant, each product is a point, consisting of an `id`, a ` } } ``` -The `id` is a unique identifier for the point in your collection. The `vector` is a mathematical representation of similarity to other points in the collection. -Finally, the `payload` holds metadata that directly describes the point. -Though we may not be able to decipher the vector, we are able to derive additional information about the item from its metadata, In this specific case, **we are looking at a data point for a laptop that costs $899.99**. +- `id`: a unique identifier within the collection. +- `vector`: a dense embedding representing the point's semantic position in the vector space. +- `payload`: arbitrary metadata attached to the point, queryable via filter conditions. + +Though we may not be able to decipher the vector, we are able to derive additional information about the item from its metadata, In this specific case, we are looking at a data point for a laptop that costs $899.99. -## What is filtering? +## What Is Filtering? -When searching for the perfect computer, your customers may end up with results that are mathematically similar to the search entry, but not exact. For example, if they are searching for **laptops under $1000**, a simple [vector search](/advanced-search/) without constraints might still show other laptops over $1000. +When searching for the perfect computer, your customers may end up with results that are mathematically similar to the search entry, but not exact. For example, if they are searching for laptops under $1000, a simple [vector search](https://qdrant.tech/advanced-search/) without constraints might still show other laptops over $1000. -This is why [semantic search](/advanced-search/) alone **may not be enough**. In order to get the exact result, you would need to enforce a payload filter on the `price`. Only then can you be sure that the search results abide by the chosen characteristic. +This is why semantic search alone may not be enough. In order to get the exact result, you would need to enforce a payload filter on the price. Only then can you be sure that the search results abide by the chosen characteristic. > This is called **filtering** and it is one of the key features of [vector databases](https://qdrant.tech). -Here is how a **filtered vector search** looks behind the scenes. We'll cover its mechanics in the following section. +Here is how a filtered vector search looks behind the scenes. We’ll cover its mechanics in the following section. ```http POST /collections/online_store/points/search @@ -70,70 +83,78 @@ POST /collections/online_store/points/search } ``` -The filtered result will be a combination of the semantic search and the filtering conditions imposed upon the query. In the following pages, we will show that **filtering is a key practice in vector search for two reasons:** +The filtered result will be a combination of the semantic search and the filtering conditions imposed upon the query. In the following pages, we will show that filtering is a key practice in vector search for two reasons: -1. With filtering in Qdrant, you can **dramatically increase search precision**. More on this in the next section.
-2. Filtering helps control resources and **reduce compute use**. More on this in [**Payload Indexing**](#filtering-with-the-payload-index). +1. With filtering in Qdrant, you can dramatically increase search precision. More on this in the next section. +2. Filtering helps control resources and reduce compute use. More on this in [Payload Indexing](https://qdrant.tech/articles/vector-search-filtering/#filtering-with-the-payload-index). -## What you will learn in this guide: +## When to Filter vs. When Not To -In [vector search](/advanced-search/), filtering and sorting are more interdependent than they are in traditional databases. While databases like SQL use commands such as `WHERE` and `ORDER BY`, the interplay between these processes in vector search is a bit more complex. +In [vector search](/advanced-search/), filtering and sorting are more interdependent than they are in traditional databases. While databases like SQL use commands such as `WHERE` and `ORDER BY`, the interplay between these processes in vector search is a bit more complex. Not every query benefits from filtering. Applying a filter has coordination cost: the query planner must resolve the filter against the index, estimate cardinality, and select an execution strategy. For very broad filters (high cardinality, matching most of the dataset), the overhead may outweigh the gain. -Most people use default settings and build vector search apps that aren't properly configured or even setup for precise retrieval. In this guide, we will show you how to **use filtering to get the most out of vector search** with some basic and advanced strategies that are easy to implement. - -#### Remember to run all tutorial code in Qdrant's Dashboard - -The easiest way to reach that "Hello World" moment is to [**try filtering in a live cluster**](/documentation/cloud-quickstart/). Our interactive tutorial will show you how to create a cluster, add data and try some filtering clauses. +Use this decision tree to determine whether filtering is appropriate for a given query: -![qdrant-filtering-tutorial](/articles_data/vector-search-filtering/qdrant-filtering-tutorial.png) +![Decision tree for choosing whether to apply a filter based on filter selectivity](/articles_data/vector-search-filtering/filtering-decision-tree.png) -## Qdrant's approach to filtering +## How Qdrant Handles Filtered Vector Search -Qdrant follows a specific method of searching and filtering through dense vectors. +### The Filterable HNSW Index -Let's take a look at this **3-stage diagram**. In this case, we are trying to find the nearest neighbour to the query vector **(green)**. Your search journey starts at the bottom **(orange)**. +Qdrant's default vector index is HNSW (Hierarchical Navigable Small World), a graph-based approximate nearest neighbor structure. Each point is a node; edges connect points that are geometrically close. -By default, Qdrant connects all your data points within the [**vector index**](/documentation/manage-data/indexing/). After you [**introduce filters**](/documentation/search/filtering/), some data points become disconnected. Vector search can't cross the grayed out area and it won't reach the nearest neighbor. -How can we bridge this gap? +When a filter is applied, some nodes become ineligible. In a naive implementation, this disconnects the graph: the HNSW traversal cannot cross excluded nodes to reach a valid nearest neighbor. -**Figure 1:** How Qdrant maintains a filterable vector index. -![filterable-vector-index](/articles_data/vector-search-filtering/filterable-vector-index.png) +Qdrant solves this by building **additional edges** between eligible nodes that would otherwise be separated. This produces a filterable HNSW graph that remains traversable after any subset of nodes is excluded. -[**Filterable vector index**](/documentation/manage-data/indexing/): This technique builds additional links **(orange)** between leftover data points. The filtered points which stay behind are now traversible once again. Qdrant uses special category-based methods to connect these data points. +![Qdrant's filterable HNSW graph maintains additional links between eligible points so traversal can reach valid nearest neighbors after filtering](/articles_data/vector-search-filtering/filterable-hnsw.png) -### Qdrant's approach vs traditional filtering methods +### Pre-Filtering and Post-Filtering: Why Neither Works Alone -![stepping-lens](/articles_data/vector-search-filtering/stepping-lens.png) +The filterable vector index is Qdrant’s solves pre and post-filtering problems by adding specialized links to the search graph. It aims to maintain the speed advantages of vector search while allowing for precise filtering, addressing the inefficiencies that can occur when applying filters after the vector search. -The filterable vector index is Qdrant's solves pre and post-filtering problems by adding specialized links to the search graph. It aims to maintain the speed advantages of vector search while allowing for precise filtering, addressing the inefficiencies that can occur when applying filters after the vector search. +**Pre-filtering** narrows the dataset by payload predicate first, then runs vector search over the filtered subset. This is efficient when the filter is highly selective (few results survive). When the filter is broad (many results survive), the HNSW graph over the filtered subset is fragmented—too many links have been removed—and search accuracy degrades. -#### Pre-filtering +![Pre-filtering narrows candidates by payload before similarity ranking](/articles_data/vector-search-filtering/pre-filtering.png) -In pre-filtering, a search engine first narrows down the dataset based on chosen metadata values, and then searches within that filtered subset. This reduces unnecessary computation over a dataset that is potentially much larger. +**Post-filtering** runs vector search first, retrieves a large candidate set, then applies the filter. This is accurate when the filter is broad (most candidates pass). When the filter is selective, most retrieved candidates are discarded, wasting computation—and if the qualifying points were not in the initial candidate set, they will never appear in results at all. -The choice between pre-filtering and using the filterable HNSW index depends on filter cardinality. When metadata cardinality is too low, the filter becomes restrictive and it can disrupt the connections within the graph. This leads to fragmented search paths (as in **Figure 1**). When the semantic search process begins, it won’t be able to travel to those locations. +![Post-filtering applies payload constraints after similarity ranking, discarding candidates that don't meet the filter](/articles_data/vector-search-filtering/post-filtering.png) -However, Qdrant still benefits from pre-filtering **under certain conditions**. In cases of low cardinality, Qdrant's query planner stops using HNSW and switches over to the payload index alone. This makes the search process much cheaper and faster than if using HNSW. +Qdrant's filterable HNSW avoids this trade-off by building graph links that remain valid under any filter. The query planner also switches dynamically between HNSW traversal and payload-index-based retrieval depending on estimated filter cardinality—see [Payload indexing](#filtering-with-the-payload-index) below. -**Figure 2:** On the user side, this is how filtering looks. We start with five products with different prices. First, the $1000 price **filter** is applied, narrowing down the selection of laptops. Then, a vector search finds the relevant **results** within this filtered set. +## Filtering Mechanics: Conditions and Clauses -![pre-filtering-vector-search](/articles_data/vector-search-filtering/pre-filtering.png) +### Clauses -In conclusion, pre-filtering is efficient in specific cases when you use small datasets with low cardinality metadata. However, pre-filtering should not be used over large datasets as it breaks too many links in the HNSW graph, causing lower accuracy. +Clauses determine how multiple conditions are combined: -#### Post-filtering +| Clause | Behaviour | SQL analogue | +|---|---|---| +| `must` | All conditions must match | `AND` | +| `should` | At least one condition must match | `OR` | +| `must_not` | No conditions may match | `NOT` | +| Clause combination | Combines multiple clauses | `AND` between clause groups | -In post-filtering, a search engine first looks for similar vectors and retrieves a larger set of results. Then, it applies filters to those results based on metadata. The problem with post-filtering becomes apparent when using low-cardinality filters. +### Conditions -> When you apply a low-cardinality filter after performing a vector search, you often end up discarding a large portion of the results that the vector search returned. +| Condition | Description | +|---|---| +| `match` | Exact value match on a keyword or integer field | +| `match_any` | Matches any value in a provided list | +| `match_except` | Excludes specific values | +| `range` | Numeric or datetime range | +| `datetime_range` | Explicit datetime range with timezone support | +| `geo_bounding_box` / `geo_radius` / `geo_polygon` | Geospatial conditions | +| `full_text` | Tokenised text search within a field | +| `nested` | Evaluates conditions per element of an object array | +| `is_empty` | Matches points where a field is absent or an empty array | +| `is_null` | Matches points where a field is `null` | +| `has_id` | Matches specific point IDs | +| `values_count` | Matches on the number of elements in an array field | -**Figure 3:** In the same example, we have five laptops. First, the vector search finds the top two relevant **results**, but they may not meet the price match. When the $1000 price **filter** is applied, other potential results are discarded. +For the complete reference, see the [filtering documentation](/documentation/search/filtering/). -![post-filtering-vector-search](/articles_data/vector-search-filtering/post-filtering.png) - -The system will waste computational resources by first finding similar vectors and then discarding many that don't meet the filter criteria. You're also limited to filtering only from the initial set of [vector search](/advanced-search/) results. If your desired items aren't in this initial set, you won't find them, even if they exist in the database. - -## Basic filtering example: ecommerce and laptops +## Basic Filtering Example: E-commerce and Laptops We know that there are three possible laptops that suit our price point. Let's see how Qdrant's filterable vector index works and why it is the best method of capturing all available results. @@ -203,7 +224,7 @@ This specific example uses the `range` condition for filtering. Qdrant, however, **For detailed usage examples, [filtering](/documentation/search/filtering/) docs are the best resource.** -### Scrolling instead of searching +### Scrolling Instead of Searching You don't need to use our `search` and `query` APIs to filter through data. The `scroll` API is another option that lets you retrieve lists of points which meet the filters. @@ -238,11 +259,12 @@ POST /collections/online_store/points/scroll ] } ``` + The response contains a batch of points that match the criteria and a reference (offset or next page token) to retrieve the next set of points. > [**Scrolling**](/documentation/manage-data/points/#scroll-points) is designed to be efficient. It minimizes the load on the server and reduces memory consumption on the client side by returning only manageable chunks of data at a time. -#### Available filtering conditions +#### Available Filtering Conditions | **Condition** | **Usage** | **Condition** | **Usage** | |-----------------------|------------------------------------------|-----------------------|------------------------------------------| @@ -256,304 +278,15 @@ The response contains a batch of points that match the criteria and a reference > All clauses and conditions are outlined in Qdrant's [filtering](/documentation/search/filtering/) documentation. -#### Filtering clauses to remember +#### Filtering Clauses to Remember | **Clause** | **Description** | **Clause** | **Description** | |---------------------|-------------------------------------------------------|---------------------|-------------------------------------------------------| | **Must** | Includes items that meet the condition
(similar to `AND`). | **Should** | Filters if at least one condition is met
(similar to `OR`). | | **Must Not** | Excludes items that meet the condition
(similar to `NOT`). | **Clauses Combination** | Combines multiple clauses to refine filtering
(similar to `AND`). | -## Advanced filtering example: dinosaur diets - -![advanced-payload-filtering](/articles_data/vector-search-filtering/advanced-payload-filtering.png) - -We can also use nested filtering to query arrays of objects within the payload. In this example, we have two points. They each represent a dinosaur with a list of food preferences (diet) that indicate what type of food they like or dislike: - -```json -[ - { - "id": 1, - "dinosaur": "t-rex", - "diet": [ - { "food": "leaves", "likes": false}, - { "food": "meat", "likes": true} - ] - }, - { - "id": 2, - "dinosaur": "diplodocus", - "diet": [ - { "food": "leaves", "likes": true}, - { "food": "meat", "likes": false} - ] - } -] -``` -To ensure that both conditions are applied to the same array element (e.g., food = meat and likes = true must refer to the same diet item), you need to use a nested filter. - -Nested filters are used to apply conditions within an array of objects. They ensure that the conditions are evaluated per array element, rather than across all elements. -```http -POST /collections/dinosaurs/points/scroll -{ - "filter": { - "must": [ - { - "key": "diet[].food", - "match": { - "value": "meat" - } - }, - { - "key": "diet[].likes", - "match": { - "value": true - } - } - ] - } -} -``` - -```python -client.scroll( - collection_name="dinosaurs", - scroll_filter=models.Filter( - must=[ - models.FieldCondition( - key="diet[].food", match=models.MatchValue(value="meat") - ), - models.FieldCondition( - key="diet[].likes", match=models.MatchValue(value=True) - ), - ], - ), -) -``` - -```typescript -client.scroll("dinosaurs", { - filter: { - must: [ - { - key: "diet[].food", - match: { value: "meat" }, - }, - { - key: "diet[].likes", - match: { value: true }, - }, - ], - }, -}); -``` - -```rust -use qdrant_client::qdrant::{Condition, Filter, ScrollPointsBuilder}; - -client - .scroll( - ScrollPointsBuilder::new("dinosaurs").filter(Filter::must([ - Condition::matches("diet[].food", "meat".to_string()), - Condition::matches("diet[].likes", true), - ])), - ) - .await?; -``` - -```java -import java.util.List; - -import static io.qdrant.client.ConditionFactory.match; -import static io.qdrant.client.ConditionFactory.matchKeyword; - -import io.qdrant.client.QdrantClient; -import io.qdrant.client.QdrantGrpcClient; -import io.qdrant.client.grpc.Common.Filter; -import io.qdrant.client.grpc.Points.ScrollPoints; - -QdrantClient client = - new QdrantClient(QdrantGrpcClient.newBuilder("localhost", 6334, false).build()); - -client - .scrollAsync( - ScrollPoints.newBuilder() - .setCollectionName("dinosaurs") - .setFilter( - Filter.newBuilder() - .addAllMust( - List.of(matchKeyword("diet[].food", "meat"), match("diet[].likes", true))) - .build()) - .build()) - .get(); -``` - -```csharp -using Qdrant.Client; -using static Qdrant.Client.Grpc.Conditions; - -var client = new QdrantClient("localhost", 6334); - -await client.ScrollAsync( - collectionName: "dinosaurs", - filter: MatchKeyword("diet[].food", "meat") & Match("diet[].likes", true) -); -``` - -This happens because both points are matching the two conditions: - -- the "t-rex" matches food=meat on `diet[1].food` and likes=true on `diet[1].likes` -- the "diplodocus" matches food=meat on `diet[1].food` and likes=true on `diet[0].likes` - -To retrieve only the points where the conditions apply to a specific element within an array (such as the point with id 1 in this example), you need to use a nested object filter. - -Nested object filters enable querying arrays of objects independently, ensuring conditions are checked within individual array elements. - -This is done by using the `nested` condition type, which consists of a payload key that targets an array and a filter to apply. The key should reference an array of objects and can be written with or without bracket notation (e.g., "data" or "data[]"). - -```http -POST /collections/dinosaurs/points/scroll -{ - "filter": { - "must": [{ - "nested": { - "key": "diet", - "filter":{ - "must": [ - { - "key": "food", - "match": { - "value": "meat" - } - }, - { - "key": "likes", - "match": { - "value": true - } - } - ] - } - } - }] - } -} -``` - -```python -client.scroll( - collection_name="dinosaurs", - scroll_filter=models.Filter( - must=[ - models.NestedCondition( - nested=models.Nested( - key="diet", - filter=models.Filter( - must=[ - models.FieldCondition( - key="food", match=models.MatchValue(value="meat") - ), - models.FieldCondition( - key="likes", match=models.MatchValue(value=True) - ), - ] - ), - ) - ) - ], - ), -) -``` - -```typescript -client.scroll("dinosaurs", { - filter: { - must: [ - { - nested: { - key: "diet", - filter: { - must: [ - { - key: "food", - match: { value: "meat" }, - }, - { - key: "likes", - match: { value: true }, - }, - ], - }, - }, - }, - ], - }, -}); -``` - -```rust -use qdrant_client::qdrant::{Condition, Filter, NestedCondition, ScrollPointsBuilder}; - -client - .scroll( - ScrollPointsBuilder::new("dinosaurs").filter(Filter::must([NestedCondition { - key: "diet".to_string(), - filter: Some(Filter::must([ - Condition::matches("food", "meat".to_string()), - Condition::matches("likes", true), - ])), - } - .into()])), - ) - .await?; -``` - -```java -import java.util.List; - -import static io.qdrant.client.ConditionFactory.match; -import static io.qdrant.client.ConditionFactory.matchKeyword; -import static io.qdrant.client.ConditionFactory.nested; - -import io.qdrant.client.grpc.Common.Filter; -import io.qdrant.client.grpc.Points.ScrollPoints; - -client - .scrollAsync( - ScrollPoints.newBuilder() - .setCollectionName("dinosaurs") - .setFilter( - Filter.newBuilder() - .addMust( - nested( - "diet", - Filter.newBuilder() - .addAllMust( - List.of( - matchKeyword("food", "meat"), match("likes", true))) - .build())) - .build()) - .build()) - .get(); -``` - -```csharp -using Qdrant.Client; -using static Qdrant.Client.Grpc.Conditions; - -var client = new QdrantClient("localhost", 6334); - -await client.ScrollAsync( - collectionName: "dinosaurs", - filter: Nested("diet", MatchKeyword("food", "meat") & Match("likes", true)) -); -``` - -The matching logic is adjusted to operate at the level of individual elements within an array in the payload, rather than on all array elements together. - -Nested filters function as though each element of the array is evaluated separately. The parent document will be considered a match if at least one array element satisfies all the nested filter conditions. - -## Other creative uses for filters +## Other Uses of Filters You can use filters to retrieve data points without knowing their `id`. You can search through data and manage it, solely by using filters. Let's take a look at some creative uses for filters: @@ -564,18 +297,15 @@ You can use filters to retrieve data points without knowing their `id`. You can | [Order Points](/documentation/manage-data/points/#order-points-by-payload-key) | Lists all points, sorted by the filter. | [Delete Payload](/documentation/manage-data/payload/#delete-payload-keys) | Deletes fields for points matching the filter. | | [Count Points](/documentation/manage-data/points/#counting-points) | Totals the points matching the filter. | | | -## Filtering with the payload index +## Filtering with the Payload Index -![vector-search-filtering-vector-search](/articles_data/vector-search-filtering/scanning-lens.png) When you start working with Qdrant, your data is by default organized in a vector index. In addition to this, we recommend adding a secondary data structure - **the payload index**. Just how the vector index organizes vectors, the payload index will structure your metadata. -**Figure 4:** The payload index is an additional data structure that supports vector search. A payload index (in green) organizes candidate results by cardinality, so that semantic search (in red) can traverse the vector index quickly. - -![payload-index-vector-search](/articles_data/vector-search-filtering/payload-index-vector-search.png) +![The payload index organizes candidates by cardinality so semantic search can traverse the vector index quickly](/articles_data/vector-search-filtering/payload-index.png) On its own, semantic searching over terabytes of data can take up lots of RAM. [**Filtering**](/documentation/search/filtering/) and [**Indexing**](/documentation/manage-data/indexing/) are two easy strategies to reduce your compute usage and still get the best results. Remember, this is only a guide. For an exhaustive list of filtering options, you should read the [filtering documentation](/documentation/search/filtering/). @@ -588,6 +318,8 @@ PUT /collections/computers/index "field_schema": "keyword" } ``` + + ```python from qdrant_client import QdrantClient @@ -599,11 +331,11 @@ client.create_payload_index( field_schema="keyword", ) ``` + Once you mark a field indexable, **you don't need to do anything else**. Qdrant will handle all optimizations in the background. -#### Why should you index metadata? +#### Why Should You Index Metadata? -![payload-index-filtering](/articles_data/vector-search-filtering/payload-index-filtering.png) The payload index acts as a secondary data structure that speeds up retrieval. Whenever you run vector search with a filter, Qdrant will consult a payload index - if there is one. @@ -613,7 +345,7 @@ Indexing your metadata has a significant positive effect on search performance w As your dataset grows in complexity, Qdrant takes up additional resources to go through all data points. Without a proper data structure, the search can take longer - or run out of resources. -#### Payload indexing helps evaluate the most restrictive filters +#### Payload Indexing Helps Evaluate the Most Restrictive Filters The payload index is also used to accurately estimate **filter cardinality**, which helps the query planning choose a search strategy. **Filter cardinality** refers to the number of distinct values that a filter can match within a dataset. Qdrant's search strategy can switch from **HNSW search** to **payload index-based search** if the cardinality is too low. @@ -627,13 +359,13 @@ The payload index is also used to accurately estimate **filter cardinality**, wh Our default full scan threshold is 10 kilobytes. -#### What happens if you don't use payload indexes? +#### What Happens If You Don't Use Payload Indexes? When using filters while querying, Qdrant needs to estimate cardinality of those filters to define a proper query plan. If you don't create a payload index, Qdrant will not be able to do this. It may end up choosing a sub-optimal way of searching causing extremely slow search times or low accuracy results. If you only rely on **searching for the nearest vector**, Qdrant will have to go through the entire vector index. It will calculate similarities against each vector in the collection, relevant or not. Alternatively, when you filter with the help of a payload index, the HSNW algorithm won't have to evaluate every point. Furthermore, the payload index will help HNSW construct the graph with additional links. -## How does the payload index look? +## How Does the Payload Index Look? A payload index is similar to conventional document-oriented databases. It connects metadata fields with their corresponding point id’s for quick retrieval. @@ -650,13 +382,14 @@ Payload Index by keyword: | keyboard | 10, 11 | +------------+-------------+ ``` + When fields are properly indexed, the search engine roughly knows where it can start its journey. It can start looking up points that contain relevant metadata, and it doesn’t need to scan the entire dataset. This reduces the engine’s workload by a lot. As a result, query results are faster and the system can easily scale. > You may create as many payload indexes as you want, and we recommend you do so for each field that you filter by. If your users are often filtering by **laptop** when looking up a product **category**, indexing all computer metadata will speed up retrieval and make the results more precise. -#### Different types of payload indexes +#### Different Types of Payload Indexes | Index Type | Description | |---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| @@ -666,7 +399,7 @@ If your users are often filtering by **laptop** when looking up a product **cate |[On-Disk Index](/documentation/manage-data/indexing/#on-disk-payload-index) | Stores indexes on disk to manage large datasets without memory usage. | | [Parameterized Index](/documentation/manage-data/indexing/#parameterized-index) | Allows for dynamic querying, where the index can adapt based on different parameters or conditions provided by the user. Useful for numeric data like prices or timestamps. | -### Indexing payloads in multitenant setups +### Indexing Payloads in Multitenant Setups Some applications need to have data segregated, whereby different users need to see different data inside of the same program. When setting up storage for such a complex application, many users think they need multiple databases for segregated users. @@ -686,14 +419,15 @@ PUT /collections/{collection_name}/index } } ``` + Additionally, we offer a way of organizing data efficiently by means of the tenant index. This is another variant of the payload index that makes tenant data more accessible. This time, the request will specify the field as a tenant. This means that you can mark various customer types and user id’s as `is_tenant: true`. Read more about setting up [tenant defragmentation](/documentation/manage-data/indexing/?q=tenant#tenant-index) in multitenant environments, -## Key takeaways in filtering and indexing -![best-practices](/articles_data/vector-search-filtering/best-practices.png) +## Key Takeaways in Filtering and Indexing -### Filtering with float-point (decimal) numbers + +### Filtering with Floating Point (Decimal) Numbers If you filter by the float data type, your search precision may be limited and inaccurate. Float Datatype numbers have a decimal point and are 64 bits in size. Here is an example: @@ -721,7 +455,8 @@ Here is a sample JSON range filter for values greater than or equal to 11.99 and } } ``` -### Working with pagination in queries + +### Working with Pagination in Queries When you're implementing pagination in filtered queries, indexing becomes even more critical. When paginating results, you often need to exclude items you've already seen. This is typically managed by applying filters that specify which IDs should not be included in the next set of results. @@ -729,7 +464,7 @@ However, an interesting aspect of Qdrant's data model is that a single point can Proper indexing ensures that these queries are efficient, preventing duplicate results and making pagination smoother. -## Conclusion: Real-life use cases of filtering +## Conclusion: Real-Life Use Cases of Filtering Filtering in a [vector database](https://qdrant.tech) like Qdrant can significantly enhance search capabilities by enabling more precise and efficient retrieval of data. @@ -742,10 +477,8 @@ As a conclusion to this guide, let's look at some real-life use cases where filt | [Geospatial Search in Ride-Sharing](/articles/geo-polygon-filter-gsoc/)| Find similar drivers or delivery partners | Filter by rating, distance radius, vehicle type | | [Fraud & Anomaly Detection](/data-analysis-anomaly-detection/) | Detect transactions similar to known fraud cases | Filter by amount, time, location | -#### Before you go - all the code is in Qdrant's Dashboard +#### Before You Go: All the Code Is in Qdrant's Dashboard The easiest way to reach that "Hello World" moment is to [**try filtering in a live cluster**](/documentation/cloud-quickstart/). Our interactive tutorial will show you how to create a cluster, add data and try some filtering clauses. **It's all in your free cluster!** - -[![qdrant-hybrid-cloud](/docs/homepage/cloud-cta.png)](https://qdrant.to/cloud) diff --git a/qdrant-landing/static/articles_data/vector-search-filtering/budget-query-example.png b/qdrant-landing/static/articles_data/vector-search-filtering/budget-query-example.png new file mode 100644 index 0000000000..5f841a83f3 Binary files /dev/null and b/qdrant-landing/static/articles_data/vector-search-filtering/budget-query-example.png differ diff --git a/qdrant-landing/static/articles_data/vector-search-filtering/filterable-hnsw.png b/qdrant-landing/static/articles_data/vector-search-filtering/filterable-hnsw.png new file mode 100644 index 0000000000..e015a5389e Binary files /dev/null and b/qdrant-landing/static/articles_data/vector-search-filtering/filterable-hnsw.png differ diff --git a/qdrant-landing/static/articles_data/vector-search-filtering/filtering-decision-tree.png b/qdrant-landing/static/articles_data/vector-search-filtering/filtering-decision-tree.png new file mode 100644 index 0000000000..a71cc98284 Binary files /dev/null and b/qdrant-landing/static/articles_data/vector-search-filtering/filtering-decision-tree.png differ diff --git a/qdrant-landing/static/articles_data/vector-search-filtering/payload-index.png b/qdrant-landing/static/articles_data/vector-search-filtering/payload-index.png new file mode 100644 index 0000000000..b3572abbe0 Binary files /dev/null and b/qdrant-landing/static/articles_data/vector-search-filtering/payload-index.png differ diff --git a/qdrant-landing/static/articles_data/vector-search-filtering/post-filtering.png b/qdrant-landing/static/articles_data/vector-search-filtering/post-filtering.png index 193c64bcc0..525db90d21 100644 Binary files a/qdrant-landing/static/articles_data/vector-search-filtering/post-filtering.png and b/qdrant-landing/static/articles_data/vector-search-filtering/post-filtering.png differ diff --git a/qdrant-landing/static/articles_data/vector-search-filtering/pre-filtering.png b/qdrant-landing/static/articles_data/vector-search-filtering/pre-filtering.png index c060d978fd..980f5e1169 100644 Binary files a/qdrant-landing/static/articles_data/vector-search-filtering/pre-filtering.png and b/qdrant-landing/static/articles_data/vector-search-filtering/pre-filtering.png differ