diff --git a/qdrant-landing/content/course/beginners/_index.md b/qdrant-landing/content/course/beginners/_index.md new file mode 100644 index 0000000000..8f52a9d29a --- /dev/null +++ b/qdrant-landing/content/course/beginners/_index.md @@ -0,0 +1,157 @@ +--- +title: "Beginners Course" +page_title: "Qdrant Beginners Course" +short_description: "Learn the fundamentals of vector search: why keyword search fails, how semantic search works, embeddings, distance metrics, and hybrid systems." +description: "Understand the fundamentals of vector search. Learn why keyword search breaks, how semantic search with vectors solves it, and build your first search system." +content: + sidebarTitle: "Beginners Course" + menuTitle: + text: Course Overview + url: /course/beginners/ + nextButton: Continue to Next Step + nextDay: Complete + title: "Beginners Course" + description: "Understand the fundamentals of vector search. Learn why keyword search breaks, how semantic search with vectors solves it, and build your first search system." +partition: course +isLesson: true +--- + +# Beginners Course + +**Learn the fundamentals of vector search** + +Understand why traditional search struggles and how modern semantic search improves it. Learn about embeddings, distance metrics, and hybrid search systems. + +
+ +{{< cards-list >}} +- icon: /icons/outline/play-white.svg + title: Multiple modules + content: Focused lessons building from fundamentals to practical applications +- icon: /icons/outline/cloud-check-blue.svg + title: Shareable certificate + content: Earn a digital certificate upon completion +- icon: /icons/outline/time-blue.svg + title: Flexible schedule + content: Learn at your own pace +- icon: /icons/outline/plan.svg + title: Beginner level + content: No prior experience required + +{{< /cards-list >}} + +
+ +## What you'll learn +{{< course-card + title="Skills you'll gain:" + image="/icons/outline/training-white.svg" + type="wide-list">}} + +- Why keyword search breaks and how semantic search solves it +- How embeddings convert text to vectors that capture meaning +- Distance metrics: cosine similarity, dot product, and Euclidean +- Hybrid search: combining dense and sparse retrieval +- Building your first Qdrant collection and queries + +{{< /course-card >}} + +### The Path + +**Module 0**: Setup. Configure your environment and get started with the basics. + +**Module 1**: Let's Understand Search. Understand why traditional search struggles and how modern semantic search improves it. + +**Module 2**: First Principles of Vector Search. Learn what vectors are, how dimensions represent meaning, similarity metrics, and build your first Qdrant collection. + +## How the course works + +{{< cards-list >}} + +- icon: /icons/outline/training-purple.svg + title: Clear lessons + content: Focused modules by the Qdrant team +- icon: /icons/outline/hacker-purple.svg + title: Hands-on learning + content: Practical examples and exercises +- icon: /icons/outline/similarity-blue.svg + title: Progressive learning + content: Build from fundamentals to advanced concepts +- icon: /icons/outline/copy.svg + title: Self-paced + content: Learn at your own speed + {{< /cards-list >}} + +
+ +## Syllabus + +{{< accordion >}} +- title: "Module 0: Setting Up Dependencies" + content: | + - Qdrant Cloud Setup + - Implementing a Basic Vector Search + - Project: Building Your First Vector Search System +
+
+

→ Start Module 0

+ +- title: "Module 1: Let's Understand Search" + content: | + - The Problem: Why Keyword Search Breaks + - How Traditional Search Improved + - Enter Semantic Search + - How It Works: Embeddings + - Comparing Meaning: Distance Metrics + - Why Similarity Alone Is Not Enough + - Modern Search = Hybrid Systems + - References & Further Reading +
+
+

→ Coming soon

+ +- title: "Module 2: First Principles of Vector Search" + content: | + - What is a Vector? + - How Dimensions Represent Meaning + - Similarity Under the Hood + - Your First Qdrant Collection + - Points, Payloads, and Queries +
+
+

→ Coming soon

+ +- title: "Module 3: Sparse vs Dense vs Hybrid Search" + content: | + - The Two Families of Search + - Hybrid Search: Dense + Sparse + Filters + - Setting Up Hybrid Search in Qdrant + - Fusion Strategies + - Beyond Text: Multimodal Search + - Real-World Use Cases +
+
+

→ Coming soon

+{{< /accordion >}} + +## Who it's for + +Anyone new to vector search who wants to understand the fundamentals. No prior experience with Qdrant or vector databases required. + +## Time commitment + +- Duration: Multiple modules +- Self-paced learning +- Flexible schedule + + +{{< course-card + title="Ready to start your vector search journey?" + image="/icons/outline/rocket-white-light.svg" + link="/course/beginners/module-0/">}} +**What you'll get** +- Understand the fundamentals of vector search +- Learn why semantic search outperforms keyword search +- Build your first Qdrant collection +- Foundation for advanced courses +{{< /course-card >}} diff --git a/qdrant-landing/content/course/beginners/module-0/_index.md b/qdrant-landing/content/course/beginners/module-0/_index.md new file mode 100644 index 0000000000..1fd62b8d5a --- /dev/null +++ b/qdrant-landing/content/course/beginners/module-0/_index.md @@ -0,0 +1,21 @@ +--- +title: "Module 0: Setting Up Dependencies" +short_description: "Module 0 of the Beginners course: set up Qdrant Cloud, build a first vector search, and get started with the basics." +description: "Set up Qdrant and build your first vector search app. Learn how to configure Qdrant Cloud, run a basic search, and get started with the fundamentals." +isLesson: true +weight: 10 +--- + +{{< date >}} Module 0 {{< /date >}} + +# Setting Up Dependencies + +Get started with Qdrant by setting up your environment and building your first vector search application. + +## Today's path + +1. Qdrant Cloud Setup +2. Implementing a Basic Vector Search +3. Project: Building Your First Vector Search System + +By the end, you'll have a working Qdrant setup and a complete first search running. diff --git a/qdrant-landing/content/course/beginners/module-0/qdrant-cloud.md b/qdrant-landing/content/course/beginners/module-0/qdrant-cloud.md new file mode 100644 index 0000000000..e005cf3815 --- /dev/null +++ b/qdrant-landing/content/course/beginners/module-0/qdrant-cloud.md @@ -0,0 +1,176 @@ +--- +title: "Qdrant Setup" +short_description: "Spin up a managed Qdrant Cloud cluster, generate API keys, and explore the Web UI for collections, points, and cluster monitoring." +description: Set up your Qdrant Cloud cluster in minutes. Learn to create collections, manage data, access the Web UI, and connect securely from Python. +weight: 2 +isLesson: true +--- + +{{< date >}} Module 0 {{< /date >}} + +# Qdrant Setup + +
+ +
+ +
+ +Spin up production-grade vector search in minutes. Qdrant Cloud gives you a managed endpoint with TLS, automatic backups, high-availability options, and a clean API. + +## Create your cluster + +1. Sign up at [cloud.qdrant.io](https://cloud.qdrant.io/signup) with email, Google, or GitHub. +2. Open **Clusters** → **Create a Free Cluster**. The Free Tier is enough for this course. + +![Create cluster](/docs/gettingstarted/gui-quickstart/create-cluster.png) + +3. Pick a region close to your users or app. +4. When the cluster is ready, copy the API key and store it securely. You can make new keys later from **API Keys** on the cluster page. + +![Get API key](/docs/gettingstarted/gui-quickstart/api-key.png) + + +## Access the Web UI + +1. Click **Cluster UI** in the top-right of the cluster page to open the dashboard. + +![Access dashboard](/docs/gettingstarted/gui-quickstart/access-dashboard.png) + +### What you can do in the Web UI + +Use the Web UI to manage collections, inspect data, and debug search performance. + +#### Main Navigation + +**Console**: Run REST calls in the browser. Test endpoints, inspect responses, and debug queries without writing code. Handy for exploring the full API. + +**Collections**: See and manage all collections. Create collections, upload snapshots, and track status, size, and configuration at a glance. + +**Tutorial**: Follow an interactive walkthrough with sample data. Create a collection, add vectors, and run semantic search with live results. + +![Interactive tutorial](/docs/gettingstarted/gui-quickstart/interactive-tutorial.png) + +**Datasets**: Bulk-load preconfigured public datasets into your cluster. + +#### Inside a Collection + +When you open a collection by clicking it's name, + +![Select collection](/courses/day0/select-collection.png) + +you'll get a detailed view with these tabs: + +![Collection points](/courses/day0/collection-points.png) + +* **Points Tab**: Inspect, search, and manage individual points. Use the search bar to find by ID or filter by payload fields (e.g., `colony: "Mars"`). For each point, you can: + + * See its payload and vector(s). + * Click **Find Similar** to run an ad-hoc similarity search. + * Click **Open Graph** to jump to a graph view of its HNSW connections. + +* **Info Tab**: Get a full overview of collection health, config, and stats. Key fields: + + * `status`: `green` means healthy. + * `points_count`: Number of active points. + * `indexed_vectors_count`: Points currently in the HNSW index. If this lags behind `points_count`, background indexing is still running. + * `config`: JSON view of all parameters, from vector settings to optimizer options. + +* **Cluster Tab**: See how shards are placed across nodes. Use it to monitor health, find hot spots, and verify shard placement in distributed setups. + +* **Search Quality Tab**: Evaluate and benchmark retrieval precision against ground truth. Tune parameters and measure the impact on accuracy. + +* **Snapshots Tab**: Manage backups for this collection. Create a [snapshot](/documentation/snapshots/), restore it later, or migrate it to another cluster. + +* **Visualize Tab**: Explore your vector space with an interactive 2D projection. See clusters, spot outliers, and build intuition about your embeddings. + +* **Graph Tab**: Explore the HNSW graph interactively. Start from any point, follow nearest neighbors, and see how the graph structure powers fast search. + +## Connect from Python + +Store credentials in an `.env` file at the root of your working directory or in colab: + +```env +QDRANT_URL=https://YOUR-CLUSTER.cloud.qdrant.io:6333 +QDRANT_API_KEY=YOUR_API_KEY +``` + +Load the credentials from the environment and create a Qdrant client: + +```python +from qdrant_client import QdrantClient, models +import os + +client = QdrantClient(url=os.getenv("QDRANT_URL"), api_key=os.getenv("QDRANT_API_KEY")) + +# For Colab: +# from google.colab import userdata +# client = QdrantClient(url=userdata.get("QDRANT_URL"), api_key=userdata.get("QDRANT_API_KEY")) + +# Quick health check +collections = client.get_collections() +print(f"Connected to Qdrant Cloud: {len(collections.collections)} collections") +``` + +## Other ways to connect + +You can also send your key in the `Authorization` header: + +```bash +# Using api-key header +curl -X GET https://xyz-example.eu-central.aws.cloud.qdrant.io:6333/collections \ + --header 'api-key: ' + +# Using Authorization header +curl -X GET https://xyz-example.eu-central.aws.cloud.qdrant.io:6333/collections \ + --header 'Authorization: Bearer ' +``` + +## Quick validation + +Check basic connectivity: + +```bash +# Service health +curl -s "$QDRANT_URL/healthz" -H "api-key: $QDRANT_API_KEY" + +# List collections +curl -s "$QDRANT_URL/collections" -H "api-key: $QDRANT_API_KEY" +``` + +## Good practices + +* Keep secrets out of code; use environment variables or a secret manager. +* Restrict access with IP allow-lists or private networking. +* Rotate API keys regularly from the cluster **Access** tab. +* Use HTTPS only; turn on RBAC and strict limits when exposing endpoints to untrusted clients. + +## Common issues + +* **Authentication error**: Recheck the API key and the `api-key` header. +* **Connection error**: Confirm cluster status and region URL; some corporate proxies block outbound TLS. + +## Qdrant Cloud Inference + +Qdrant Cloud also offers **[Cloud Inference](/cloud-inference/)**—managed embedding generation for text and images. Skip running your own embedding models; create vectors in Qdrant Cloud and write them straight into your collections. + +
+ +
+ +Cut steps from your pipeline: send raw text or images to Qdrant, get vectors and search results in one API call. This helps prototypes and production systems alike by ending the separate embedding-infrastructure layer. + +Learn more: [Qdrant Cloud Inference](/documentation/cloud/inference/) + diff --git a/qdrant-landing/content/course/beginners/module-1/_index.md b/qdrant-landing/content/course/beginners/module-1/_index.md new file mode 100644 index 0000000000..164abf90e4 --- /dev/null +++ b/qdrant-landing/content/course/beginners/module-1/_index.md @@ -0,0 +1,236 @@ +--- +title: "Module 1: Let's Understand Search" +short_description: "Module 1 of the Beginners course: Understand why traditional search struggles and how modern semantic search improves it." +description: "Understand why traditional search struggles and how modern semantic search improves it. Learn about embeddings, distance metrics, and hybrid search systems." +isLesson: true +weight: 20 +--- + +{{< date >}} Module 1 {{< /date >}} + +# Let's Understand Search + +Understand why traditional search struggles and how modern semantic search improves it. + +## Today's path + +1. The Problem: Why Keyword Search Struggles +2. How Traditional Search Improved +3. Enter Semantic Search +4. How It Works: Embeddings +5. Comparing Meaning: Distance Metrics +6. Why Similarity Alone Is Not Enough +7. Modern Search = Hybrid Systems +8. References & Further Reading + +By the end, you'll understand the limitations of keyword search and how semantic search with vectors addresses these problems. + +## 1. The Problem: Why Keyword Search Struggles + +Traditional search works by matching exact words. That's it. If the query string appears in the document, it's a hit. If it doesn't, it's a miss - no matter how closely related the meaning is. + +```python +# Simple keyword search +if "car repair" in document: + return document +``` + +![Keyword search only matches documents containing the exact words "car" and "repair"](/courses/beginners/module-1/car-repair.png) + +This approach works for predictable, structured queries. It breaks immediately on the language real users actually write. + +### Real-World Failure Examples + +| Query | Document in the index | Result | +|-------|----------------------|--------| +| car repair | automobile maintenance guide | ❌ Missed | +| cheap flights NYC | affordable airfare to New York | ❌ Missed | +| Apple stock | fruit company disambiguation? | ❌ Missed | + +![Cheap Flights Example](/courses/beginners/module-1/cheap-flights.png) + +### The Four Core Failure Modes + +- **Synonyms**: "car" ≠ "automobile" to a keyword engine, even though they mean the same thing. No word overlap = no match. +- **Paraphrasing**: Same meaning, completely different words. "cheap flights" vs. "affordable airfare" are identical in intent, invisible to grep. +- **Polysemy**: One word, multiple meanings. "apple" is a fruit company, a fruit, a music label. Context determines meaning that keywords can't. +- **Word order**: "dog bites man" and "man bites dog" use identical words. Keyword search treats them as equivalent. + +## 2. How Traditional Search Improved + +Over time, search systems became more diverse. However, they all shared the same fundamental ceiling: they work on words, not meaning. + +### Evolution of Search Techniques + +1. **Grep / Exact Match** +Find the exact string + +2. **Inverted Index** +Fast word lookup + +3. **TF-IDF / BM25 / SPLADE** +Weighted ranking + +4. **Semantic Search** +Meaning-aware + +### What Each Improvement Added + +| Technique | What it added | Still missing | +|-----------|---------------|---------------| +| Inverted index | Fast lookup across millions of documents without scanning each one | No ranking, no relevance - just presence or absence | +| TF-IDF / BM25 | Relevance ranking based on term frequency and inverse document frequency | No synonyms, no semantic understanding | +| Keyword matching | Tolerance for typos and near-spellings (receive → receive) | Still word-based - 'automobile' is not a typo of 'car' | +| Stemming | Reduces words to their root form (running → run) | Misses cross-vocabulary synonyms entirely | + +### Core limitation + +All of these techniques still rely on matching words, not understanding meaning. They can't know that "car" and "automobile" are synonyms unless you hard-code that fact. And you can't hard-code the entire language. + +## 3. Enter Semantic Search + +Semantic search changes the question from: + +**Keyword search asks:** +"Does this document contain the same words?" + +**Semantic search asks:** +"Does this document mean the same concept?" + +## 4. How It Works: Embeddings + +Semantic search works by converting text into vectors - lists of numbers that capture meaning. Similar meanings produce vectors that are close together in high-dimensional space. Different meanings produce vectors that are far apart. + +### Generating a Vector + +An embedding model takes a piece of text and returns a fixed-length array of floating-point numbers. The exact numbers are less important than the relationships between them. + +```python +from sentence_transformers import SentenceTransformer + +model = SentenceTransformer("all-MiniLM-L6-v2") + +query_vec = model.encode("car repair") +doc_vec = model.encode("automobile maintenance") + +print(len(query_vec)) # 384 dimensions +print(query_vec[:5]) # [-0.021, 0.104, -0.048, 0.231, -0.008] +``` + +![Generating a vector from text](/courses/beginners/module-1/generating-vector.png) + +### What Are Dimensions? + +Each dimension in the vector captures some aspect of the text's meaning. A 384-dimension model has 384 such aspects. No single dimension maps cleanly to a human concept like "color" or "emotional tone" - it's the combination of all dimensions together that encodes meaning. + +### Model Types + +- **Small models (128–384 dims)**: Fast, low memory. Good for well-scoped domains like product search or FAQ retrieval. +- **Large models (768–1536 dims)**: More nuanced. Better for open-domain question answering and long-document retrieval. +- **Domain-specific models**: Fine-tuned on legal, medical, or code corpora. Outperform general models on specialized content. +- **Multimodal models**: Project text and images into the same vector space. Used by Tripadvisor and others (see Module 4). + +Vector embeddings aren't limited by these models, however. They are theoretically capable of capturing any data into a transformed structured format. + +## 5. Comparing Meaning: Distance Metrics + +Once we have vectors, we need a way to measure how similar two of them are. Different metrics suit different situations. + +### Cosine Similarity + +The most common metric for text. It measures the angle between two vectors, ignoring their magnitude (length) and focusing purely on direction. A score of 1.0 means that the vectors are pointing in the same direction and have exactly the same semantic meaning (identical meaning). A score of 0.0 means on the other hand can be interpreted as two sentences being semantically unrelated. + +$$ +\text{cosine\_similarity}(A, B) = \frac{A \cdot B}{\lVert A \rVert \, \lVert B \rVert} +$$ + +![Cosine Similarity](/courses/beginners/module-1/cosine-similarity.png) + +For example, embedding "car repair" and "automobile maintenance" and comparing the two vectors with this formula yields a similarity score around 0.847 - close to 1.0, reflecting their shared meaning despite having no words in common. + +### Distance Metric Comparison + +| Metric | Best for | Notes | +|--------|----------|-------| +| Cosine | Text similarity, NLP models | Robust to different vector magnitudes. Most common default. | +| Dot product | When embeddings are normalized | Faster than cosine if vectors are unit-normalized at index time. | +| Euclidean (L2) | Image embeddings, spatial data | Sensitive to magnitude - works best with models trained for it. | +| Manhattan (L1) | Sparse or grid-like data | Sums absolute differences instead of squaring them - less sensitive to outliers than Euclidean. | + +## 6. Why Similarity Alone Is Not Enough + +Vector similarity is a powerful primitive. But a real search system needs several more things working alongside it: + +- **Filtering**: Return only documents within the last 30 days. Return only items the current user has permission to see. +- **Exact matching**: A query for "SKU-48291" must match that exact SKU. Semantic similarity might drift to adjacent IDs. +- **Access control**: Multi-tenant systems must scope results to the current workspace. Similarity search crosses tenant boundaries. +- **Ranking signals**: Recency, popularity, personalization - payload values that should influence result order beyond pure similarity. + +### The SKU Problem - A Concrete Example + +```python +# Query: "SKU-48291 issue" +# Semantic model may return: +# SKU-48292 (score: 0.91) ← wrong product +# SKU-48291 (score: 0.89) ← correct product +# SKU-48290 (score: 0.87) ← wrong product + +# With an exact filter applied: +# must: { key: "sku", match: { value: "SKU-48291" } } +# SKU-48291 (score: 0.89) ← only correct result returned +``` + +### Key Insight + +Dense similarity finds the neighborhood. Filters, exact matches, and payload constraints find the right point within that neighborhood. You need both. + +## 7. Modern Search = Hybrid Systems + +Production search today combines multiple retrieval signals in a single pipeline. Each signal handles a different class of query. Together, they cover the full spectrum of how real users search. + +### Hybrid Search Components + +- **Dense**: Semantic / vector - Intent, vibe, meaning +- **Sparse**: BM25 / keyword - Exact terms, rare tokens + +### Where Hybrid Search Is Used + +- **Multimodal RAG** - retrieve relevant text, images, audio, and video for LLM context windows +- **Agentic AI systems** - multi-step agents that query different data sources sequentially +- **E-commerce** - find semantically similar products, then filter by price, brand, and availability +- **Knowledge bases** - semantic over documents, keyword for exact references and code snippets +- and more... + +### Quick Comparison + +| Approach | Strength | Limitation | +|----------|----------|------------| +| Keyword / Grep | Fast, exact matching | No semantic understanding - misses synonyms and paraphrases | +| BM25 / TF-IDF | Great for rare or specific terms | No synonym handling - relies entirely on word overlap | +| Semantic / Dense | Understands meaning and intent | Can miss exact tokens - 'SKU-48291' may drift to similar IDs | +| Hybrid | Best of both worlds | More complex to build, tune, and operate | + +## 8. References & Further Reading + +- **Qdrant Concepts** - [Qdrant Overview](https://qdrant.tech/documentation/concepts/) + - Overview of Qdrant's vector search engine - collections, points, payloads, and APIs. + +- **Distance Metrics Deep Dive** - [Distance Metrics - Qdrant](https://qdrant.tech/documentation/concepts/#distance-metrics) + - Cosine, dot product, Euclidean, and Manhattan - when to use each. + +- **Filtering & Hybrid Search** - [Filtering - Qdrant](https://qdrant.tech/documentation/concepts/filtering/) + - Payload filter syntax, indexed fields, and combining filters with vector queries. + +- **RAG Tutorials** - [RAG Tutorials - Qdrant](https://qdrant.tech/rag) + - End-to-end retrieval-augmented generation tutorials using Qdrant as the retriever. + +## What's Next - Module 2 + +In the next module, we'll break down: + +- What is a vector and why does it have hundreds of dimensions? +- How do dimensions actually represent meaning? +- How similarity really works under the hood - and when it fails. +- Your first Qdrant collection: points, payloads, and your first query. + +End of Module 1. Continue to Module 2: First Principles of Vector Search. diff --git a/qdrant-landing/static/courses/beginners/module-1/car-repair.png b/qdrant-landing/static/courses/beginners/module-1/car-repair.png new file mode 100644 index 0000000000..2a4c9ee529 Binary files /dev/null and b/qdrant-landing/static/courses/beginners/module-1/car-repair.png differ diff --git a/qdrant-landing/static/courses/beginners/module-1/cheap-flights.png b/qdrant-landing/static/courses/beginners/module-1/cheap-flights.png new file mode 100644 index 0000000000..65e3ca05dd Binary files /dev/null and b/qdrant-landing/static/courses/beginners/module-1/cheap-flights.png differ diff --git a/qdrant-landing/static/courses/beginners/module-1/cosine-similarity.png b/qdrant-landing/static/courses/beginners/module-1/cosine-similarity.png new file mode 100644 index 0000000000..bdac0ec404 Binary files /dev/null and b/qdrant-landing/static/courses/beginners/module-1/cosine-similarity.png differ diff --git a/qdrant-landing/static/courses/beginners/module-1/generating-vector.png b/qdrant-landing/static/courses/beginners/module-1/generating-vector.png new file mode 100644 index 0000000000..b9e373c1cf Binary files /dev/null and b/qdrant-landing/static/courses/beginners/module-1/generating-vector.png differ