diff --git a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-extensions.md b/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-extensions.md index 22b4fe41bab..0a41feb3ccd 100644 --- a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-extensions.md +++ b/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-extensions.md @@ -124,4 +124,4 @@ ext/ ## See also - [Dapr Agents core concepts]({{< ref dapr-agents-core-concepts.md >}}) -- [Integrations]({{< ref dapr-agents-integrations.md >}}) +- [Integrations]({{< ref integrations >}}) diff --git a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-integrations.md b/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-integrations.md deleted file mode 100644 index 95860742687..00000000000 --- a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-integrations.md +++ /dev/null @@ -1,425 +0,0 @@ ---- -type: docs -title: "Integrations" -linkTitle: "Integrations" -weight: 60 -description: "Various integrations and tools available in Dapr Agents" -aliases: - - /developing-applications/dapr-agents/dapr-agents-integrations ---- - -# Out-of-the-box Tools - -## Text Splitter - -The Text Splitter module is a foundational integration in `Dapr Agents` designed to preprocess documents for use in [Retrieval-Augmented Generation (RAG)](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) workflows and other `in-context learning` applications. Its primary purpose is to break large documents into smaller, meaningful chunks that can be embedded, indexed, and efficiently retrieved based on user queries. - -By focusing on manageable chunk sizes and preserving contextual integrity through overlaps, the Text Splitter ensures documents are processed in a way that supports downstream tasks like question answering, summarization, and document retrieval. - -### Why Use a Text Splitter? - -When building RAG pipelines, splitting text into smaller chunks serves these key purposes: - -* **Enabling Effective Indexing**: Chunks are embedded and stored in a vector database, making them retrievable based on similarity to user queries. -* **Maintaining Semantic Coherence**: Overlapping chunks help retain context across splits, ensuring the system can connect related pieces of information. -* **Handling Model Limitations**: Many models have input size limits. Splitting ensures text fits within these constraints while remaining meaningful. - -This step is crucial for preparing knowledge to be embedded into a searchable format, forming the backbone of retrieval-based workflows. - -### Strategies for Text Splitting - -The Text Splitter supports multiple strategies to handle different types of documents effectively. These strategies balance the size of each chunk with the need to maintain context. - -#### 1. Character-Based Length - -* **How It Works**: Counts the number of characters in each chunk. -* **Use Case**: Simple and effective for text splitting without dependency on external tokenization tools. - -Example: - -```python -from dapr_agents.document.splitter.text import TextSplitter - -# Character-based splitter (default) -splitter = TextSplitter(chunk_size=1024, chunk_overlap=200) -``` - -#### 2. Token-Based Length - -* **How It Works**: Counts tokens, which are the semantic units used by language models (e.g., words or subwords). -* **Use Case**: Ensures compatibility with models like GPT, where token limits are critical. - -**Example**: - -```python -import tiktoken -from dapr_agents.document.splitter.text import TextSplitter - -enc = tiktoken.get_encoding("cl100k_base") - -def length_function(text: str) -> int: - return len(enc.encode(text)) - -splitter = TextSplitter( - chunk_size=1024, - chunk_overlap=200, - chunk_size_function=length_function -) -``` - -The flexibility to define the chunk size function makes the Text Splitter adaptable to various scenarios. - -### Chunk Overlap - -To preserve context, the Text Splitter includes a chunk overlap feature. This ensures that parts of one chunk carry over into the next, helping maintain continuity when chunks are processed sequentially. - -Example: - -* With `chunk_size=1024` and `chunk_overlap=200`, the last `200` tokens or characters of one chunk appear at the start of the next. -* This design helps in tasks like text generation, where maintaining context across chunks is essential. - -### How to Use the Text Splitter - -Here's a practical example of using the Text Splitter to process a PDF document: - -#### Step 1: Load a PDF - -```python -import requests -from pathlib import Path - -# Download PDF -pdf_url = "https://arxiv.org/pdf/2412.05265.pdf" -local_pdf_path = Path("arxiv_paper.pdf") - -if not local_pdf_path.exists(): - response = requests.get(pdf_url) - response.raise_for_status() - with open(local_pdf_path, "wb") as pdf_file: - pdf_file.write(response.content) -``` - -#### Step 2: Read the Document - -For this example, we use Dapr Agents' `PyPDFReader`. - -{{% alert title="Note" color="primary" %}} -The PyPDF Reader relies on the [pypdf python library](https://pypi.org/project/pypdf/), which is not included in the Dapr Agents core module. This design choice helps maintain modularity and avoids adding unnecessary dependencies for users who may not require this functionality. To use the PyPDF Reader, ensure that you install the library separately. -{{% /alert %}} - -```python -pip install pypdf -``` - -Then, initialize the reader to load the PDF file. - -```python -from dapr_agents.document.reader.pdf.pypdf import PyPDFReader - -reader = PyPDFReader() -documents = reader.load(local_pdf_path) -``` - -#### Step 3: Split the Document - -```python -splitter = TextSplitter( - chunk_size=1024, - chunk_overlap=200, - chunk_size_function=length_function -) -chunked_documents = splitter.split_documents(documents) -``` - -#### Step 4: Analyze Results - -```python -print(f"Original document pages: {len(documents)}") -print(f"Total chunks: {len(chunked_documents)}") -print(f"First chunk: {chunked_documents[0]}") -``` - -### Key Features - -* **Hierarchical Splitting**: Splits text by separators (e.g., paragraphs), then refines chunks further if needed. -* **Customizable Chunk Size**: Supports character-based and token-based length functions. -* **Overlap for Context**: Retains portions of one chunk in the next to maintain continuity. -* **Metadata Preservation**: Each chunk retains metadata like page numbers and start/end indices for easier mapping. - -By understanding and leveraging the `Text Splitter`, you can preprocess large documents effectively, ensuring they are ready for embedding, indexing, and retrieval in advanced workflows like RAG pipelines. - -## Arxiv Fetcher - -The Arxiv Fetcher module in `Dapr Agents` provides a powerful interface to interact with the [arXiv API](https://info.arxiv.org/help/api/index.html). It is designed to help users programmatically search for, retrieve, and download scientific papers from arXiv. With advanced querying capabilities, metadata extraction, and support for downloading PDF files, the Arxiv Fetcher is ideal for researchers, developers, and teams working with academic literature. - -### Why Use the Arxiv Fetcher? - -The Arxiv Fetcher simplifies the process of accessing research papers, offering features like: - -* **Automated Literature Search**: Query arXiv for specific topics, keywords, or authors. -* **Metadata Retrieval**: Extract structured metadata, such as titles, abstracts, authors, categories, and submission dates. -* **Precise Filtering**: Limit search results by date ranges (e.g., retrieve the latest research in a field). -* **PDF Downloading**: Fetch full-text PDFs of papers for offline use. - -### How to Use the Arxiv Fetcher - -#### Step 1: Install Required Modules - -{{% alert title="Note" color="primary" %}} -The Arxiv Fetcher relies on a [lightweight Python wrapper](https://github.com/lukasschwab/arxiv.py) for the arXiv API, which is not included in the Dapr Agents core module. This design choice helps maintain modularity and avoids adding unnecessary dependencies for users who may not require this functionality. To use the Arxiv Fetcher, ensure you install the [library](https://pypi.org/project/arxiv/) separately. -{{% /alert %}} - -```python -pip install arxiv -``` - -#### Step 2: Initialize the Fetcher - -Set up the `ArxivFetcher` to begin interacting with the arXiv API. - -```python -from dapr_agents.document import ArxivFetcher - -# Initialize the fetcher -fetcher = ArxivFetcher() -``` - -#### Step 3: Perform Searches - -**Basic Search by Query String** - -Search for papers using simple keywords. The results are returned as Document objects, each containing: - -* `text`: The abstract of the paper. -* `metadata`: Structured metadata such as title, authors, categories, and submission dates. - -```python -# Search for papers related to "machine learning" -results = fetcher.search(query="machine learning", max_results=5) - -# Display metadata and summaries -for doc in results: - print(f"Title: {doc.metadata['title']}") - print(f"Authors: {', '.join(doc.metadata['authors'])}") - print(f"Summary: {doc.text}\n") -``` - -**Advanced Querying** - -Refine searches using logical operators like AND, OR, and NOT or perform field-specific searches, such as by author. - -Examples: - -Search for papers on "agents" and "cybersecurity": - -```python -results = fetcher.search(query="all:(agents AND cybersecurity)", max_results=10) -``` - -Exclude specific terms (e.g., "quantum" but not "computing"): - -```python -results = fetcher.search(query="all:(quantum NOT computing)", max_results=10) -``` - -Search for papers by a specific author: - -```python -results = fetcher.search(query='au:"John Doe"', max_results=10) -``` - -**Filter Papers by Date** - -Limit search results to a specific time range, such as papers submitted in the last 24 hours. - -```python -from datetime import datetime, timedelta - -# Calculate the date range -last_24_hours = (datetime.now() - timedelta(days=1)).strftime("%Y%m%d") -today = datetime.now().strftime("%Y%m%d") - -# Search for recent papers -recent_results = fetcher.search( - query="all:(agents AND cybersecurity)", - from_date=last_24_hours, - to_date=today, - max_results=5 -) - -# Display metadata -for doc in recent_results: - print(f"Title: {doc.metadata['title']}") - print(f"Authors: {', '.join(doc.metadata['authors'])}") - print(f"Published: {doc.metadata['published']}") - print(f"Summary: {doc.text}\n") -``` - -#### Step 4: Download PDFs - -Fetch the full-text PDFs of papers for offline use. Metadata is preserved alongside the downloaded files. - -```python -import os -from pathlib import Path - -# Create a directory for downloads -os.makedirs("arxiv_papers", exist_ok=True) - -# Download PDFs -download_results = fetcher.search( - query="all:(agents AND cybersecurity)", - max_results=5, - download=True, - dirpath=Path("arxiv_papers") -) - -for paper in download_results: - print(f"Downloaded Paper: {paper['title']}") - print(f"File Path: {paper['file_path']}\n") -``` - -#### Step 5: Extract and Process PDF Content - -Use `PyPDFReader` from `Dapr Agents` to extract content from downloaded PDFs. Each page is treated as a separate Document object with metadata. - -```python -from pathlib import Path -from dapr_agents.document import PyPDFReader - -reader = PyPDFReader() -docs_read = [] - -for paper in download_results: - local_pdf_path = Path(paper["file_path"]) - documents = reader.load(local_pdf_path, additional_metadata=paper) - docs_read.extend(documents) - -# Verify results -print(f"Extracted {len(docs_read)} documents.") -print(f"First document text: {docs_read[0].text}") -print(f"Metadata: {docs_read[0].metadata}") -``` - -### Practical Applications - -The Arxiv Fetcher enables various use cases for researchers and developers: - -* **Literature Reviews**: Quickly retrieve and organize relevant papers on a given topic or by a specific author. -* **Trend Analysis**: Identify the latest research in a domain by filtering for recent submissions. -* **Offline Research Workflows**: Download and process PDFs for local analysis and archiving. - -### Next Steps - -While the Arxiv Fetcher provides robust functionality for retrieving and processing research papers, its output can be integrated into advanced workflows: - -* **Building a Searchable Knowledge Base**: Combine fetched papers with integrations like text splitting and vector embeddings for advanced search capabilities. -* **Retrieval-Augmented Generation (RAG)**: Use processed papers as inputs for RAG pipelines to power question-answering systems. -* **Automated Literature Surveys**: Generate summaries or insights based on the fetched and processed research. - -## Vector Stores - -Dapr Agents includes built-in vector store implementations for use with `ConversationVectorMemory` and RAG pipelines. Each store is available from `dapr_agents.storage.vectorstores`. - -### ChromaVectorStore - -Uses [ChromaDB](https://www.trychroma.com/) for in-memory or persistent vector search. No additional infrastructure is required for development. - -```python -from dapr_agents.storage.vectorstores import ChromaVectorStore -from dapr_agents.document.embedder.openai import OpenAIEmbedder - -store = ChromaVectorStore( - collection_name="my_collection", - embedding_function=OpenAIEmbedder(), -) -``` - -### PostgresVectorStore - -Uses [PostgreSQL with pgvector](https://github.com/pgvector/pgvector) for production-grade vector similarity search. - -```python -from dapr_agents.storage.vectorstores import PostgresVectorStore -from dapr_agents.document.embedder.openai import OpenAIEmbedder - -store = PostgresVectorStore( - connection_string="postgresql://user:pass@localhost:5432/mydb", - embedding_function=OpenAIEmbedder(), - embedding_dimensions=1536, -) -``` - -### RedisVectorStore - -Uses [Redis Stack](https://redis.io/docs/latest/develop/ai/search-and-query/vectors/) via the `redisvl` library for vector similarity search. - -{{% alert title="Note" color="warning" %}} -The Redis instance started by `dapr init` is a **vanilla Redis server** and does **not** include the Search/vector modules required by Redis Stack. To use `RedisVectorStore`, you must run [Redis Stack](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/) (or a Redis deployment with the `RediSearch` module enabled) separately. -{{% /alert %}} - -Requires `redisvl` (`pip install redisvl`). - -```python -from dapr_agents.storage.vectorstores import RedisVectorStore -from dapr_agents.document.embedder.openai import OpenAIEmbedder - -store = RedisVectorStore( - url="redis://localhost:6379", - index_name="my_agent", - embedding_function=OpenAIEmbedder(), - embedding_dimensions=1536, - distance_metric="cosine", # "cosine", "l2", or "ip" - storage_type="hash", # "hash" or "json" -) -``` - -All three vector stores share the same interface and are interchangeable as the `vector_store` argument to `ConversationVectorMemory`: - -```python -from dapr_agents.memory import ConversationVectorMemory - -memory = ConversationVectorMemory( - vector_store=store, - distance_metric="cosine", -) -``` - -## Tools - -### Agents as Tools - -Dapr Agents supports invoking other agents as tools within an instance of a `DurableAgent` reasoning loop, including agents from other frameworks such as OpenAI Agents, LangGraph, and CrewAI. For full documentation and code examples, see [Agents as Tools]({{% ref "dapr-agents-core-concepts.md#agents-as-tools" %}}). - -### MCP Toolbox for databases - -Dapr Agents support integrating with [MCP Toolbox for Databases](https://googleapis.github.io/genai-toolbox/getting-started/introduction/) by implementing a wrapper that loads the available tools into the `Tool` model Dapr Agents utilize. - -To integrate the Toolbox, load the tools as follows: - -```python -from toolbox_core import ToolboxSyncClient -client = ToolboxSyncClient("http://127.0.0.1:5000") -agent_tools = AgentTool.from_toolbox_many(client.load_toolset("your-tools-name-here")) -agent = DurableAgent( - .. - tools=agent_tools -) - -.. -# Remember to close the tool -finally: - client.close() -``` - -Or wrap it in a `with` statement: - -```python -from toolbox_core import ToolboxSyncClient -with ToolboxSyncClient("http://127.0.0.1:5000") as client: - agent_tools = AgentTool.from_toolbox_many(client.load_toolset("your-tools-name-here")) - agent = DurableAgent( - .. - tools=agent_tools - ) -``` \ No newline at end of file diff --git a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-introduction.md b/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-introduction.md index 283b5f51c6a..93d1c241d34 100644 --- a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-introduction.md +++ b/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-introduction.md @@ -74,7 +74,7 @@ Get started with Dapr Agents by following the instructions on the [Getting Start ### Framework integrations -Dapr Agents integrates with popular Python frameworks and tools. For detailed integration guides and examples, see the [integrations page]({{% ref "developing-ai/dapr-agents/dapr-agents-integrations.md" %}}). +Dapr Agents integrates with popular Python frameworks and tools. For detailed integration guides and examples, see the [integrations page]({{% ref integrations %}}). ## Operational support diff --git a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-quickstarts.md b/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-quickstarts.md index 3dddb500b22..d67d9793bd3 100644 --- a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-quickstarts.md +++ b/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-quickstarts.md @@ -58,3 +58,16 @@ The [Dapr Agents examples](https://github.com/dapr/dapr-agents/tree/main/example | [Agents as Activities with Observability](https://github.com/dapr/dapr-agents/tree/main/examples/07-agents-as-activities-observability) | Trace agent activities end-to-end with OpenTelemetry and Zipkin | | [Agents as Tools](https://github.com/dapr/dapr-agents/tree/main/examples/08-agents-as-tools) | Invoke other `DurableAgent` instances—and agents from other frameworks—as child workflow tools | | [Durable Agent Hot-Reload](https://github.com/dapr/dapr-agents/tree/main/examples/09-durable-agent-hot-reload) | Hot-reload agent persona and LLM settings at runtime without restarting | +| [Echo Agent Executor](https://github.com/dapr/dapr-agents/tree/main/examples/10-agent-executor-echo) | Run an agent with a stateful agent runtime | +| [Agent with Auto-Discovered MCPServer Tools](https://github.com/dapr/dapr-agents/tree/main/examples/10-mcpserver) | Automatically discover tools from loaded MCPServer resources | +| [Expert Agent – Chainlit UI with RAG-via-hook (Tavily)](https://github.com/dapr/dapr-agents/tree/main/examples/11-expert-agent-tavily) | Inject fresh web context into every LLM turn without explicit tool calls | +| [OpenTelemetry Observability on Kubernetes](https://github.com/dapr/dapr-agents/tree/main/examples/demo-otel-k8s) | Configure OpenTelemetry observability for Dapr Agents running on Kubernetes | + + +## Extension Examples + +The following [Dapr Agents examples](https://github.com/dapr/dapr-agents/tree/main/examples) demonstrate provider-specific integrations: + +| Example | What You'll Learn | +|---|---| +| [Drasi Change-Driven Agents on Kubernetes](https://github.com/dapr/dapr-agents/tree/main/examples/ext-drasi-change-driven-agents-k8s) | React to real-time data changes detected by Drasi over Dapr pub/sub | \ No newline at end of file diff --git a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-why.md b/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-why.md index 04093294962..00ec1249ca1 100644 --- a/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-why.md +++ b/daprdocs/content/en/developing-ai/dapr-agents/dapr-agents-why.md @@ -106,7 +106,7 @@ Dapr Agents builds on Dapr's Workflow API, which represents each agent as an act ### Data-centric AI agents -With built-in connectivity to over 50 enterprise data sources, Dapr Agents efficiently handles structured and unstructured data. From basic [PDF extraction]({{% ref "/developing-ai/dapr-agents/dapr-agents-integrations.md" %}}) to large-scale database interactions, it enables data-driven AI workflows with minimal code changes. Dapr's [bindings]({{% ref bindings-overview.md %}}) and [state stores]({{% ref supported-state-stores.md %}}), along with MCP support, provide access to numerous data sources for agent data ingestion. +With built-in connectivity to over 50 enterprise data sources, Dapr Agents efficiently handles structured and unstructured data. From [basic PDF extraction to large-scale database interactions]({{% ref integrations %}}), it enables data-driven AI workflows with minimal code changes. Dapr's [bindings]({{% ref bindings-overview.md %}}) and [state stores]({{% ref supported-state-stores.md %}}), along with MCP support, provide access to numerous data sources for agent data ingestion. ### Accelerated development diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/_index.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/_index.md new file mode 100644 index 00000000000..a0a6512a427 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/_index.md @@ -0,0 +1,12 @@ +--- +type: docs +title: "Integrations" +linkTitle: "Integrations" +description: "Integrations available for Dapr Agents" +weight: 60 +aliases: + - /developing-applications/dapr-agents/dapr-agents-integrations + - /developing-ai/dapr-agents/dapr-agents-integrations +--- + +Dapr Agents offers various integrations that are available out-of-the-box or with a separate installation. diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/loaders/_index.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/loaders/_index.md new file mode 100644 index 00000000000..69c56eaa775 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/loaders/_index.md @@ -0,0 +1,7 @@ +--- +type: docs +title: "Loaders" +linkTitle: "Loaders" +weight: 30 +description: "Integrations for document loading" +--- diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/loaders/arxiv.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/loaders/arxiv.md new file mode 100644 index 00000000000..1fdf6168892 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/loaders/arxiv.md @@ -0,0 +1,190 @@ +--- +type: docs +title: "arXiv" +linkTitle: "arXiv" +weight: 10 +description: "Load research papers with the arXiv API" +--- + +The Arxiv Fetcher module in Dapr Agents provides a powerful interface to interact with the [arXiv API](https://info.arxiv.org/help/api/index.html). It is designed to help users programmatically search for, retrieve, and download scientific papers from arXiv. With advanced querying capabilities, metadata extraction, and support for downloading PDF files, the Arxiv Fetcher is ideal for researchers, developers, and teams working with academic literature. + +### Why Use the Arxiv Fetcher? + +The Arxiv Fetcher simplifies the process of accessing research papers, offering features like: + +* **Automated Literature Search**: Query arXiv for specific topics, keywords, or authors. +* **Metadata Retrieval**: Extract structured metadata, such as titles, abstracts, authors, categories, and submission dates. +* **Precise Filtering**: Limit search results by date ranges (e.g., retrieve the latest research in a field). +* **PDF Downloading**: Fetch full-text PDFs of papers for offline use. + +### How to Use the Arxiv Fetcher + +#### Step 1: Install Required Modules + +{{% alert title="Note" color="primary" %}} +The Arxiv Fetcher relies on a [lightweight Python wrapper](https://github.com/lukasschwab/arxiv.py) for the arXiv API, which is not included in the Dapr Agents core module. This design choice helps maintain modularity and avoids adding unnecessary dependencies for users who may not require this functionality. To use the Arxiv Fetcher, ensure you install the [library](https://pypi.org/project/arxiv/) separately. +{{% /alert %}} + +{{< tabpane text=true >}} + +{{% tab header="pip" %}} + +```bash +pip install arxiv +``` + +{{% /tab %}} + +{{% tab header="uv" %}} + +```bash +uv add arxiv +``` + +{{% /tab %}} + +{{< /tabpane >}} + +#### Step 2: Initialize the Fetcher + +Set up the `ArxivFetcher` to begin interacting with the arXiv API. + +```python +from dapr_agents.document import ArxivFetcher + +# Initialize the fetcher +fetcher = ArxivFetcher() +``` + +#### Step 3: Perform Searches + +**Basic Search by Query String** + +Search for papers using simple keywords. The results are returned as Document objects, each containing: + +* `text`: The abstract of the paper. +* `metadata`: Structured metadata such as title, authors, categories, and submission dates. + +```python +# Search for papers related to "machine learning" +results = fetcher.search(query="machine learning", max_results=5) + +# Display metadata and summaries +for doc in results: + print(f"Title: {doc.metadata['title']}") + print(f"Authors: {', '.join(doc.metadata['authors'])}") + print(f"Summary: {doc.text}\n") +``` + +**Advanced Querying** + +Refine searches using logical operators like AND, OR, and NOT or perform field-specific searches, such as by author. + +Examples: + +Search for papers on "agents" and "cybersecurity": + +```python +results = fetcher.search(query="all:(agents AND cybersecurity)", max_results=10) +``` + +Exclude specific terms (e.g., "quantum" but not "computing"): + +```python +results = fetcher.search(query="all:(quantum NOT computing)", max_results=10) +``` + +Search for papers by a specific author: + +```python +results = fetcher.search(query='au:"John Doe"', max_results=10) +``` + +**Filter Papers by Date** + +Limit search results to a specific time range, such as papers submitted in the last 24 hours. + +```python +from datetime import datetime, timedelta + +# Calculate the date range +last_24_hours = (datetime.now() - timedelta(days=1)).strftime("%Y%m%d") +today = datetime.now().strftime("%Y%m%d") + +# Search for recent papers +recent_results = fetcher.search( + query="all:(agents AND cybersecurity)", + from_date=last_24_hours, + to_date=today, + max_results=5 +) + +# Display metadata +for doc in recent_results: + print(f"Title: {doc.metadata['title']}") + print(f"Authors: {', '.join(doc.metadata['authors'])}") + print(f"Published: {doc.metadata['published']}") + print(f"Summary: {doc.text}\n") +``` + +#### Step 4: Download PDFs + +Fetch the full-text PDFs of papers for offline use. Metadata is preserved alongside the downloaded files. + +```python +import os +from pathlib import Path + +# Create a directory for downloads +os.makedirs("arxiv_papers", exist_ok=True) + +# Download PDFs +download_results = fetcher.search( + query="all:(agents AND cybersecurity)", + max_results=5, + download=True, + dirpath=Path("arxiv_papers") +) + +for paper in download_results: + print(f"Downloaded Paper: {paper['title']}") + print(f"File Path: {paper['file_path']}\n") +``` + +#### Step 5: Extract and Process PDF Content + +Use `PyPDFReader` from `Dapr Agents` to extract content from downloaded PDFs. Each page is treated as a separate Document object with metadata. + +```python +from pathlib import Path +from dapr_agents.document import PyPDFReader + +reader = PyPDFReader() +docs_read = [] + +for paper in download_results: + local_pdf_path = Path(paper["file_path"]) + documents = reader.load(local_pdf_path, additional_metadata=paper) + docs_read.extend(documents) + +# Verify results +print(f"Extracted {len(docs_read)} documents.") +print(f"First document text: {docs_read[0].text}") +print(f"Metadata: {docs_read[0].metadata}") +``` + +### Practical Applications + +The Arxiv Fetcher enables various use cases for researchers and developers: + +* **Literature Reviews**: Quickly retrieve and organize relevant papers on a given topic or by a specific author. +* **Trend Analysis**: Identify the latest research in a domain by filtering for recent submissions. +* **Offline Research Workflows**: Download and process PDFs for local analysis and archiving. + +### Next Steps + +While the Arxiv Fetcher provides robust functionality for retrieving and processing research papers, its output can be integrated into advanced workflows: + +* **Building a Searchable Knowledge Base**: Combine fetched papers with integrations like text splitting and vector embeddings for advanced search capabilities. +* **Retrieval-Augmented Generation (RAG)**: Use processed papers as inputs for RAG pipelines to power question-answering systems. +* **Automated Literature Surveys**: Generate summaries or insights based on the fetched and processed research. diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/providers/_index.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/providers/_index.md new file mode 100644 index 00000000000..6cec6022c22 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/providers/_index.md @@ -0,0 +1,7 @@ +--- +type: docs +title: "Providers" +linkTitle: "Providers" +weight: 10 +description: "Integrations by provider" +--- diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/providers/drasi.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/providers/drasi.md new file mode 100644 index 00000000000..1baa798b5ab --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/providers/drasi.md @@ -0,0 +1,134 @@ +--- +type: docs +title: "Drasi" +linkTitle: "Drasi" +weight: 10 +description: "Integrations for Drasi" +--- + +{{% alert title="Note" color="primary" %}} +This integration is currently in Alpha; breaking changes may occur at any time. +{{% /alert %}} + +The Drasi extension in Dapr Agents enables `DurableAgent` runs to be triggered by Drasi events. + +## Why Use Drasi? + +Many systems need to react to changes produced by other systems in near real-time. Traditional polling cannot detect the absence of change, or changes that occur at an extremely high frequency without unnecessary load on source systems (even without considering network delay). Raw changes are usually not actionable, requiring custom change data capture (CDC) pipelines to process changes at scale and convert them into meaningful domain events. However, these pipelines can be expensive (if managed) or difficult to set up and maintain (if self-hosted). + +For many use cases, [Drasi](https://drasi.io/) is a viable alternative. Drasi is a **CNCF Sandbox** project that addresses the issues mentioned above with a simple architecture centered around detecting and reacting to changes: +- **Sources** to ingest changes from existing systems +- **Queries** allowing high-level "business conditions" to be defined across a variety of data sources, which emit events when those conditions are satisfied +- **Reactions** to push events to downstream consumers + +## Installation + +{{< tabpane text=true >}} + +{{% tab header="pip" %}} + +```bash +pip install "dapr-agents[drasi]" +``` + +{{% /tab %}} + +{{% tab header="uv" %}} + +```bash +uv add dapr-agents[drasi] +``` + +{{% /tab %}} + +{{< /tabpane >}} + +## Usage + +```python +from dapr_agents import AgentRunner, DurableAgent +from dapr_agents.agents.configs import AgentPubSubConfig + +from dapr_agents.ext.drasi import drasi_trigger + +agent = DurableAgent( + name="InventoryAgent", + pubsub=AgentPubSubConfig( + pubsub_name="pubsub", # Replace with your pub/sub component + agent_topic="inventory-agent", # Replace with your agent pub/sub topic + ), +) + +drasi_trigger( + agent, + query_id="low-stock-products", # Replace with your Drasi query ID +) + +AgentRunner().serve(agent) +``` + +## API + +### `drasi_trigger` + +`drasi_trigger` creates a static subscription to a Drasi query and allows agents to be triggered by Drasi events via Dapr pub/sub. + +#### Parameters + +Parameter | Type | Required | Details | Example +--------- | ---- | -------- | ------- | ------- +`agent` | `DurableAgent` | Y | The target agent. | N/A +`query_id` | `str` | Y | The Drasi query ID to subscribe to. | `"low-stock-products"` +`pubsub` | `str` | N | The name of the Dapr pub/sub component to use. Defaults to the agent's pub/sub component. | `"pubsub"` +`topic` | `str` | N | The topic to subscribe to. Defaults to `"drasi-events-" + query_id`. | N/A +`dead_letter_topic` | `str` | N | Dead-letter topic to publish failed messages to. | `"low-stock-events-dlq"` +`task_mapper` | `Callable[[DrasiChangeEvent, MessageContext], TriggerAction]` | N | Callable to map Drasi change events to agent task messages. Defaults to instructing the agent to return the serialized Drasi event as-is. | N/A +`operations` | `DrasiOperation \| str \| list[DrasiOperation \| str]` | N | Drasi operation(s) to filter change events by. Accepts `DrasiOperation` or equivalent string literals. | N/A +`change_model` | `type[Any]` | N | Model to use to validate the change data in Drasi events. | N/A + +### `DrasiOperation` + +`DrasiOperation` is an enum representing the supported Drasi change operations. + +#### Operations + +Operation | Value | Details +--------- | ----- | ------- +`DrasiOperation.i` | `"i"` | A record was added to the result set tracked by the Drasi query. +`DrasiOperation.u` | `"u"` | A record was updated in the result set tracked by the Drasi query. +`DrasiOperation.d` | `"d"` | A record was deleted from the result set tracked by the Drasi query. + +### `DrasiChangeEvent` + +`DrasiChangeEvent` is a Pydantic model representing a change event emitted by a query. + +#### Attributes + +Attribute | Type | Required | Details | Example +--------- | ---- | -------- | ------- | ------- +`op` | `DrasiOperation` | Y | The change event operation (insert, update, delete). | `DrasiOperation.u` +`ts_ms` | `int` | Y | The timestamp of the change event in milliseconds. | `42` +`seq` | `int` | Y | The sequence number of the change event. | `1` +`payload` | `dict[str, Any]` | Y | The change data for the change event. | `{"source": {"queryId": "low-stock-products", "ts_ms": 42}, "before": {"a": 1}, "after": {"a": 2}}` + +#### Example Structure + +```json +{ + "op": "u", + "ts_ms": 42, + "seq": 1, + "payload": { + "source": { + "queryId": "low-stock-products", + "ts_ms": 42 + }, + "before": {"a": 1}, + "after": {"a": 2} + } +} +``` + +## Examples + +See the [Extension Examples]({{< ref "developing-ai/dapr-agents/dapr-agents-quickstarts.md#extension-examples" >}}) for a list of working examples for the Drasi extension. diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/splitters/_index.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/splitters/_index.md new file mode 100644 index 00000000000..42781771246 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/splitters/_index.md @@ -0,0 +1,7 @@ +--- +type: docs +title: "Splitters" +linkTitle: "Splitters" +weight: 40 +description: "Integrations for document splitting" +--- diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/splitters/text.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/splitters/text.md new file mode 100644 index 00000000000..d09ca052ecc --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/splitters/text.md @@ -0,0 +1,167 @@ +--- +type: docs +title: "Text" +linkTitle: "Text" +weight: 10 +description: "Split text documents into chunks" +--- + +The Text Splitter module is a foundational integration in Dapr Agents designed to preprocess documents for use in [Retrieval-Augmented Generation (RAG)](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) workflows and other in-context learning applications. Its primary purpose is to break large documents into smaller, meaningful chunks that can be embedded, indexed, and efficiently retrieved based on user queries. + +By focusing on manageable chunk sizes and preserving contextual integrity through overlaps, the Text Splitter ensures documents are processed in a way that supports downstream tasks like question answering, summarization, and document retrieval. + +### Why Use a Text Splitter? + +When building RAG pipelines, splitting text into smaller chunks serves these key purposes: + +* **Enabling Effective Indexing**: Chunks are embedded and stored in a vector database, making them retrievable based on similarity to user queries. +* **Maintaining Semantic Coherence**: Overlapping chunks help retain context across splits, ensuring the system can connect related pieces of information. +* **Handling Model Limitations**: Many models have input size limits. Splitting ensures text fits within these constraints while remaining meaningful. + +This step is crucial for preparing knowledge to be embedded into a searchable format, forming the backbone of retrieval-based workflows. + +### Strategies for Text Splitting + +The Text Splitter supports multiple strategies to handle different types of documents effectively. These strategies balance the size of each chunk with the need to maintain context. + +#### 1. Character-Based Length + +* **How It Works**: Counts the number of characters in each chunk. +* **Use Case**: Simple and effective for text splitting without dependency on external tokenization tools. + +Example: + +```python +from dapr_agents.document.splitter.text import TextSplitter + +# Character-based splitter (default) +splitter = TextSplitter(chunk_size=1024, chunk_overlap=200) +``` + +#### 2. Token-Based Length + +* **How It Works**: Counts tokens, which are the semantic units used by language models (e.g., words or subwords). +* **Use Case**: Ensures compatibility with models like GPT, where token limits are critical. + +**Example**: + +```python +import tiktoken +from dapr_agents.document.splitter.text import TextSplitter + +enc = tiktoken.get_encoding("cl100k_base") + +def length_function(text: str) -> int: + return len(enc.encode(text)) + +splitter = TextSplitter( + chunk_size=1024, + chunk_overlap=200, + chunk_size_function=length_function +) +``` + +The flexibility to define the chunk size function makes the Text Splitter adaptable to various scenarios. + +### Chunk Overlap + +To preserve context, the Text Splitter includes a chunk overlap feature. This ensures that parts of one chunk carry over into the next, helping maintain continuity when chunks are processed sequentially. + +Example: + +* With `chunk_size=1024` and `chunk_overlap=200`, the last `200` tokens or characters of one chunk appear at the start of the next. +* This design helps in tasks like text generation, where maintaining context across chunks is essential. + +### How to Use the Text Splitter + +Here's a practical example of using the Text Splitter to process a PDF document: + +#### Step 1: Load a PDF + +```python +import requests +from pathlib import Path + +# Download PDF +pdf_url = "https://arxiv.org/pdf/2412.05265.pdf" +local_pdf_path = Path("arxiv_paper.pdf") + +if not local_pdf_path.exists(): + response = requests.get(pdf_url) + response.raise_for_status() + with open(local_pdf_path, "wb") as pdf_file: + pdf_file.write(response.content) +``` + +#### Step 2: Read the Document + +For this example, we use Dapr Agents' `PyPDFReader`. + +{{% alert title="Note" color="primary" %}} +The PyPDF Reader relies on the [pypdf python library](https://pypi.org/project/pypdf/), which is not included in the Dapr Agents core module. This design choice helps maintain modularity and avoids adding unnecessary dependencies for users who may not require this functionality. To use the PyPDF Reader, ensure that you install the library separately. +{{% /alert %}} + +{{< tabpane text=true >}} + +{{% tab header="pip" %}} + +```bash +pip install pypdf +``` + +{{% /tab %}} + +{{% tab header="uv" %}} + +```bash +uv add pypdf +``` + +{{% /tab %}} + +{{< /tabpane >}} + +Then, initialize the reader to load the PDF file. + +```python +from dapr_agents.document.reader.pdf.pypdf import PyPDFReader + +reader = PyPDFReader() +documents = reader.load(local_pdf_path) +``` + +#### Step 3: Split the Document + +```python +import tiktoken +from dapr_agents.document.splitter.text import TextSplitter + +enc = tiktoken.get_encoding("cl100k_base") + +def length_function(text: str) -> int: + return len(enc.encode(text)) + +splitter = TextSplitter( + chunk_size=1024, + chunk_overlap=200, + chunk_size_function=length_function +) +chunked_documents = splitter.split_documents(documents) +``` + +#### Step 4: Analyze Results + +```python +print(f"Original document pages: {len(documents)}") +print(f"Total chunks: {len(chunked_documents)}") +print(f"First chunk: {chunked_documents[0]}") +``` + +### Key Features + +* **Hierarchical Splitting**: Splits text by separators (e.g., paragraphs), then refines chunks further if needed. +* **Customizable Chunk Size**: Supports character-based and token-based length functions. +* **Overlap for Context**: Retains portions of one chunk in the next to maintain continuity. +* **Metadata Preservation**: Each chunk retains metadata like page numbers and start/end indices for easier mapping. + +By understanding and leveraging the `Text Splitter`, you can preprocess large documents effectively, ensuring they are ready for embedding, indexing, and retrieval in advanced workflows like RAG pipelines. diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/tools/_index.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/tools/_index.md new file mode 100644 index 00000000000..e3ca61f25f9 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/tools/_index.md @@ -0,0 +1,7 @@ +--- +type: docs +title: "Tools" +linkTitle: "Tools" +weight: 70 +description: "Integrations for tools" +--- diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/tools/agents-as-tools.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/tools/agents-as-tools.md new file mode 100644 index 00000000000..d8c11e8a845 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/tools/agents-as-tools.md @@ -0,0 +1,11 @@ +--- +type: docs +title: "Agents as Tools" +linkTitle: "Agents as Tools" +weight: 10 +description: "Invoke heterogeneous agents as tools" +--- + +Dapr Agents supports invoking other agents as tools within an instance of a `DurableAgent` reasoning loop, including agents from other frameworks such as OpenAI Agents, LangGraph, and CrewAI. + +For full documentation and code examples, see [Agents as Tools]({{% ref "dapr-agents-core-concepts.md#agents-as-tools" %}}). diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/tools/mcp-toolbox.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/tools/mcp-toolbox.md new file mode 100644 index 00000000000..1067a749e37 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/tools/mcp-toolbox.md @@ -0,0 +1,37 @@ +--- +type: docs +title: "MCP Toolbox for Databases" +linkTitle: "MCP Toolbox for Databases" +weight: 50 +description: "Load MCP Toolbox tools for interacting with databases" +--- + +Dapr Agents supports integrating with [MCP Toolbox for Databases](https://mcp-toolbox.dev/documentation/introduction/) by implementing a wrapper that loads the available tools into the `Tool` model Dapr Agents utilize. + +To integrate the Toolbox, load the tools as follows: + +```python +from toolbox_core import ToolboxSyncClient + +client = ToolboxSyncClient("http://127.0.0.1:5000") +try: + agent_tools = AgentTool.from_toolbox_many(client.load_toolset("your-tools-name-here")) + agent = DurableAgent( + # ... + tools=agent_tools, + ) +finally: + client.close() +``` + +Or wrap it in a `with` statement: + +```python +from toolbox_core import ToolboxSyncClient +with ToolboxSyncClient("http://127.0.0.1:5000") as client: + agent_tools = AgentTool.from_toolbox_many(client.load_toolset("your-tools-name-here")) + agent = DurableAgent( + # ... + tools=agent_tools, + ) +``` diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/_index.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/_index.md new file mode 100644 index 00000000000..d1b8cf379b7 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/_index.md @@ -0,0 +1,28 @@ +--- +type: docs +title: "Vector Stores" +linkTitle: "Vector Stores" +weight: 60 +description: "Integrations for vector stores" +--- + +Dapr Agents includes built-in vector store implementations for use with `ConversationVectorMemory` and RAG pipelines. Each store is available from `dapr_agents.storage.vectorstores`. + +Vector stores share the same interface and are interchangeable as the `vector_store` argument to `ConversationVectorMemory`: + +```python +from dapr_agents.storage.vectorstores import ChromaVectorStore # Replace with your vector store +from dapr_agents.document.embedder.openai import OpenAIEmbedder # Replace with your embedding model +from dapr_agents.memory import ConversationVectorMemory + +store = ChromaVectorStore( + collection_name="my_collection", + embedding_function=OpenAIEmbedder(), +) +memory = ConversationVectorMemory( + vector_store=store, + distance_metric="cosine", +) +``` + +To keep the core installation minimal, vector store dependencies must be installed separately. diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/chroma.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/chroma.md new file mode 100644 index 00000000000..30f05307e0d --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/chroma.md @@ -0,0 +1,43 @@ +--- +type: docs +title: "Chroma" +linkTitle: "Chroma" +weight: 10 +description: "Perform similarity searches with in-memory or persistent Chroma storage" +--- + +Uses [ChromaDB](https://www.trychroma.com/) for in-memory or persistent vector search. + +## Installation + +{{< tabpane text=true >}} + +{{% tab header="pip" %}} + +```bash +pip install chromadb +``` + +{{% /tab %}} + +{{% tab header="uv" %}} + +```bash +uv add chromadb +``` + +{{% /tab %}} + +{{< /tabpane >}} + +## Usage + +```python +from dapr_agents.storage.vectorstores import ChromaVectorStore +from dapr_agents.document.embedder.openai import OpenAIEmbedder # Replace with your embedding model + +store = ChromaVectorStore( + collection_name="my_collection", + embedding_function=OpenAIEmbedder(), +) +``` diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/postgres.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/postgres.md new file mode 100644 index 00000000000..1ef4cb1baf2 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/postgres.md @@ -0,0 +1,44 @@ +--- +type: docs +title: "Postgres" +linkTitle: "Postgres" +weight: 30 +description: "Perform similarity searches with persistent Postgres storage" +--- + +Uses [Postgres with pgvector](https://github.com/pgvector/pgvector) for production-grade vector similarity search. + +## Installation + +{{< tabpane text=true >}} + +{{% tab header="pip" %}} + +```bash +pip install "psycopg[binary,pool]" pgvector +``` + +{{% /tab %}} + +{{% tab header="uv" %}} + +```bash +uv add 'psycopg[binary,pool]' pgvector +``` + +{{% /tab %}} + +{{< /tabpane >}} + +## Usage + +```python +from dapr_agents.storage.vectorstores import PostgresVectorStore +from dapr_agents.document.embedder.openai import OpenAIEmbedder # Replace with your embedding model + +store = PostgresVectorStore( + connection_string="postgresql://user:pass@localhost:5432/mydb", + embedding_function=OpenAIEmbedder(), + embedding_dimensions=1536, +) +``` diff --git a/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/redis.md b/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/redis.md new file mode 100644 index 00000000000..1dad44f9408 --- /dev/null +++ b/daprdocs/content/en/developing-ai/dapr-agents/integrations/vector-stores/redis.md @@ -0,0 +1,51 @@ +--- +type: docs +title: "Redis" +linkTitle: "Redis" +weight: 50 +description: "Perform similarity searches with in-memory or persistent Redis storage" +--- + +Uses [Redis Stack](https://redis.io/docs/latest/develop/ai/search-and-query/vectors/) via the `redisvl` library for vector similarity search. + +## Installation + +{{% alert title="Note" color="primary" %}} +The Redis instance started by `dapr init` is a **vanilla Redis server** and does **not** include the Search/vector modules required by Redis Stack. To use `RedisVectorStore`, you must run [Redis Stack](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/) (or a Redis deployment with the `RediSearch` module enabled) separately. +{{% /alert %}} + +{{< tabpane text=true >}} + +{{% tab header="pip" %}} + +```bash +pip install redisvl +``` + +{{% /tab %}} + +{{% tab header="uv" %}} + +```bash +uv add redisvl +``` + +{{% /tab %}} + +{{< /tabpane >}} + +## Usage + +```python +from dapr_agents.storage.vectorstores import RedisVectorStore +from dapr_agents.document.embedder.openai import OpenAIEmbedder # Replace with your embedding model + +store = RedisVectorStore( + url="redis://localhost:6379", + index_name="my_agent", + embedding_function=OpenAIEmbedder(), + embedding_dimensions=1536, + distance_metric="cosine", # "cosine", "l2", or "ip" + storage_type="hash", # "hash" or "json" +) +```