Skip to content

Added documentation for guardrail config's streamProcessingMode #564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dlew wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Added documentation for guardrail config's streamProcessingMode #564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions content/docs/configuration/librechat_yaml/object_structure/aws_bedrock.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ endpoints:
guardrailIdentifier: "your-guardrail-id"
guardrailVersion: "1"
trace: "enabled"
streamProcessingMode: "sync"
```

> **Note:** AWS Bedrock endpoint supports all [Shared Endpoint Settings](/docs/configuration/librechat_yaml/object_structure/shared_endpoint_settings), including `streamRate`, `titleModel`, `titleMethod`, `titlePrompt`, `titlePromptTemplate`, and `titleEndpoint`. The settings shown below are specific to Bedrock or have Bedrock-specific defaults.
Expand Down Expand Up @@ -148,6 +149,7 @@ endpoints:
['guardrailIdentifier', 'String', 'The unique identifier of the guardrail to apply.', 'Required when using guardrails.'],
['guardrailVersion', 'String', 'The version of the guardrail to use.', 'Required when using guardrails.'],
['trace', 'String', 'Controls guardrail trace output for debugging. Options: "enabled", "enabled_full", or "disabled".', 'Optional. Default: "disabled"'],
['streamProcessingMode', 'String', 'Controls guardrail stream processing mode. Options: "sync" or "async".', 'Optional. Default: "sync"'],
]}
/>

Expand All @@ -159,12 +161,14 @@ endpoints:
guardrailIdentifier: "abc123xyz"
guardrailVersion: "1"
trace: "enabled"
streamProcessingMode: "sync"
```

**Notes:**
- Guardrails help ensure responsible AI usage by filtering harmful content, PII, and other sensitive information
- The `guardrailIdentifier` can be found in the AWS Bedrock console under Guardrails
- Set `trace` to `"enabled"` or `"enabled_full"` during development to see which guardrail policies are triggered
- Set `streamProcessingMode` to `"async"` to stream responses faster (at the cost of guardrail possibly allowing inappropriate content through until its scan completes)
- For production, set `trace` to `"disabled"` to reduce response payload size

## Notes
Expand Down
2 changes: 2 additions & 0 deletions content/docs/configuration/pre_configured_ai/bedrock.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ endpoints:
guardrailIdentifier: "abc123xyz"
guardrailVersion: "1"
trace: "enabled"
streamProcessingMode: "sync"
```

- `streamRate`: (Optional) Set the rate of processing each new token in milliseconds.
Expand All @@ -81,6 +82,7 @@ endpoints:
- `guardrailIdentifier`: The guardrail ID or ARN from your AWS Bedrock Console.
- `guardrailVersion`: The guardrail version number (e.g., `"1"`) or `"DRAFT"`.
- `trace`: (Optional) Enable trace logging: `"enabled"`, `"disabled"`, or `"enabled_full"`.
- `streamProcessingMode`: (Optional) Set stream processing mode: `"sync"` or `"async"` (defaults to `"sync"`).
- See [AWS Bedrock Guardrails documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-how.html) for creating and managing guardrails.

## Inference Profiles
Expand Down