diff --git a/docs/book/component-guide/alerters/README.md b/docs/book/component-guide/alerters/README.md index 11136043cb4..8b03a96482a 100644 --- a/docs/book/component-guide/alerters/README.md +++ b/docs/book/component-guide/alerters/README.md @@ -5,14 +5,11 @@ icon: message-exclamation # Alerters -**Alerters** allow you to send messages to chat services (like Slack, Discord, Mattermost, etc.) from within your -pipelines. This is useful to immediately get notified when failures happen, for general monitoring/reporting, and also -for building human-in-the-loop ML. +**Alerters** allow you to send messages to chat services (like Slack, Discord, Mattermost, etc.) from within your pipelines. This is useful to immediately get notified when failures happen, for general monitoring/reporting, and also for building human-in-the-loop ML. ## Alerter Flavors -Currently, the [SlackAlerter](slack.md) and [DiscordAlerter](discord.md) are the available alerter integrations. However, it is straightforward to -extend ZenML and [build an alerter for other chat services](custom.md). +Currently, the [SlackAlerter](slack.md) and [DiscordAlerter](discord.md) are the available alerter integrations. However, it is straightforward to extend ZenML and [build an alerter for other chat services](custom.md). | Alerter | Flavor | Integration | Notes | |------------------------------------|-----------|-------------|--------------------------------------------------------------------| @@ -49,8 +46,7 @@ zenml stack register ... -al Stacks can have multiple alerters attached. ZenML treats the first attached alerter as the default one, and singular accessors such as `Client().active_stack.alerter` resolve to that default alerter. {% endhint %} -Afterward, you can import the alerter standard steps provided by the respective integration and directly use them in -your pipelines. +Afterward, you can import the alerter standard steps provided by the respective integration and directly use them in your pipelines. ## Using the Ask Step for Human-in-the-Loop Workflows diff --git a/docs/book/component-guide/alerters/slack.md b/docs/book/component-guide/alerters/slack.md index 6160714b365..fcc0ba12a68 100644 --- a/docs/book/component-guide/alerters/slack.md +++ b/docs/book/component-guide/alerters/slack.md @@ -4,19 +4,16 @@ description: Sending automated alerts to a Slack channel. # Slack Alerter -The `SlackAlerter` enables you to send messages or ask questions within a -dedicated Slack channel directly from within your ZenML pipelines and steps. +The `SlackAlerter` enables you to send messages or ask questions within a dedicated Slack channel directly from within your ZenML pipelines and steps. ## How to Create ### Set up a Slack app -In order to use the `SlackAlerter`, you first need to have a Slack workspace -set up with a channel that you want your pipelines to post to. +In order to use the `SlackAlerter`, you first need to have a Slack workspace set up with a channel that you want your pipelines to post to. Then, you need to [create a Slack App](https://api.slack.com/apps?new\_app=1) -with a bot in your workspace. Make sure to give it the following permissions in -the `OAuth & Permissions` tab under `Scopes`: +with a bot in your workspace. Make sure to give it the following permissions in the `OAuth & Permissions` tab under `Scopes`: * `chat:write`, * `channels:read` @@ -24,28 +21,24 @@ the `OAuth & Permissions` tab under `Scopes`: ![Slack OAuth Permissions](../../.gitbook/assets/slack-alerter-oauth-permissions.png) -In order to be able to use the `ask()` functionality, you need to invite the app -to your channel. You can either use the `/invite` command directly in the +In order to be able to use the `ask()` functionality, you need to invite the app to your channel. You can either use the `/invite` command directly in the desired channel or add it through the channel settings: ![Slack OAuth Permissions](../../.gitbook/assets/slack-channel-settings.png) {% hint style="warning" %} -It might take some time for your app to register within your workspace and -show up in the available list of applications. +It might take some time for your app to register within your workspace and show up in the available list of applications. {% endhint %} ### Registering a Slack Alerter in ZenML -To create a `SlackAlerter`, you first need to install ZenML's `slack` -integration: +To create a `SlackAlerter`, you first need to install ZenML's `slack` integration: ```shell zenml integration install slack -y ``` -Once the integration is installed, you can use the ZenML CLI to create a -secret and register an alerter linked to the app you just created: +Once the integration is installed, you can use the ZenML CLI to create a secret and register an alerter linked to the app you just created: ```shell zenml secret create slack_token --oauth_token= @@ -70,8 +63,7 @@ Slack app settings under `OAuth & Permissions`. ![Slack Token Image](../../.gitbook/assets/slack-alerter-token.png) -After you have registered the `slack_alerter`, you can add it to your stack -like this: +After you have registered the `slack_alerter`, you can add it to your stack like this: ```shell zenml stack register ... -al slack_alerter --set @@ -83,8 +75,7 @@ In ZenML, you can use alerters in various ways. ### Use the `post()` and `ask()` directly -You can use the client to fetch the active alerter within your stack and -use the `post` and `ask` methods directly: +You can use the client to fetch the active alerter within your stack and use the `post` and `ask` methods directly: ```python from zenml import pipeline, step @@ -120,8 +111,7 @@ In case of an error, the output of the `ask()` method default to `False`. ### Use it with custom settings -The Slack alerter comes equipped with a set of options that you can set during -runtime: +The Slack alerter comes equipped with a set of options that you can set during runtime: ```python from zenml import pipeline, step @@ -221,8 +211,7 @@ if __name__ == "__main__": ### Use the predefined steps -If you want to only use it in a simple manner, you can also use the steps -`slack_alerter_post_step` and `slack_alerter_ask_step`, that are built-in to +If you want to only use it in a simple manner, you can also use the steps `slack_alerter_post_step` and `slack_alerter_ask_step`, that are built-in to the Slack integration of ZenML: ```python @@ -269,7 +258,6 @@ The `ask()` method and `slack_alerter_ask_step` recognize these keywords by defa **Slack Case Handling**: The Slack alerter implementation automatically converts all response keywords to lowercase before matching, making responses case-insensitive. You can respond with `LGTM`, `lgtm`, or `Lgtm` - they'll all work. {% endhint %} -For more information and a full list of configurable attributes of the Slack -alerter, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-slack.html#zenml.integrations.slack) . +For more information and a full list of configurable attributes of the Slack alerter, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-slack.html#zenml.integrations.slack) .
ZenML Scarf
diff --git a/docs/book/component-guide/annotators/README.md b/docs/book/component-guide/annotators/README.md index 508d3c89ed3..7a89eb5a16f 100644 --- a/docs/book/component-guide/annotators/README.md +++ b/docs/book/component-guide/annotators/README.md @@ -5,53 +5,44 @@ description: Annotating the data in your workflow. # Annotators -Annotators are a stack component that enables the use of data annotation as part of your ZenML stack and pipelines. You -can use the associated CLI command to launch annotation, configure your datasets and get stats on how many labeled tasks +Annotators are a stack component that enables the use of data annotation as part of your ZenML stack and pipelines. You can use the associated CLI command to launch annotation, configure your datasets and get stats on how many labeled tasks you have ready for use. -Data annotation/labeling is a core part of MLOps that is frequently left out of the conversation. ZenML will -incrementally start to build features that support an iterative annotation workflow that sees the people doing +Data annotation/labeling is a core part of MLOps that is frequently left out of the conversation. ZenML will incrementally start to build features that support an iterative annotation workflow that sees the people doing labeling (and their workflows/behaviors) as integrated parts of their ML process(es). ![When and where to annotate.](../../.gitbook/assets/annotation-when-where.png) There are a number of different places in the ML lifecycle where this can happen: -* **At the start**: You might be starting out without any data, or with a ton of data but no clear sense of which parts - of it are useful to your particular problem. It’s not uncommon to have a lot of data but to be lacking accurate labels +* **At the start**: You might be starting out without any data, or with a ton of data but no clear sense of which parts of it are useful to your particular problem. It’s not uncommon to have a lot of data but to be lacking accurate labels for that data. So you can start and get great value from bootstrapping your model: label some data, train your model, and use your model to suggest labels allowing you to speed up your labeling, iterating on and on in this way. Labeling data early on in the process also helps clarify and condense down your specific rules and standards. For example, you might realize that you need to have specific definitions for certain concepts so that your labeling efforts are consistent across your team. -* **As new data comes in**: New data will likely continue to come in, and you might want to check in with the labeling - process at regular intervals to expose yourself to this new data. (You’ll probably also want to have some kind of +* **As new data comes in**: New data will likely continue to come in, and you might want to check in with the labeling process at regular intervals to expose yourself to this new data. (You’ll probably also want to have some kind of automation around detecting data or concept drift, but for certain kinds of unstructured data you probably can never completely abandon the instant feedback of actual contact with the raw data.) -* **Samples generated for inference**: Your model will be making predictions on real-world data being passed in. If you - store and label this data, you’ll gain a valuable set of data that you can use to compare your labels with what the +* **Samples generated for inference**: Your model will be making predictions on real-world data being passed in. If you store and label this data, you’ll gain a valuable set of data that you can use to compare your labels with what the model was predicting, another possible way to flag drifts of various kinds. This data can then (subject to privacy/user consent) be used in retraining or fine-tuning your model. -* **Other ad hoc interventions**: You will probably have some kind of process to identify bad labels, or to find the - kinds of examples that your model finds really difficult to make correct predictions. For these, and for areas where +* **Other ad hoc interventions**: You will probably have some kind of process to identify bad labels, or to find the kinds of examples that your model finds really difficult to make correct predictions. For these, and for areas where you have clear class imbalances, you might want to do ad hoc annotation to supplement the raw materials your model has to learn from. -ZenML currently offers standard steps that help you tackle the above use cases, but the stack component and abstraction -will continue to be developed to make it easier to use. +ZenML currently offers standard steps that help you tackle the above use cases, but the stack component and abstraction will continue to be developed to make it easier to use. ### When to use it -The annotator is an optional stack component in the ZenML Stack. We designed our abstraction to fit into the larger ML -use cases, particularly the training and deployment parts of the lifecycle. +The annotator is an optional stack component in the ZenML Stack. We designed our abstraction to fit into the larger ML use cases, particularly the training and deployment parts of the lifecycle. The core parts of the annotation workflow include: * using labels or annotations in your training steps in a seamless way * handling the versioning of annotation data * allow for the conversion of annotation data to and from custom formats -* handle annotator-specific tasks, for example, the generation of UI config files that Label Studio requires for the web - annotation interface +* handle annotator-specific tasks, for example, the generation of UI config files that Label Studio requires for the web annotation interface ### List of available annotators @@ -75,21 +66,16 @@ zenml annotator flavor list ### How to use it The available implementation of the annotator is built on top of the Label -Studio integration, which means that using an annotator currently is no -different from what's described on the [Label Studio page: How to use -it?](label-studio.md#how-do-you-use-it). ([Pigeon](pigeon.md) is also supported, but has a -very limited functionality and only works within Jupyter notebooks.) +Studio integration, which means that using an annotator currently is no different from what's described on the [Label Studio page: How to use +it?](label-studio.md#how-do-you-use-it). ([Pigeon](pigeon.md) is also supported, but has a very limited functionality and only works within Jupyter notebooks.) ### A note on names -The various annotation tools have mostly standardized around the naming of key concepts as part of how they build their -tools. Unfortunately, this hasn't been completely unified so ZenML takes an opinion on which names we use for our stack +The various annotation tools have mostly standardized around the naming of key concepts as part of how they build their tools. Unfortunately, this hasn't been completely unified so ZenML takes an opinion on which names we use for our stack components and integrations. Key differences to note: -* Label Studio refers to the grouping of a set of annotations/tasks as a 'Project', whereas most other tools use the - term 'Dataset', so ZenML also calls this grouping a 'Dataset'. -* The individual meta-unit for 'an annotation + the source data' is referred to in different ways, but at ZenML (and - with Label Studio) we refer to them as 'tasks'. +* Label Studio refers to the grouping of a set of annotations/tasks as a 'Project', whereas most other tools use the term 'Dataset', so ZenML also calls this grouping a 'Dataset'. +* The individual meta-unit for 'an annotation + the source data' is referred to in different ways, but at ZenML (and with Label Studio) we refer to them as 'tasks'. The remaining core concepts ('annotation' and 'prediction', in particular) are broadly used among annotation tools. diff --git a/docs/book/component-guide/annotators/pigeon.md b/docs/book/component-guide/annotators/pigeon.md index 85dc1db5e81..17ed4077734 100644 --- a/docs/book/component-guide/annotators/pigeon.md +++ b/docs/book/component-guide/annotators/pigeon.md @@ -96,18 +96,14 @@ You can also use the `zenml annotator dataset` commands to manage your datasets: * `zenml annotator dataset delete ` - Delete a specific dataset * `zenml annotator dataset stats ` - Get statistics for a specific dataset -Annotation files are saved as JSON files in the specified output directory. Each -annotation file represents a dataset, with the filename serving as the dataset +Annotation files are saved as JSON files in the specified output directory. Each annotation file represents a dataset, with the filename serving as the dataset name. ## Acknowledgements -Pigeon was created by [Anastasis Germanidis](https://github.com/agermanidis) and -released as a [Python package](https://pypi.org/project/pigeon-jupyter/) and -[Github repository](https://github.com/agermanidis/pigeon). It is licensed under -the Apache License. It has been updated to work with more recent `ipywidgets` -versions and some small UI improvements were added. We are grateful to Anastasis -for creating this tool and making it available to the community. +Pigeon was created by [Anastasis Germanidis](https://github.com/agermanidis) and released as a [Python package](https://pypi.org/project/pigeon-jupyter/) and +[Github repository](https://github.com/agermanidis/pigeon). It is licensed under the Apache License. It has been updated to work with more recent `ipywidgets` +versions and some small UI improvements were added. We are grateful to Anastasis for creating this tool and making it available to the community.
ZenML Scarf
diff --git a/docs/book/component-guide/annotators/prodigy.md b/docs/book/component-guide/annotators/prodigy.md index a350ae6a9b1..ed67e44fdb8 100644 --- a/docs/book/component-guide/annotators/prodigy.md +++ b/docs/book/component-guide/annotators/prodigy.md @@ -4,10 +4,8 @@ description: Annotating data using Prodigy. # Prodigy -[Prodigy](https://prodi.gy/) is a modern annotation tool for creating training -and evaluation data for machine learning models. You can also use Prodigy to -help you inspect and clean your data, do error analysis and develop rule-based -systems to use in combination with your statistical models. +[Prodigy](https://prodi.gy/) is a modern annotation tool for creating training and evaluation data for machine learning models. You can also use Prodigy to +help you inspect and clean your data, do error analysis and develop rule-based systems to use in combination with your statistical models. ![Prodigy Annotator](../../.gitbook/assets/prodigy-annotator.png) @@ -16,24 +14,19 @@ Prodigy is a paid tool. A license is required to download and use it with ZenML. {% endhint %} -The Prodigy Python library includes a range of pre-built workflows and -command-line commands for various tasks, and well-documented components for -implementing your own workflow scripts. Your scripts can specify how the data is -loaded and saved, change which questions are asked in the annotation interface, -and can even define custom HTML and JavaScript to change the behavior of the -front-end. The web application is optimized for fast, intuitive and efficient +The Prodigy Python library includes a range of pre-built workflows and command-line commands for various tasks, and well-documented components for +implementing your own workflow scripts. Your scripts can specify how the data is loaded and saved, change which questions are asked in the annotation interface, +and can even define custom HTML and JavaScript to change the behavior of the front-end. The web application is optimized for fast, intuitive and efficient annotation. ### When would you want to use it? -If you need to label data as part of your ML workflow, that is the point at -which you could consider adding the optional annotator stack component as part +If you need to label data as part of your ML workflow, that is the point at which you could consider adding the optional annotator stack component as part of your ZenML stack. ### How to deploy it? -The Prodigy Annotator flavor is provided by the Prodigy ZenML integration. You -need to install it to be able to register it as an Annotator and add it to your +The Prodigy Annotator flavor is provided by the Prodigy ZenML integration. You need to install it to be able to register it as an Annotator and add it to your stack: ```shell @@ -41,8 +34,7 @@ zenml integration export-requirements --output-file prodigy-requirements.txt pro ``` Note that you'll need to install Prodigy separately since it requires a license. -Please [visit the Prodigy docs](https://prodi.gy/docs/install) for information -on how to install it. Currently Prodigy also requires the `urllib3<2` +Please [visit the Prodigy docs](https://prodi.gy/docs/install) for information on how to install it. Currently Prodigy also requires the `urllib3<2` dependency, so make sure to install that. Then register your annotator with ZenML: @@ -52,8 +44,7 @@ zenml annotator register prodigy --flavor prodigy # optionally also pass in --custom_config_path="" ``` -See https://prodi.gy/docs/install#config for more on custom Prodigy config -files. Passing a `custom_config_path` allows you to override the default Prodigy +See https://prodi.gy/docs/install#config for more on custom Prodigy config files. Passing a `custom_config_path` allows you to override the default Prodigy config. Finally, add all these components to a stack and set it as your active stack. @@ -67,36 +58,29 @@ zenml stack set annotation zenml stack describe ``` -Now if you run a simple CLI command like `zenml annotator dataset list` this -should work without any errors. You're ready to use your annotator in your ML +Now if you run a simple CLI command like `zenml annotator dataset list` this should work without any errors. You're ready to use your annotator in your ML workflow! ### How do you use it? -With Prodigy, there is no need to specially start the annotator ahead of time -like with [Label Studio](label-studio.md). Instead, just use Prodigy as per the -[Prodigy docs](https://prodi.gy) and then you can use the ZenML wrapper / API to -get your labeled data etc using our Python methods. +With Prodigy, there is no need to specially start the annotator ahead of time like with [Label Studio](label-studio.md). Instead, just use Prodigy as per the +[Prodigy docs](https://prodi.gy) and then you can use the ZenML wrapper / API to get your labeled data etc using our Python methods. ZenML supports access to your data and annotations via the `zenml annotator ...` CLI command. -You can access information about the datasets you're using with the `zenml -annotator dataset list`. To work on annotation for a particular dataset, you can -run `zenml annotator dataset annotate `. This is -the equivalent of running `prodigy ` in the terminal. For +You can access information about the datasets you're using with the `zenml annotator dataset list`. To work on annotation for a particular dataset, you can +run `zenml annotator dataset annotate `. This is the equivalent of running `prodigy ` in the terminal. For example, you might run: ```shell zenml annotator dataset annotate your_dataset --command="textcat.manual news_topics ./news_headlines.jsonl --label Technology,Politics,Economy,Entertainment" ``` -This would launch the Prodigy interface for [the `textcat.manual` recipe](https://prodi.gy/docs/recipes#textcat-manual) with the -`news_topics` dataset and the labels `Technology`, `Politics`, `Economy`, and +This would launch the Prodigy interface for [the `textcat.manual` recipe](https://prodi.gy/docs/recipes#textcat-manual) with the `news_topics` dataset and the labels `Technology`, `Politics`, `Economy`, and `Entertainment`. The data would be loaded from the `news_headlines.jsonl` file. -A common workflow for Prodigy is to annotate data as you would usually do, and -then use the connection into ZenML to import those annotations within a step in +A common workflow for Prodigy is to annotate data as you would usually do, and then use the connection into ZenML to import those annotations within a step in your pipeline (if running locally). For example, within a ZenML step: ```python @@ -113,21 +97,16 @@ def import_annotations() -> List[Dict[str, Any]: return annotations ``` -If you're running in a cloud environment, you can manually export the -annotations, store them somewhere in a cloud environment and then reference or -use those within ZenML. The precise way you do this will be very case-dependent, -however, so it's difficult to provide a one-size-fits-all solution. +If you're running in a cloud environment, you can manually export the annotations, store them somewhere in a cloud environment and then reference or +use those within ZenML. The precise way you do this will be very case-dependent, however, so it's difficult to provide a one-size-fits-all solution. #### Prodigy Annotator Stack Component -Our Prodigy annotator component inherits from the `BaseAnnotator` class. There -are some methods that are core methods that must be defined, like being able to -register or get a dataset. Most annotators handle things like the storage of -state and have their own custom features, so there are quite a few extra methods +Our Prodigy annotator component inherits from the `BaseAnnotator` class. There are some methods that are core methods that must be defined, like being able to +register or get a dataset. Most annotators handle things like the storage of state and have their own custom features, so there are quite a few extra methods specific to Prodigy. -The core Prodigy functionality that's currently enabled from within the -`annotator` stack component interface includes a way to register your datasets +The core Prodigy functionality that's currently enabled from within the `annotator` stack component interface includes a way to register your datasets and export any annotations for use in separate steps. diff --git a/docs/book/component-guide/feature-stores/README.md b/docs/book/component-guide/feature-stores/README.md index 622053730aa..8e1c6202602 100644 --- a/docs/book/component-guide/feature-stores/README.md +++ b/docs/book/component-guide/feature-stores/README.md @@ -5,21 +5,17 @@ description: Managing data in feature stores. # Feature Stores -Feature stores allow data teams to serve data via an offline store and an online low-latency store where data is kept in -sync between the two. It also offers a centralized registry where features (and feature schemas) are stored for use +Feature stores allow data teams to serve data via an offline store and an online low-latency store where data is kept in sync between the two. It also offers a centralized registry where features (and feature schemas) are stored for use within a team or wider organization. -As a data scientist working on training your model, your requirements for how you access your batch / 'offline' data -will almost certainly be different from how you access that data as part of a real-time or online inference setting. -Feast solves the problem of developing train-serve skew where those two -sources of data diverge from each other. +As a data scientist working on training your model, your requirements for how you access your batch / 'offline' data will almost certainly be different from how you access that data as part of a real-time or online inference setting. +Feast solves the problem of developing train-serve skew where those two sources of data diverge from each other. Feature stores are a relatively recent addition to commonly-used machine learning stacks. ### When to use it -The feature store is an optional stack component in the ZenML Stack. The feature store as a technology should be used to -store the features and inject them into the process on the server side. This includes +The feature store is an optional stack component in the ZenML Stack. The feature store as a technology should be used to store the features and inject them into the process on the server side. This includes * Productionalize new features * Reuse existing features across multiple pipelines and models @@ -28,8 +24,7 @@ store the features and inject them into the process on the server side. This inc ### List of available feature stores -For production use cases, some more flavors can be found in specific `integrations` modules. In terms of features -stores, ZenML features an integration of `feast`. +For production use cases, some more flavors can be found in specific `integrations` modules. In terms of features stores, ZenML features an integration of `feast`. | Feature Store | Flavor | Integration | Notes | |------------------------------------|----------|-------------|--------------------------------------------------------------------------| @@ -44,8 +39,7 @@ zenml feature-store flavor list ### How to use it -The available implementation of the feature store is built on top of the feast integration, which means that using a -feature store is no different from what's described on the [feast page: How to use it?](feast.md#how-do-you-use-it). +The available implementation of the feature store is built on top of the feast integration, which means that using a feature store is no different from what's described on the [feast page: How to use it?](feast.md#how-do-you-use-it).
ZenML Scarf
diff --git a/docs/book/component-guide/feature-stores/feast.md b/docs/book/component-guide/feature-stores/feast.md index 78124dae5df..656aa494ea0 100644 --- a/docs/book/component-guide/feature-stores/feast.md +++ b/docs/book/component-guide/feature-stores/feast.md @@ -13,10 +13,8 @@ There are two core functions that feature stores enable: * access to data from an offline / batch store for training. * access to online data at inference time. -Feast integration currently supports your choice of offline data sources for -your online feature serving. We encourage users to check out [Feast's documentation](https://docs.feast.dev/) -and [guides](https://docs.feast.dev/how-to-guides/) on how to set up your -offline and online data sources via the configuration `yaml` file. +Feast integration currently supports your choice of offline data sources for your online feature serving. We encourage users to check out [Feast's documentation](https://docs.feast.dev/) +and [guides](https://docs.feast.dev/how-to-guides/) on how to set up your offline and online data sources via the configuration `yaml` file. {% hint style="info" %} COMING SOON: While the ZenML integration has an interface to access online feature store data, it currently is not usable in production settings with deployed models. We will update the docs when we enable this functionality. diff --git a/docs/book/component-guide/model-deployers/README.md b/docs/book/component-guide/model-deployers/README.md index cd5150f0811..5e1e8bc7dc8 100644 --- a/docs/book/component-guide/model-deployers/README.md +++ b/docs/book/component-guide/model-deployers/README.md @@ -22,26 +22,20 @@ The Model Deployer abstraction focused exclusively on single-model serving, but While Model Deployer integrations remain available for backward compatibility, we strongly recommend migrating to Pipeline Deployments for new projects. {% endhint %} -Model Deployment is the process of making a machine learning model available to make predictions and decisions on -real-world data. Getting predictions from trained models can be done in different ways depending on the use case, a -batch prediction is used to generate predictions for a large amount of data at once, while a real-time prediction is -used to generate predictions for a single data point at a time. +Model Deployment is the process of making a machine learning model available to make predictions and decisions on real-world data. Getting predictions from trained models can be done in different ways depending on the use case, a +batch prediction is used to generate predictions for a large amount of data at once, while a real-time prediction is used to generate predictions for a single data point at a time. Model deployers are stack components responsible for serving models on a real-time or batch basis. -Online serving is the process of hosting and loading machine-learning models as part of a managed web service and -providing access to the models through an API endpoint like HTTP or GRPC. Once deployed, model inference can be -triggered at any time, and you can send inference requests to the model through the web service's API and receive fast, -low-latency responses. +Online serving is the process of hosting and loading machine-learning models as part of a managed web service and providing access to the models through an API endpoint like HTTP or GRPC. Once deployed, model inference can be +triggered at any time, and you can send inference requests to the model through the web service's API and receive fast, low-latency responses. -Batch inference or offline inference is the process of making a machine learning model make predictions on a batch of -observations. This is useful for generating predictions for a large amount of data at once. The predictions are usually +Batch inference or offline inference is the process of making a machine learning model make predictions on a batch of observations. This is useful for generating predictions for a large amount of data at once. The predictions are usually stored as files or in a database for end users or business applications. ### When to use it? -The model deployers are optional components in the ZenML stack. They are used to deploy machine learning models to a -target environment, either a development (local) or a production (Kubernetes or cloud) environment. The model deployers are mainly used to deploy models for real-time inference use cases. With the model deployers and other stack components, you can build pipelines that are continuously trained and deployed to production. +The model deployers are optional components in the ZenML stack. They are used to deploy machine learning models to a target environment, either a development (local) or a production (Kubernetes or cloud) environment. The model deployers are mainly used to deploy models for real-time inference use cases. With the model deployers and other stack components, you can build pipelines that are continuously trained and deployed to production. ### How model deployers slot into the stack @@ -51,8 +45,7 @@ Here is an architecture diagram that shows how model deployers fit into the over #### Model Deployers Flavors -ZenML comes with a `local` MLflow model deployer which is a simple model deployer that deploys models to a local MLflow -server. Additional model deployers that can be used to deploy models on production environments are provided by +ZenML comes with a `local` MLflow model deployer which is a simple model deployer that deploys models to a local MLflow server. Additional model deployers that can be used to deploy models on production environments are provided by integrations: | Model Deployer | Flavor | Integration | Notes | @@ -66,8 +59,7 @@ integrations: | [Custom Implementation](custom.md) | _custom_ | | Extend the Artifact Store abstraction and provide your own implementation | {% hint style="info" %} -Every model deployer may have different attributes that must be configured in order to interact with the model serving -tool, framework, or platform (e.g. hostnames, URLs, references to credentials, and other client-related configuration +Every model deployer may have different attributes that must be configured in order to interact with the model serving tool, framework, or platform (e.g. hostnames, URLs, references to credentials, and other client-related configuration parameters). The following example shows the configuration of the MLflow and Seldon Core model deployers: ```shell @@ -104,8 +96,7 @@ zenml model-deployer register seldon --flavor=seldon \ * Lifecycle Management: Provides mechanisms for comprehensive lifecycle management of model servers, including the ability to start, stop, and delete model servers, as well as to update existing servers with new model versions, thereby optimizing resource utilization and facilitating continuous delivery of model updates. Some core methods that can be used to interact with the remote model server include: - `deploy_model` - Deploys a model to the serving environment and returns a Service object that represents the deployed model server. - - `find_model_server` - Finds and returns a list of Service objects that - represent model servers that have been deployed to the serving environment, + - `find_model_server` - Finds and returns a list of Service objects that represent model servers that have been deployed to the serving environment, the `services` are stored in the DB and can be used as a reference to know what and where the model is deployed. - `stop_model_server` - Stops a model server that is currently running in the serving environment. - `start_model_server` - Starts a model server that has been stopped in the serving environment. @@ -152,8 +143,7 @@ ZenML uses the Service object to represent a model server that has been deployed #### How to Interact with a model deployer after deployment? -When a Model Deployer is part of the active ZenML Stack, it is also possible to interact with it from the CLI to list, -start, stop, or delete the model servers that is managed: +When a Model Deployer is part of the active ZenML Stack, it is also possible to interact with it from the CLI to list, start, stop, or delete the model servers that is managed: ``` $ zenml model-deployer models list @@ -197,8 +187,7 @@ $ zenml model-deployer models get-url 8cbe671b-9fce-4394-a051-68e001f92765 $ zenml model-deployer models delete 8cbe671b-9fce-4394-a051-68e001f92765 ``` -In Python, you can alternatively discover the prediction URL of a deployed model by inspecting the metadata of the step -that deployed the model: +In Python, you can alternatively discover the prediction URL of a deployed model by inspecting the metadata of the step that deployed the model: ```python from zenml.client import Client @@ -208,11 +197,9 @@ deployer_step = pipeline_run.steps[""] deployed_model_url = deployer_step.run_metadata["deployed_model_url"].value ``` -The ZenML integrations that provide Model Deployer stack components also include standard pipeline steps that can -directly be inserted into any pipeline to achieve a continuous model deployment workflow. These steps take care of all +The ZenML integrations that provide Model Deployer stack components also include standard pipeline steps that can directly be inserted into any pipeline to achieve a continuous model deployment workflow. These steps take care of all the aspects of continuously deploying models to an external server and saving the Service configuration into the -Artifact Store, where they can be loaded at a later time and re-create the initial conditions used to serve a particular -model. +Artifact Store, where they can be loaded at a later time and re-create the initial conditions used to serve a particular model.
ZenML Scarf
diff --git a/docs/book/component-guide/orchestrators/kubernetes.md b/docs/book/component-guide/orchestrators/kubernetes.md index 202849ae073..058f6bb4993 100644 --- a/docs/book/component-guide/orchestrators/kubernetes.md +++ b/docs/book/component-guide/orchestrators/kubernetes.md @@ -124,12 +124,9 @@ E.g., you can use these labels to manually delete all pods related to a specific kubectl delete pod -n zenml -l pipeline=kubernetes_example_pipeline ``` -ZenML sanitizes these label values before sending them to Kubernetes. Invalid -characters are replaced, leading/trailing punctuation is removed, and values are -truncated to fit Kubernetes label limits. In practice, a pipeline called -`My GPU Pipeline!!!` will not appear as that exact string in a label selector. -If a selector using the raw name does not match anything, inspect one of the -created pods first: +ZenML sanitizes these label values before sending them to Kubernetes. Invalid characters are replaced, leading/trailing punctuation is removed, and values are +truncated to fit Kubernetes label limits. In practice, a pipeline called `My GPU Pipeline!!!` will not appear as that exact string in a label selector. +If a selector using the raw name does not match anything, inspect one of the created pods first: ```shell kubectl get pods -n zenml --show-labels @@ -166,12 +163,9 @@ The following configuration options can be set either through the orchestrator c - **`pod_failure_retry_delay`** (default: 10): The delay (in seconds) between retries to create a step pod that fails to start. - **`pod_failure_backoff`** (default: 1.0): The backoff factor for pod failure retries and pod startup retries. - **`backoff_limit_margin`** (default 0): The value to add to the backoff limit in addition to the [step retries](../../how-to/steps-pipelines/advanced_features.md#automatic-step-retries). The retry configuration defined on -the step defines the maximum number of retries that the server will accept for a step. For this orchestrator, this controls how often the -job running the step will try to start the step pod. There are some circumstances however where the job will start the pod, but the pod -doesn't actually get to the point of running the step. That means the server will not receive the maximum amount of retry requests, -which in turn causes other inconsistencies like wrong step statuses. To mitigate this, this attribute allows to add a margin to the -backoff limit. This means that the job will retry the pod startup for the configured amount of times plus the margin, which increases -the chance of the server receiving the maximum amount of retry requests. +the step defines the maximum number of retries that the server will accept for a step. For this orchestrator, this controls how often the job running the step will try to start the step pod. There are some circumstances however where the job will start the pod, but the pod +doesn't actually get to the point of running the step. That means the server will not receive the maximum amount of retry requests, which in turn causes other inconsistencies like wrong step statuses. To mitigate this, this attribute allows to add a margin to the +backoff limit. This means that the job will retry the pod startup for the configured amount of times plus the margin, which increases the chance of the server receiving the maximum amount of retry requests. - **`fail_on_container_waiting_reasons`**: List of container waiting reasons that should cause the job to fail immediately. This should be set to a list of nonrecoverable reasons, which if found in any `pod.status.containerStatuses[*].state.waiting.reason` of a job pod, should cause the job to fail immediately. - **`job_monitoring_interval`** (default 3): The interval in seconds to monitor the job. Each interval is used to check for container issues and streaming logs for the job pods. @@ -185,13 +179,11 @@ the chance of the server receiving the maximum amount of retry requests. For production setups, use separate identities for: -1. the identity that starts the orchestrator job (typically via a linked Service - Connector), +1. the identity that starts the orchestrator job (typically via a linked Service Connector), 2. the service account used by the orchestrator pod, 3. the service account used by step pods. -Using one service account for all three works, but it is broader than -necessary. +Using one service account for all three works, but it is broader than necessary. **Starter identity (Service Connector / kubeconfig identity)** @@ -227,8 +219,7 @@ If `pass_zenml_token_as_secret=True`, also grant: **Step pod service account (`step_pod_service_account_name`)** -Step containers do not need Kubernetes API access for normal execution in this -orchestrator flow. Unless your step code explicitly calls the Kubernetes API, +Step containers do not need Kubernetes API access for normal execution in this orchestrator flow. Unless your step code explicitly calls the Kubernetes API, you can keep this account with no additional Kubernetes RBAC grants. ```python @@ -473,8 +464,7 @@ scheduled_pipeline = my_kubernetes_pipeline.with_options( scheduled_pipeline() ``` -{% hint style="info" %} -`starting_deadline_seconds` controls how late a CronJob can start after its scheduled time (missed-schedule window), while `active_deadline_seconds` limits how long a running job can execute (runtime timeout). These are independent settings that apply at different stages of the job lifecycle. +{% hint style="info" %} `starting_deadline_seconds` controls how late a CronJob can start after its scheduled time (missed-schedule window), while `active_deadline_seconds` limits how long a running job can execute (runtime timeout). These are independent settings that apply at different stages of the job lifecycle. {% endhint %} Cron expressions follow the standard format (`minute hour day-of-month month day-of-week`): diff --git a/docs/book/component-guide/orchestrators/sagemaker.md b/docs/book/component-guide/orchestrators/sagemaker.md index fa4d5026f08..0e6a55b5803 100644 --- a/docs/book/component-guide/orchestrators/sagemaker.md +++ b/docs/book/component-guide/orchestrators/sagemaker.md @@ -556,8 +556,7 @@ zenml orchestrator update sagemaker-orchestrator \ {% hint style="info" %} The IAM role that you are using on the client side can come from multiple sources depending on how you configured your orchestrator, such as explicit credentials, a service connector or an implicit authentication. -If you are using a service connector, keep in mind, this only works with authentication methods that involve IAM roles (IAM role, Implicit authentication). LINK -{% endhint %} +If you are using a service connector, keep in mind, this only works with authentication methods that involve IAM roles (IAM role, Implicit authentication). LINK {% endhint %} This is particularly useful when: diff --git a/docs/book/component-guide/service-connectors/connector-types/gcp-service-connector.md b/docs/book/component-guide/service-connectors/connector-types/gcp-service-connector.md index 6ecb8a80efd..4e1ef5fe68a 100644 --- a/docs/book/component-guide/service-connectors/connector-types/gcp-service-connector.md +++ b/docs/book/component-guide/service-connectors/connector-types/gcp-service-connector.md @@ -140,8 +140,7 @@ If set, the resource name must identify a GAR or GCR registry using one of the f * Google Artifact Registry name: `projects//locations//repositories/` * (legacy) GCR repository URI: `[https://][us.|eu.|asia.]gcr.io/[/]` -The connector can only be used to access GAR and GCR registries in the GCP\ -project that it is configured to use. +The connector can only be used to access GAR and GCR registries in the GCP\ project that it is configured to use. ## Authentication Methods @@ -1123,8 +1122,7 @@ Credentials saved to file: [/home/stefan/.config/gcloud/application_default_cred These credentials will be used by any library that requests Application Default Credentials (ADC). -Quota project "zenml-core" was added to ADC which can be used by Google client libraries for billing -and quota. Note that some services may still bill the project owning the resource. +Quota project "zenml-core" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource. ``` ```` {% endcode %} @@ -1504,8 +1502,7 @@ Credentials saved to file: [/home/stefan/.config/gcloud/application_default_cred These credentials will be used by any library that requests Application Default Credentials (ADC). -Quota project "zenml-core" was added to ADC which can be used by Google client libraries for billing -and quota. Note that some services may still bill the project owning the resource. +Quota project "zenml-core" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource. ``` ```` {% endcode %} diff --git a/docs/book/component-guide/step-operators/azureml.md b/docs/book/component-guide/step-operators/azureml.md index 09a13552fd7..2ac28e66624 100644 --- a/docs/book/component-guide/step-operators/azureml.md +++ b/docs/book/component-guide/step-operators/azureml.md @@ -16,10 +16,8 @@ You should use the AzureML step operator if: ### How to deploy it {% hint style="info" %} -Would you like to skip ahead and deploy a full ZenML cloud stack already, -including an AzureML step operator? Check out the [in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), -the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), -or [the ZenML Azure Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) +Would you like to skip ahead and deploy a full ZenML cloud stack already, including an AzureML step operator? Check out the [in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), +the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), or [the ZenML Azure Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register this stack component. {% endhint %} @@ -105,8 +103,7 @@ ZenML will build a Docker image called `/zenml: 1`) | **Multi-node instance types** | Job creation fails with a `400 Bad Request` | -This is because a regular `@step` runs the ZenML entrypoint, so its container must contain `zenml`, -your code and dependencies — that is a *custom image*. A [`CommandStep`](https://docs.zenml.io/how-to/steps-pipelines/command_steps), -by contrast, is an opaque command that can run on a **stock public image** (with `skip_build=True`), -which any Baseten organization can pull — so the single-node `CommandStep` path works out of the -box. If you plan to run regular `@step`s or multi-node jobs, ask Baseten to enable the -corresponding entitlement first. These are Baseten account gates, not ZenML limitations — the +This is because a regular `@step` runs the ZenML entrypoint, so its container must contain `zenml`, your code and dependencies — that is a *custom image*. A [`CommandStep`](https://docs.zenml.io/how-to/steps-pipelines/command_steps), +by contrast, is an opaque command that can run on a **stock public image** (with `skip_build=True`), which any Baseten organization can pull — so the single-node `CommandStep` path works out of the +box. If you plan to run regular `@step`s or multi-node jobs, ask Baseten to enable the corresponding entitlement first. These are Baseten account gates, not ZenML limitations — the operator builds, submits, polls, cancels and records metadata for every mode regardless. ## How to use it @@ -59,10 +50,8 @@ To use the Baseten step operator, you need: zenml integration install baseten ``` -* a [remote artifact store](https://docs.zenml.io/stacks/artifact-stores/) as part of your - stack (steps run remotely and write artifacts over the network), -* a [remote container registry](https://docs.zenml.io/stacks/container-registries/) and an - [image builder](https://docs.zenml.io/stacks/image-builders/) as part of your stack, so the +* a [remote artifact store](https://docs.zenml.io/stacks/artifact-stores/) as part of your stack (steps run remotely and write artifacts over the network), +* a [remote container registry](https://docs.zenml.io/stacks/container-registries/) and an [image builder](https://docs.zenml.io/stacks/image-builders/) as part of your stack, so the step image can be built and pulled by Baseten. Register the step operator and add it to your stack: @@ -81,8 +70,7 @@ zenml stack register baseten_stack \ --set ``` -If your step image lives in a private registry, store the registry credentials -(`username:password`) as a Baseten secret and reference it on the step operator with +If your step image lives in a private registry, store the registry credentials (`username:password`) as a Baseten secret and reference it on the step operator with `--registry_auth_secret=`. {% hint style="info" %} @@ -115,15 +103,12 @@ def my_pipeline() -> None: train() ``` -The `accelerator` type (`H100` or `H200`) is a step operator setting; the number of GPUs per -node comes from `ResourceSettings.gpu_count`. +The `accelerator` type (`H100` or `H200`) is a step operator setting; the number of GPUs per node comes from `ResourceSettings.gpu_count`. ### Multi-node distributed training -Multi-node runs the same container on every node, so a regular step would duplicate its -artifacts, outputs and logs. Multi-node is therefore only allowed for a -[`CommandStep`](https://docs.zenml.io/how-to/steps-pipelines/command_steps), which ZenML treats -as an opaque command and never runs its machinery inside. Set `node_count > 1` and let your +Multi-node runs the same container on every node, so a regular step would duplicate its artifacts, outputs and logs. Multi-node is therefore only allowed for a +[`CommandStep`](https://docs.zenml.io/how-to/steps-pipelines/command_steps), which ZenML treats as an opaque command and never runs its machinery inside. Set `node_count > 1` and let your command own the distributed launch. Baseten injects these environment variables on every node, which you wire into `torchrun`: @@ -163,13 +148,11 @@ def training_pipeline() -> None: train() ``` -The image must already contain your training code and dependencies. A regular step submitted -with `node_count > 1` is rejected with a clear error. +The image must already contain your training code and dependencies. A regular step submitted with `node_count > 1` is rejected with a clear error. ### Passing secrets -Sensitive environment variables are never inlined into the job config. Store your own secrets -(API tokens, credentials) as Baseten secrets and map them with the `secrets` setting — the value +Sensitive environment variables are never inlined into the job config. Store your own secrets (API tokens, credentials) as Baseten secrets and map them with the `secrets` setting — the value is referenced rather than inlined: ```python @@ -178,17 +161,13 @@ BasetenStepOperatorSettings( ) ``` -The ZenML store API token (which regular steps need to call back to the server) is handled -automatically: the operator upserts it into a managed Baseten secret named -`zenml-store-api-token-` on each run and references that, so the token never lands -in the inlined job config. You can override this by mapping the token name explicitly in -`secrets`. Command steps never talk to the ZenML server, so the token is dropped for them -entirely. +The ZenML store API token (which regular steps need to call back to the server) is handled automatically: the operator upserts it into a managed Baseten secret named +`zenml-store-api-token-` on each run and references that, so the token never lands in the inlined job config. You can override this by mapping the token name explicitly in +`secrets`. Command steps never talk to the ZenML server, so the token is dropped for them entirely. ### Caching and checkpointing -Baseten can [persist a training cache](https://docs.baseten.co/training/loading) so datasets and -model weights downloaded by a job survive across jobs (avoiding re-downloads), and it can manage +Baseten can [persist a training cache](https://docs.baseten.co/training/loading) so datasets and model weights downloaded by a job survive across jobs (avoiding re-downloads), and it can manage checkpoint storage. Both are **disabled by default** and opt-in through settings: ```python @@ -201,16 +180,12 @@ BasetenStepOperatorSettings( ) ``` -When `enable_cache` is on, write your downloads (e.g. `HF_HOME`, dataset staging) into the cache so -subsequent runs reuse them. `cache_enable_legacy_hf_mount` additionally mounts the legacy Hugging -Face cache location for code that downloads to the default HF path, and `cache_require_affinity` -(default `True`) controls whether the job must land on nodes that already hold the cache — set it -`False` to let the same project run across different GPU types. When `enable_checkpointing` is on, -write checkpoints to the Baseten checkpoint directory exposed as +When `enable_cache` is on, write your downloads (e.g. `HF_HOME`, dataset staging) into the cache so subsequent runs reuse them. `cache_enable_legacy_hf_mount` additionally mounts the legacy Hugging +Face cache location for code that downloads to the default HF path, and `cache_require_affinity` (default `True`) controls whether the job must land on nodes that already hold the cache — set it +`False` to let the same project run across different GPU types. When `enable_checkpointing` is on, write checkpoints to the Baseten checkpoint directory exposed as `$BT_CHECKPOINT_DIR`. -For more information and a full list of configurable attributes, check out the -[SDK docs](https://sdkdocs.zenml.io/latest/). +For more information and a full list of configurable attributes, check out the [SDK docs](https://sdkdocs.zenml.io/latest/).
ZenML Scarf
diff --git a/docs/book/component-guide/step-operators/modal.md b/docs/book/component-guide/step-operators/modal.md index d1e6e196e8a..bd6dd7a50df 100644 --- a/docs/book/component-guide/step-operators/modal.md +++ b/docs/book/component-guide/step-operators/modal.md @@ -31,13 +31,10 @@ To use the Modal step operator, we need: zenml integration install modal ``` * Docker installed and running. -* A cloud artifact store as part of your stack. This is needed so that both your - orchestration environment and Modal can read and write step artifacts. Any +* A cloud artifact store as part of your stack. This is needed so that both your orchestration environment and Modal can read and write step artifacts. Any cloud artifact store supported by ZenML will work with Modal. -* A cloud container registry as part of your stack. Any cloud container - registry supported by ZenML will work with Modal. -* An Image Builder in your stack. ZenML uses it to build the Docker image that - runs on Modal. +* A cloud container registry as part of your stack. Any cloud container registry supported by ZenML will work with Modal. +* An Image Builder in your stack. ZenML uses it to build the Docker image that runs on Modal. The Modal step operator can use Modal authentication settings from the stack component configuration. If `token_id` and `token_secret` are configured on the step operator, ZenML creates an explicit Modal SDK client from those credentials and passes that client to Modal SDK calls. If these fields are not configured, ZenML passes no explicit client and Modal uses its normal authentication behavior, such as existing environment variables or `~/.modal.toml`. @@ -99,8 +96,7 @@ You can remove `prevent_build_reuse=True` again after ZenML has built and pushed #### Additional configuration -You can specify the hardware requirements for each step using the -`ResourceSettings` class as described in our documentation on [resource settings](https://docs.zenml.io/user-guides/tutorial/distributed-training): +You can specify the hardware requirements for each step using the `ResourceSettings` class as described in our documentation on [resource settings](https://docs.zenml.io/user-guides/tutorial/distributed-training): ```python from zenml import step @@ -146,15 +142,11 @@ Note that `cpu_count` specifies a soft minimum limit - Modal will guarantee at l This will run `my_modal_step` on a Modal instance with 1 A100 GPU, 2 CPUs, and 32GB of CPU memory. -Check out the [Modal docs](https://modal.com/docs/guide/gpu) for the -full list of supported GPU types and the [SDK +Check out the [Modal docs](https://modal.com/docs/guide/gpu) for the full list of supported GPU types and the [SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-modal.html) for more details on the available settings. The settings allow you to specify the Modal environment, region, and cloud provider. `modal_environment` selects the Modal environment used for the app lookup; ZenML passes it as `environment_name` to `modal.App.lookup(...)`. The sandbox then belongs to that app, while ZenML step runtime environment variables are still passed separately to the sandbox runtime with `env=...`. Region and cloud provider settings are only available for Modal Enterprise and Team plan customers. -Certain combinations of settings are not available. It is suggested to -err on the side of looser settings rather than more restrictive ones to avoid -pipeline execution failures. In the case of failures, however, Modal provides -detailed error messages that can help identify what is incompatible. See more in -the [Modal docs on region selection](https://modal.com/docs/guide/region-selection) for more -details. +Certain combinations of settings are not available. It is suggested to err on the side of looser settings rather than more restrictive ones to avoid +pipeline execution failures. In the case of failures, however, Modal provides detailed error messages that can help identify what is incompatible. See more in +the [Modal docs on region selection](https://modal.com/docs/guide/region-selection) for more details. diff --git a/docs/book/component-guide/step-operators/runai.md b/docs/book/component-guide/step-operators/runai.md index dfd2c6be696..560c0ef578a 100644 --- a/docs/book/component-guide/step-operators/runai.md +++ b/docs/book/component-guide/step-operators/runai.md @@ -225,10 +225,8 @@ settings = RunAIStepOperatorSettings( | `parallelism` | int | None | Run:AI training workload parallelism | | `completions` | int | None | Run:AI training workload completions | -A useful way to think about the timeout settings is: `pending_timeout` covers -the queuing phase before Run:AI can start the workload, while -`workload_timeout` covers the running phase after the workload has started. If a -workload sits in `Pending` because no quota is available, `pending_timeout` is +A useful way to think about the timeout settings is: `pending_timeout` covers the queuing phase before Run:AI can start the workload, while +`workload_timeout` covers the running phase after the workload has started. If a workload sits in `Pending` because no quota is available, `pending_timeout` is the timer that lets ZenML stop waiting instead of hanging indefinitely. Environment variables are configured through the standard ZenML `environment` settings on steps or pipelines; the Run:AI step operator does not introduce an additional environment-specific setting. diff --git a/docs/book/getting-started/deploying-zenml/deploy-with-docker.md b/docs/book/getting-started/deploying-zenml/deploy-with-docker.md index fea009e14b7..0e4fa06dc8e 100644 --- a/docs/book/getting-started/deploying-zenml/deploy-with-docker.md +++ b/docs/book/getting-started/deploying-zenml/deploy-with-docker.md @@ -482,8 +482,7 @@ The following additional rules are applied concerning the creation and lifetime * the persistent backup file or database is NOT cleaned up after a failed migration. This allows the user to manually inspect and/or apply the backup if the automatic recovery fails. {% hint style="warning" %} -When running in production where database sizes are large, you should use the `mydumper` backup strategy or write -your own custom backup engine. The other backup strategies are not recommended because they are inefficient and will take a long time and consume a lot of resources to handle large databases. +When running in production where database sizes are large, you should use the `mydumper` backup strategy or write your own custom backup engine. The other backup strategies are not recommended because they are inefficient and will take a long time and consume a lot of resources to handle large databases. {% endhint %} The following example shows how to deploy the ZenML server to use a mounted host directory to persist the database backup file during a database migration: diff --git a/docs/book/getting-started/deploying-zenml/deploy-with-helm.md b/docs/book/getting-started/deploying-zenml/deploy-with-helm.md index 4cc6085f1d3..cc6218b48da 100644 --- a/docs/book/getting-started/deploying-zenml/deploy-with-helm.md +++ b/docs/book/getting-started/deploying-zenml/deploy-with-helm.md @@ -702,8 +702,7 @@ Several database backup strategies are supported, depending on where and how the > **NOTE:** You should also set the `podSecurityContext.fsGroup` option if you are using a persistent volume to store the dump file. {% hint style="warning" %} -When running in production where database sizes are large, you should use the `mydumper` backup strategy or write -your own custom backup engine. The other backup strategies are not recommended because they are inefficient and will take a long time and consume a lot of resources to handle large databases. +When running in production where database sizes are large, you should use the `mydumper` backup strategy or write your own custom backup engine. The other backup strategies are not recommended because they are inefficient and will take a long time and consume a lot of resources to handle large databases. {% endhint %} The following additional rules are applied concerning the creation and lifetime of the backup: diff --git a/docs/book/getting-started/deploying-zenml/live-event-streaming.md b/docs/book/getting-started/deploying-zenml/live-event-streaming.md index 56f5f209226..d3c71d3df74 100644 --- a/docs/book/getting-started/deploying-zenml/live-event-streaming.md +++ b/docs/book/getting-started/deploying-zenml/live-event-streaming.md @@ -4,42 +4,30 @@ description: Enable live event streaming on the ZenML server and consume the HTT # Live event streaming -ZenML can send events published from inside a pipeline run to any HTTP -client subscribed to that run. Use it for LLM token streaming, progress -updates, live dashboards, and similar cases where you need to surface -intermediate output from a running step. - -This page covers operating streaming on the server: how to turn it on, -which broker to pick, how to consume the SSE feed, and the wire contract -clients code against. For the producer-side Python API (calling -`zenml.streaming.publish()` from a step), see +ZenML can send events published from inside a pipeline run to any HTTP client subscribed to that run. Use it for LLM token streaming, progress +updates, live dashboards, and similar cases where you need to surface intermediate output from a running step. + +This page covers operating streaming on the server: how to turn it on, which broker to pick, how to consume the SSE feed, and the wire contract +clients code against. For the producer-side Python API (calling `zenml.streaming.publish()` from a step), see [Streaming Events](../../how-to/steps-pipelines/streaming_events.md). {% hint style="warning" %} -Streaming is best-effort, not persistent storage. Events are size-capped, -can be dropped under load, and disappear when the broker's retention -window elapses. **Once an event is lost it is gone — there is no -secondary store, no replay endpoint, and no fallback.** If you need to +Streaming is best-effort, not persistent storage. Events are size-capped, can be dropped under load, and disappear when the broker's retention +window elapses. **Once an event is lost it is gone — there is no secondary store, no replay endpoint, and no fallback.** If you need to keep something, write it as run metadata or an artifact from the step. {% endhint %} ## Enable streaming -Streaming is off by default. The single setting that enables it is -`stream_broker_implementation_source` on the server config (or -`streaming.streamBrokerImplementationSource` on the Helm chart). Until -that field is set, the streaming endpoints return `501 Not Implemented`, -producer-side `publish()` calls are dropped without sending HTTP, and -the server does not open a broker connection. +Streaming is off by default. The single setting that enables it is `stream_broker_implementation_source` on the server config (or +`streaming.streamBrokerImplementationSource` on the Helm chart). Until that field is set, the streaming endpoints return `501 Not Implemented`, +producer-side `publish()` calls are dropped without sending HTTP, and the server does not open a broker connection. ### Pick a broker -The Redis Streams broker -(`zenml.zen_server.streaming.brokers.redis_streams.RedisStreamsBroker`) -ships with ZenML. It requires Redis 5+ and the `redis` Python extra -(`pip install 'zenml[server-streaming]'`). It namespaces stream keys by -deployment ID so multiple ZenML servers can share a Redis cluster -without colliding. +The Redis Streams broker (`zenml.zen_server.streaming.brokers.redis_streams.RedisStreamsBroker`) +ships with ZenML. It requires Redis 5+ and the `redis` Python extra (`pip install 'zenml[server-streaming]'`). It namespaces stream keys by +deployment ID so multiple ZenML servers can share a Redis cluster without colliding. ### Configure with Helm @@ -52,10 +40,8 @@ server: ``` The chart installs an SSE-only Gateway API `HTTPRoute` rule that disables -Envoy's default 15-second request timeout for clients that send -`Accept: text/event-stream` against the `/api/v1/runs/` tree. Browsers' -`EventSource` and the ZenML server's own emitted frames match this -condition. Custom clients that send a quality-list `Accept` header fall +Envoy's default 15-second request timeout for clients that send `Accept: text/event-stream` against the `/api/v1/runs/` tree. Browsers' +`EventSource` and the ZenML server's own emitted frames match this condition. Custom clients that send a quality-list `Accept` header fall through to the default rule and are cut off at 15 seconds. ### Configure with environment variables @@ -67,10 +53,8 @@ ZENML_SERVER_STREAM_BROKER_IMPLEMENTATION_SOURCE=zenml.zen_server.streaming.brok ZENML_REDIS_BROKER_URL=redis://... ``` -Behind your own ingress, disable request timeouts and any response -buffering for the SSE path. The server emits `X-Accel-Buffering: no` and -`Cache-Control: no-cache, no-store, no-transform` to cover common -intermediaries, but you may need to set the same on your proxy. +Behind your own ingress, disable request timeouts and any response buffering for the SSE path. The server emits `X-Accel-Buffering: no` and +`Cache-Control: no-cache, no-store, no-transform` to cover common intermediaries, but you may need to set the same on your proxy. ### Server config reference @@ -83,10 +67,8 @@ intermediaries, but you may need to set the same on your proxy. ### Redis settings -Connection settings are read from the shared `ZENML_REDIS_` prefix, so -the same Redis instance can be used by the streaming broker and by other -ZenML components that talk to Redis. Settings specific to the streaming -broker are read from `ZENML_REDIS_STREAMS_BROKER_` and override the +Connection settings are read from the shared `ZENML_REDIS_` prefix, so the same Redis instance can be used by the streaming broker and by other +ZenML components that talk to Redis. Settings specific to the streaming broker are read from `ZENML_REDIS_STREAMS_BROKER_` and override the shared values when set. | Variable | Default | Notes | @@ -97,15 +79,12 @@ shared values when set. | `ZENML_REDIS_STREAMS_BROKER_MAX_STREAM_LENGTH` | `10000` | Maximum entries retained per run (`XADD MAXLEN ~`). | | `ZENML_REDIS_STREAMS_BROKER_STREAM_TTL_SECONDS` | `3600` | TTL on each run's stream, refreshed on every publish. | -At startup, the server runs a single connectivity check against the -broker. If the configured Redis URL is wrong or the host is unreachable, -the server fails to boot and reports the error, instead of returning -`503` on every later request. +At startup, the server runs a single connectivity check against the broker. If the configured Redis URL is wrong or the host is unreachable, +the server fails to boot and reports the error, instead of returning `503` on every later request. ## Consume the stream -Streams are exposed as -[Server-Sent Events (SSE)](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) +Streams are exposed as [Server-Sent Events (SSE)](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) on: ``` @@ -114,10 +93,8 @@ Accept: text/event-stream Authorization: Bearer ``` -Consuming requires `READ` permission on the run (the same permission as -viewing it in the dashboard). Publishing — covered on the -[client SDK page](../../how-to/steps-pipelines/streaming_events.md) — -requires `UPDATE`. +Consuming requires `READ` permission on the run (the same permission as viewing it in the dashboard). Publishing — covered on the +[client SDK page](../../how-to/steps-pipelines/streaming_events.md) — requires `UPDATE`. ### Browser @@ -130,8 +107,7 @@ es.addEventListener("event", (e) => console.log(JSON.parse(e.data))); es.addEventListener("end", () => es.close()); ``` -`EventSource` automatically reconnects with the standard `Last-Event-ID` -header, so transient drops resume after the last received event (see +`EventSource` automatically reconnects with the standard `Last-Event-ID` header, so transient drops resume after the last received event (see [Resuming after a disconnect](#resuming-after-a-disconnect)). ### Command line @@ -142,8 +118,7 @@ curl -N -H "Accept: text/event-stream" \ "$ZENML_URL/api/v1/runs/$RUN_ID/events/stream" ``` -`-N` disables curl's output buffering so frames arrive as the server -writes them. +`-N` disables curl's output buffering so frames arrive as the server writes them. ## SSE wire format @@ -166,20 +141,15 @@ Reserved event names: | `error` | A transient server-side error. The client should reconnect with `Last-Event-ID`. | | `cursor` | A frame the server emits for filtered-out events and for forward-compatible unknown frame types. Carries an `id:` so `Last-Event-ID` advances. `data` is `{}` for filtered events and `{"unknown_type": ""}` for frames the server didn't recognize (useful for spotting producer-vs-server version mismatches). Clients can ignore both. | -Heartbeats arrive as comment frames (`: ping\n\n`) every -`streaming_heartbeat_seconds` (default 30 s) and require no client -handling. Comments do not dispatch events (they will not trigger any -`addEventListener` callback), which is why filtered or unknown frames +Heartbeats arrive as comment frames (`: ping\n\n`) every `streaming_heartbeat_seconds` (default 30 s) and require no client +handling. Comments do not dispatch events (they will not trigger any `addEventListener` callback), which is why filtered or unknown frames use `event: cursor` instead. ### Filtering -The SSE endpoint accepts three multi-value query parameters that -restrict which events are delivered. Each parameter accepts repeated -values; within a parameter the values are ORed, and the parameters -combine with AND. Filtered-out events still advance the server cursor -via `cursor` frames — clients can reconnect with `Last-Event-ID` and -will not see them replayed. +The SSE endpoint accepts three multi-value query parameters that restrict which events are delivered. Each parameter accepts repeated +values; within a parameter the values are ORed, and the parameters combine with AND. Filtered-out events still advance the server cursor +via `cursor` frames — clients can reconnect with `Last-Event-ID` and will not see them replayed. | Parameter | Matches | Example | |---|---|---| @@ -197,8 +167,7 @@ returns only `token`-kind events from the `summarize` step. ### Resuming after a disconnect -The server honors the standard SSE `Last-Event-ID` request header on -reconnect. Browsers' `EventSource` sends it automatically. Other clients +The server honors the standard SSE `Last-Event-ID` request header on reconnect. Browsers' `EventSource` sends it automatically. Other clients should track the last `id:` they received and send it back to resume: ``` @@ -206,28 +175,21 @@ GET /api/v1/runs/{run}/events/stream Last-Event-ID: ``` -Clients that cannot set request headers (some embedded environments) can -use the `?since=` query parameter as an equivalent — both specify +Clients that cannot set request headers (some embedded environments) can use the `?since=` query parameter as an equivalent — both specify the starting cursor. If both are sent, the header wins. -If the cursor is older than the broker's retention window, the missing -events are not redelivered and the server does not signal that loss +If the cursor is older than the broker's retention window, the missing events are not redelivered and the server does not signal that loss happened. The next read returns whatever is still retained. -Subscribers can also attach to a run that has already terminated. The -server replays the broker's retained event history (up to the retention -TTL) and then closes with an `end` event. Once the TTL elapses the -history is gone and the subscribe returns just `end`. +Subscribers can also attach to a run that has already terminated. The server replays the broker's retained event history (up to the retention +TTL) and then closes with an `end` event. Once the TTL elapses the history is gone and the subscribe returns just `end`. -**Lost events are unrecoverable.** Streaming is best-effort: events -never leave the broker for any durable store, and ZenML keeps no +**Lost events are unrecoverable.** Streaming is best-effort: events never leave the broker for any durable store, and ZenML keeps no secondary copy. Artifacts and run metadata persist the run's *outcomes*, not the intermediate stream. Plan accordingly: -- If you need replay, write the relevant state as an artifact or - metadata entry from the step. -- If your consumer maintains UI state derived from the stream - (running aggregates, scrollback), design it to tolerate gaps — +- If you need replay, write the relevant state as an artifact or metadata entry from the step. +- If your consumer maintains UI state derived from the stream (running aggregates, scrollback), design it to tolerate gaps — drop accumulated state and re-derive from new events going forward, rather than expecting to "fetch what you missed". @@ -245,43 +207,30 @@ secondary copy. Artifacts and run metadata persist the run's ## Limits - Per-event payload is limited to **64 KiB** on the wire envelope. -- The producer-side queue holds up to **4 096 events per process**. When - full, the oldest queued event is dropped to make room. +- The producer-side queue holds up to **4 096 events per process**. When full, the oldest queued event is dropped to make room. - The broker stream is limited per run (default 10 000 entries). Subscribers that fall too far behind will silently miss the trimmed events — there is no wire-level signal for retention loss, and trimmed events are not stored anywhere recoverable. -- The per-run subscriber limit is `streaming_max_subscribers_per_stream` - (default 100). The 101st connection receives `503 Service Unavailable` +- The per-run subscriber limit is `streaming_max_subscribers_per_stream` (default 100). The 101st connection receives `503 Service Unavailable` with `Retry-After: 5`. ## Troubleshooting -**SSE connections drop after 15 seconds behind an ingress.** Your proxy -is enforcing a request timeout. The bundled Helm chart configures the -Gateway API `HTTPRoute` to disable it for SSE; if you run a custom -ingress, do the same for `/api/v1/runs/.../events/stream` (or any path +**SSE connections drop after 15 seconds behind an ingress.** Your proxy is enforcing a request timeout. The bundled Helm chart configures the +Gateway API `HTTPRoute` to disable it for SSE; if you run a custom ingress, do the same for `/api/v1/runs/.../events/stream` (or any path where the request carries `Accept: text/event-stream`). -**Subscribers report missing events on reconnect.** The subscriber is -falling behind the broker's retention window. The missed events are -gone — they are not stored anywhere durable. Either reduce the producer -rate, raise `ZENML_REDIS_STREAMS_BROKER_MAX_STREAM_LENGTH`, or have the -subscriber drop accumulated stream-derived state on every `gap` and -re-derive from new events going forward. - -**No events arrive.** Confirm streaming is enabled (the streaming -endpoints return something other than `501`), check the consumer has -`READ` on the run, and verify the producer is calling -`zenml.streaming.publish()` from inside a step or pipeline context. Calls +**Subscribers report missing events on reconnect.** The subscriber is falling behind the broker's retention window. The missed events are +gone — they are not stored anywhere durable. Either reduce the producer rate, raise `ZENML_REDIS_STREAMS_BROKER_MAX_STREAM_LENGTH`, or have the +subscriber drop accumulated stream-derived state on every `gap` and re-derive from new events going forward. + +**No events arrive.** Confirm streaming is enabled (the streaming endpoints return something other than `501`), check the consumer has +`READ` on the run, and verify the producer is calling `zenml.streaming.publish()` from inside a step or pipeline context. Calls made outside such a context are dropped. -**`501 Not Implemented` on the streaming endpoints.** -`stream_broker_implementation_source` is unset. Once it is configured, -both the publish endpoint and the SSE endpoint become available. They -return `501` together when streaming is disabled. +**`501 Not Implemented` on the streaming endpoints.** `stream_broker_implementation_source` is unset. Once it is configured, +both the publish endpoint and the SSE endpoint become available. They return `501` together when streaming is disabled. -**Server boot fails with "Stream broker startup probe failed".** The -configured broker cannot reach its backing store. For Redis, check -`ZENML_REDIS_BROKER_URL`, TLS settings, and network reachability from -the server pod. +**Server boot fails with "Stream broker startup probe failed".** The configured broker cannot reach its backing store. For Redis, check +`ZENML_REDIS_BROKER_URL`, TLS settings, and network reachability from the server pod. diff --git a/docs/book/getting-started/hello-world.md b/docs/book/getting-started/hello-world.md index 7e0754dd20c..a1cf67ce065 100644 --- a/docs/book/getting-started/hello-world.md +++ b/docs/book/getting-started/hello-world.md @@ -163,8 +163,7 @@ python run.py # Automatically runs on cloud infrastructure ZenML handles packaging code, building containers, orchestrating execution, and tracking artifacts automatically across all cloud providers. -
Pipeline Run in ZenML Dashboard

Your pipeline in the ZenML Pro Dashboard

-{% endstep %} +
Pipeline Run in ZenML Dashboard

Your pipeline in the ZenML Pro Dashboard

{% endstep %} {% step %} #### What's next? diff --git a/docs/book/getting-started/system-architectures.md b/docs/book/getting-started/system-architectures.md index de78e9f7531..c45379b180b 100644 --- a/docs/book/getting-started/system-architectures.md +++ b/docs/book/getting-started/system-architectures.md @@ -97,8 +97,7 @@ This architecture strikes a balance between convenience and control, making it a ![ZenML Pro self-hosted deployment](../.gitbook/assets/cloud_architecture_scenario_2.avif) -In the case of self-hosting ZenML Pro, all services, data, and secrets are deployed on the customer\ -cloud. This is meant for customers who require completely air-gapped deployments, for the tightest security standards. [Reach out to us](mailto:cloud@zenml.io) if you want to set this up. +In the case of self-hosting ZenML Pro, all services, data, and secrets are deployed on the customer\ cloud. This is meant for customers who require completely air-gapped deployments, for the tightest security standards. [Reach out to us](mailto:cloud@zenml.io) if you want to set this up.
diff --git a/docs/book/getting-started/zenml-pro/deploy-workspace-event-triggers-and-schedules.md b/docs/book/getting-started/zenml-pro/deploy-workspace-event-triggers-and-schedules.md index 584416b93f5..78c7cb2fdd1 100644 --- a/docs/book/getting-started/zenml-pro/deploy-workspace-event-triggers-and-schedules.md +++ b/docs/book/getting-started/zenml-pro/deploy-workspace-event-triggers-and-schedules.md @@ -17,69 +17,54 @@ layout: # Enable Event Triggers and Schedules for the Workspace Server -ZenML Pro [schedule triggers](triggers.md#schedule-triggers) run pipelines on a -cron or interval. [Platform event triggers](triggers.md#platform-event-triggers) -run pipelines when lifecycle events occur in the ZenML platform (for example -after another pipeline completes). On self-hosted workspaces, both are part +ZenML Pro [schedule triggers](triggers.md#schedule-triggers) run pipelines on a cron or interval. [Platform event triggers](triggers.md#platform-event-triggers) +run pipelines when lifecycle events occur in the ZenML platform (for example after another pipeline completes). On self-hosted workspaces, both are part of the same opt-in capability and use the same background infrastructure: two additional microservices—the **scheduler** and the **executor**—plus a **Redis** broker that connects them. {% hint style="warning" %} -**Commercial add-on:** Event triggers and schedules (schedule triggers and -platform event triggers) are not included in the base ZenML Pro plan. Your -organization must purchase and enable them explicitly. See the -[pricing page](https://www.zenml.io/pricing) for plans and contact ZenML if you +**Commercial add-on:** Event triggers and schedules (schedule triggers and platform event triggers) are not included in the base ZenML Pro plan. Your +organization must purchase and enable them explicitly. See the [pricing page](https://www.zenml.io/pricing) for plans and contact ZenML if you need entitlements enabled for your license. {% endhint %} {% hint style="warning" %} Deploy these microservices only for workspace servers installed with the ZenML -Helm chart on Kubernetes. Other platforms (for example AWS ECS) are not covered -here. +Helm chart on Kubernetes. Other platforms (for example AWS ECS) are not covered here. {% endhint %} {% hint style="info" %} -**Prerequisite:** Event triggers and schedules dispatch work against [pipeline -snapshots](snapshots.md). You must enable the workload manager (snapshot -support) on the workspace server before or together with the scheduler -and executor. Follow [Enable Snapshot Support](deploy-workspace-snapshots.md) +**Prerequisite:** Event triggers and schedules dispatch work against [pipeline snapshots](snapshots.md). You must enable the workload manager (snapshot +support) on the workspace server before or together with the scheduler and executor. Follow [Enable Snapshot Support](deploy-workspace-snapshots.md) first if you have not configured it yet. {% endhint %} ## Prerequisites -- **[Snapshot support (workload manager)](deploy-workspace-snapshots.md) - configured** so triggered runs can execute pipeline snapshots in Kubernetes. -- A **Redis** instance reachable from the workspace namespace. The scheduler and - executor use Redis Streams as a message broker. Use a URL such as +- **[Snapshot support (workload manager)](deploy-workspace-snapshots.md) configured** so triggered runs can execute pipeline snapshots in Kubernetes. +- A **Redis** instance reachable from the workspace namespace. The scheduler and executor use Redis Streams as a message broker. Use a URL such as `redis://:6379/0`, or `rediss://:/0` when Redis requires TLS. -- Enough cluster resources for the two microservices below (see the example - `resources`). +- Enough cluster resources for the two microservices below (see the example `resources`). ## What to configure in Helm -The ZenML Helm chart deploys optional background processes as additional -microservices, each declared under the `workerDeployments` key in your +The ZenML Helm chart deploys optional background processes as additional microservices, each declared under the `workerDeployments` key in your workspace `values.yaml`. Each map entry becomes its own Kubernetes Deployment. Add a `workerDeployments` block next to your existing `server:` configuration. -Each microservice uses the same container image as the ZenML Pro server by -default and overrides the entrypoint to run the `plugins` helper with the +Each microservice uses the same container image as the ZenML Pro server by default and overrides the entrypoint to run the `plugins` helper with the subcommands below. The example enables both the **scheduler** and **executor** microservices: -they use the `plugins` command with `start-scheduler` and `start-executor`, -share `ZENML_REDIS_BROKER_URL`, and set SQLAlchemy pool sizes appropriate for -dedicated pods. Adjust `resources`, probes, and pool sizes to match your -cluster and load. +they use the `plugins` command with `start-scheduler` and `start-executor`, share `ZENML_REDIS_BROKER_URL`, and set SQLAlchemy pool sizes appropriate for +dedicated pods. Adjust `resources`, probes, and pool sizes to match your cluster and load. {% hint style="warning" %} The **scheduler** microservice must always run as a **single replica** with a -**`Recreate`** rollout strategy. Do not scale it horizontally or switch to -`RollingUpdate`; multiple scheduler pods or overlapping rollouts can break +**`Recreate`** rollout strategy. Do not scale it horizontally or switch to `RollingUpdate`; multiple scheduler pods or overlapping rollouts can break schedule and event dispatch. {% endhint %} @@ -158,10 +143,8 @@ workerDeployments: ``` For platform event triggers, the workspace **API server** must use the -**same** Redis broker URL so it can publish pipeline lifecycle events (for -example run completion) to Redis Streams for the executor to consume. Define -this as a Kubernetes Secret in `server.secretEnvironment` and merge with -any keys you already set: +**same** Redis broker URL so it can publish pipeline lifecycle events (for example run completion) to Redis Streams for the executor to consume. Define +this as a Kubernetes Secret in `server.secretEnvironment` and merge with any keys you already set: ```yaml server: @@ -180,8 +163,7 @@ server: ## Apply the change -After updating your values file, upgrade the release (adjust release name and -namespace as you use them): +After updating your values file, upgrade the release (adjust release name and namespace as you use them): ```bash helm upgrade zenml oci://public.ecr.aws/zenml/zenml \ @@ -191,12 +173,10 @@ helm upgrade zenml oci://public.ecr.aws/zenml/zenml \ ## Related behavior -- **Triggers and snapshots:** Both schedule and platform event triggers attach - to [pipeline snapshots](snapshots.md). Snapshot support is therefore a +- **Triggers and snapshots:** Both schedule and platform event triggers attach to [pipeline snapshots](snapshots.md). Snapshot support is therefore a **prerequisite**: without the [workload manager](deploy-workspace-snapshots.md), triggered runs cannot execute as described in [Triggers](triggers.md). -- **Concepts:** See [Schedule Triggers](triggers.md#schedule-triggers) and - [Platform Event Triggers](triggers.md#platform-event-triggers) for how these +- **Concepts:** See [Schedule Triggers](triggers.md#schedule-triggers) and [Platform Event Triggers](triggers.md#platform-event-triggers) for how these triggers are modeled in ZenML Pro. ## Related resources diff --git a/docs/book/getting-started/zenml-pro/deploy-workspace-k8s.md b/docs/book/getting-started/zenml-pro/deploy-workspace-k8s.md index 0d82a6dd3fe..fd91ffc4151 100644 --- a/docs/book/getting-started/zenml-pro/deploy-workspace-k8s.md +++ b/docs/book/getting-started/zenml-pro/deploy-workspace-k8s.md @@ -278,19 +278,14 @@ zenml login ### (Optional) Opt-in Workspace Server Features -Several ZenML Pro capabilities are not turned on by the default Helm -installation. They need extra infrastructure, environment variables, or -additional microservices beyond the main workspace server. Many are -paid add-ons on top of the base plan—see the -[pricing page](https://www.zenml.io/pricing)—and must be licensed and enabled -for your organization before they work end-to-end. +Several ZenML Pro capabilities are not turned on by the default Helm installation. They need extra infrastructure, environment variables, or +additional microservices beyond the main workspace server. Many are paid add-ons on top of the base plan—see the +[pricing page](https://www.zenml.io/pricing)—and must be licensed and enabled for your organization before they work end-to-end. **What it enables** points to ZenML Pro documentation for the capability. **What it deploys** summarizes the extra components at a high level. In the -ZenML Helm chart, each optional background process is modeled as an -additional microservice (its own Kubernetes Deployment) next to the API -server. You declare those microservices in `values.yaml` under the -`workerDeployments` map; each key under that map configures one microservice. +ZenML Helm chart, each optional background process is modeled as an additional microservice (its own Kubernetes Deployment) next to the API +server. You declare those microservices in `values.yaml` under the `workerDeployments` map; each key under that map configures one microservice. The per-feature guides show the exact YAML. | Guide | What it enables | What it deploys | Minimum workspace server version | @@ -299,17 +294,12 @@ The per-feature guides show the exact YAML. | [Enable Event Triggers and Schedules](deploy-workspace-event-triggers-and-schedules.md) | [Schedule triggers](triggers.md#schedule-triggers), [platform event triggers](triggers.md#platform-event-triggers) | **Scheduler** and **executor** microservices, **Redis** (broker URL via **`secretEnvironment`**); **`server.secretEnvironment`** **`ZENML_REDIS_BROKER_URL`** for platform events; requires [snapshot support](deploy-workspace-snapshots.md) to run attached snapshots | 0.94.3 | | [Enable Resource Pools](deploy-workspace-resource-pools.md) | [Resource pools](resource-pools.md) | **Resource pool reconciler** microservice | 0.94.3 | -Deploy [snapshot support](deploy-workspace-snapshots.md) before you rely on -[event triggers and schedules](deploy-workspace-event-triggers-and-schedules.md) -end-to-end: triggers run against pipeline snapshots, which need the workload -manager to execute on the cluster. - -**Platform event triggers** also require **`ZENML_REDIS_BROKER_URL`** on the -main workspace server, set through **`server.secretEnvironment`** (or -**`server.environmentSecretKeyRefs`**) rather than plain `server.environment`, -because broker URLs may contain credentials. Schedule-only triggers do not -need Redis on the server. See -[Enable Event Triggers and Schedules](deploy-workspace-event-triggers-and-schedules.md). +Deploy [snapshot support](deploy-workspace-snapshots.md) before you rely on [event triggers and schedules](deploy-workspace-event-triggers-and-schedules.md) +end-to-end: triggers run against pipeline snapshots, which need the workload manager to execute on the cluster. + +**Platform event triggers** also require **`ZENML_REDIS_BROKER_URL`** on the main workspace server, set through **`server.secretEnvironment`** (or +**`server.environmentSecretKeyRefs`**) rather than plain `server.environment`, because broker URLs may contain credentials. Schedule-only triggers do not +need Redis on the server. See [Enable Event Triggers and Schedules](deploy-workspace-event-triggers-and-schedules.md). ## Day 2 Operations diff --git a/docs/book/getting-started/zenml-pro/deploy-workspace-resource-pools.md b/docs/book/getting-started/zenml-pro/deploy-workspace-resource-pools.md index cb3d1e561d1..91ec4df5964 100644 --- a/docs/book/getting-started/zenml-pro/deploy-workspace-resource-pools.md +++ b/docs/book/getting-started/zenml-pro/deploy-workspace-resource-pools.md @@ -17,56 +17,43 @@ layout: # Enable Resource Pools for the Workspace Server -[Resource pools](resource-pools.md) let you model shared capacity (GPUs, custom -keys, and related limits) for dynamic pipelines. Keeping pool state +[Resource pools](resource-pools.md) let you model shared capacity (GPUs, custom keys, and related limits) for dynamic pipelines. Keeping pool state consistent uses a background **reconciler** process in ZenML Pro. -On **Kubernetes** self-hosted deployments, you enable that process by adding -a microservice (the resource pool reconciler) that runs +On **Kubernetes** self-hosted deployments, you enable that process by adding a microservice (the resource pool reconciler) that runs `plugins start-resource-pool-reconciler` (same image as the workspace server). {% hint style="warning" %} -**Commercial add-on:** Resource pools are not included in the base ZenML Pro -plan. Your organization must purchase and enable them explicitly. See the -[pricing page](https://www.zenml.io/pricing) for plans and contact ZenML if -you need entitlements enabled for your license. +**Commercial add-on:** Resource pools are not included in the base ZenML Pro plan. Your organization must purchase and enable them explicitly. See the +[pricing page](https://www.zenml.io/pricing) for plans and contact ZenML if you need entitlements enabled for your license. {% endhint %} {% hint style="warning" %} Deploy this microservice only for workspace servers installed with the ZenML -Helm chart on Kubernetes. Other platforms (for example AWS ECS) are not covered -here. +Helm chart on Kubernetes. Other platforms (for example AWS ECS) are not covered here. {% endhint %} ## Prerequisites -- Resource pools apply to - [dynamic pipelines](https://docs.zenml.io/how-to/steps-pipelines/dynamic_pipelines). +- Resource pools apply to [dynamic pipelines](https://docs.zenml.io/how-to/steps-pipelines/dynamic_pipelines). Ensure your teams understand that contract before enabling the reconciler. -- Enough cluster resources for one extra microservice (see the example - `resources` below). +- Enough cluster resources for one extra microservice (see the example `resources` below). ## What to configure in Helm -The ZenML Helm chart deploys optional background processes as additional -microservices, each declared under the `workerDeployments` key in your +The ZenML Helm chart deploys optional background processes as additional microservices, each declared under the `workerDeployments` key in your workspace `values.yaml`. Each map entry becomes its own Kubernetes Deployment. -Add the **resource pool reconciler** under `workerDeployments` next to your -existing `server:` configuration. That microservice uses the same container -image as the ZenML Pro server by default and overrides the entrypoint to run -the reconciler. +Add the **resource pool reconciler** under `workerDeployments` next to your existing `server:` configuration. That microservice uses the same container +image as the ZenML Pro server by default and overrides the entrypoint to run the reconciler. -Set SQLAlchemy pool sizes appropriate for a dedicated pod. The example below -is a reasonable starting point; adjust `resources` and probes for your +Set SQLAlchemy pool sizes appropriate for a dedicated pod. The example below is a reasonable starting point; adjust `resources` and probes for your environment. {% hint style="warning" %} -The **resource pool reconciler** microservice must always run as a **single -replica** with a **`Recreate`** rollout strategy. Do not scale it horizontally -or switch to `RollingUpdate`; multiple reconciler pods or overlapping -rollouts can corrupt or confuse pool reconciliation. +The **resource pool reconciler** microservice must always run as a **single replica** with a **`Recreate`** rollout strategy. Do not scale it horizontally +or switch to `RollingUpdate`; multiple reconciler pods or overlapping rollouts can corrupt or confuse pool reconciliation. {% endhint %} ```yaml @@ -115,8 +102,7 @@ workerDeployments: ## Apply the change -After updating your values file, upgrade the release (adjust release name and -namespace as you use them): +After updating your values file, upgrade the release (adjust release name and namespace as you use them): ```bash helm upgrade zenml oci://public.ecr.aws/zenml/zenml \ diff --git a/docs/book/getting-started/zenml-pro/deploy-workspace-snapshots.md b/docs/book/getting-started/zenml-pro/deploy-workspace-snapshots.md index 1b309af5830..cab8df8001b 100644 --- a/docs/book/getting-started/zenml-pro/deploy-workspace-snapshots.md +++ b/docs/book/getting-started/zenml-pro/deploy-workspace-snapshots.md @@ -155,13 +155,11 @@ Granting these permissions can be achieved in several ways: Use separate identities for: 1. the service account running the Workspace Server pod, -2. the service account configured in - `ZENML_KUBERNETES_WORKLOAD_MANAGER_SERVICE_ACCOUNT` for runner/builder jobs. +2. the service account configured in `ZENML_KUBERNETES_WORKLOAD_MANAGER_SERVICE_ACCOUNT` for runner/builder jobs. **Workspace Server service account RBAC (in workload manager namespace)** -The Workspace Server creates, monitors, and cleans up workload manager jobs and -fetches pod logs. Grant: +The Workspace Server creates, monitors, and cleans up workload manager jobs and fetches pod logs. Grant: - `batch/jobs`: `create`, `get`, `deletecollection` - `core/pods`: `list` @@ -170,13 +168,11 @@ fetches pod logs. Grant: **Workload manager runner/builder service account** The runner/builder jobs launched by the workload manager do not need Kubernetes -API permissions for the workload manager control loop itself. This account -mainly needs: +API permissions for the workload manager control loop itself. This account mainly needs: - container registry pull permissions for runner images, - container registry push permissions if build-on-demand is enabled, -- optional cloud permissions required by your workload implementation (for - example S3 write access when AWS external logs are enabled). +- optional cloud permissions required by your workload implementation (for example S3 write access when AWS external logs are enabled). ### 4. Environment Variable Reference diff --git a/docs/book/getting-started/zenml-pro/hierarchy.md b/docs/book/getting-started/zenml-pro/hierarchy.md index 6ad19fde23c..a2b27f064c5 100644 --- a/docs/book/getting-started/zenml-pro/hierarchy.md +++ b/docs/book/getting-started/zenml-pro/hierarchy.md @@ -5,13 +5,11 @@ icon: ball-pile # Hierarchy -In ZenML Pro, there is a slightly different entity hierarchy as compared to the open-source ZenML\ -framework. This document walks you through the key differences and new concepts that are only available for Pro users. +In ZenML Pro, there is a slightly different entity hierarchy as compared to the open-source ZenML\ framework. This document walks you through the key differences and new concepts that are only available for Pro users. ![Image showing the entity hierarchy in ZenML Pro](.gitbook/assets/org_hierarchy_pro.png) -{% hint style="info" %} -s**Note**: Workspaces were previously called "Tenants" in earlier versions of ZenML Pro. We've updated the terminology to better reflect their role in organizing MLOps resources. +{% hint style="info" %} s**Note**: Workspaces were previously called "Tenants" in earlier versions of ZenML Pro. We've updated the terminology to better reflect their role in organizing MLOps resources. {% endhint %} The image above shows the hierarchy of concepts in ZenML Pro. diff --git a/docs/book/getting-started/zenml-pro/resource-pools-core-concepts.md b/docs/book/getting-started/zenml-pro/resource-pools-core-concepts.md index 929188519dd..92f9c7b53d8 100644 --- a/docs/book/getting-started/zenml-pro/resource-pools-core-concepts.md +++ b/docs/book/getting-started/zenml-pro/resource-pools-core-concepts.md @@ -9,8 +9,7 @@ This page defines **pools**, **policies**, and **requests** for ZenML Pro. ## Pools -A **resource pool** is a named shared bucket. For each resource key (for -example `gpu`), you set how many units exist in the pool. Policies on that +A **resource pool** is a named shared bucket. For each resource key (for example `gpu`), you set how many units exist in the pool. Policies on that pool further split that capacity among orchestrators and step operators. Steps and pools use the same keys and integer amounts. Typical keys are: @@ -47,40 +46,28 @@ zenml resource-pool delete training-gpus --yes ## Policies -A **policy** connects one stack component—the orchestrator or step operator that -acts as the *resource requester* for a step—to one pool. Think of three knobs +A **policy** connects one stack component—the orchestrator or step operator that acts as the *resource requester* for a step—to one pool. Think of three knobs per resource key: * **Reserved** — How much of the pool you *label* as belonging to this component -for accounting. Usage up to that amount counts as *in share*; anything above it -(while the pool still has free units) is *borrowed* idle capacity. Reserved is -not a separate pile of hardware: it is the share used to decide who is “in -their rights” versus who is on spare capacity. Across all policies on the same -pool, reserved totals per key cannot exceed the pool capacity. Reserved must -also be ≤ that policy’s limit for the same key. +for accounting. Usage up to that amount counts as *in share*; anything above it (while the pool still has free units) is *borrowed* idle capacity. Reserved is +not a separate pile of hardware: it is the share used to decide who is “in their rights” versus who is on spare capacity. Across all policies on the same +pool, reserved totals per key cannot exceed the pool capacity. Reserved must also be ≤ that policy’s limit for the same key. * **Limit** — The hard ceiling on how much this component may hold from the pool -at once for that key. Grants never go above the limit, even if the pool is -empty. For **preemptible** workloads, the space between reserved and limit is -where borrowing can happen (subject to pool free capacity). Non-preemptible -work does not use that band: each requested amount per key must be **≤ -reserved**, and a higher **limit** does not raise that ceiling (limit still caps -preemptible burst and total use). +at once for that key. Grants never go above the limit, even if the pool is empty. For **preemptible** workloads, the space between reserved and limit is +where borrowing can happen (subject to pool free capacity). Non-preemptible work does not use that band: each requested amount per key must be **≤ +reserved**, and a higher **limit** does not raise that ceiling (limit still caps preemptible burst and total use). * **Priority** — A number; higher means that component’s requests are preferred in -the queue. When the reconciler must **preempt** someone, it looks at **lower** -priority preemptible runs first as victims (see below). +the queue. When the reconciler must **preempt** someone, it looks at **lower** priority preemptible runs first as victims (see below). {% hint style="warning" %} -A single orchestrator or step operator may have **several policies** attached, -each pointing at a **different pool**. The server still treats each step as one -**resource request**. Eligibility and allocation are evaluated **per pool** -against the **full** set of requested keys: the step may be queued on more than -one pool, but at most **one** pool ends up owning the active allocation. ZenML -does **not** split a request across pools (for example GPUs from one pool and +A single orchestrator or step operator may have **several policies** attached, each pointing at a **different pool**. The server still treats each step as one +**resource request**. Eligibility and allocation are evaluated **per pool** against the **full** set of requested keys: the step may be queued on more than +one pool, but at most **one** pool ends up owning the active allocation. ZenML does **not** split a request across pools (for example GPUs from one pool and `mcpu` from another). Every key in the request must be satisfiable from the -**same** pool and policy that wins. See -[Examples — Multiple pools and multi-key requests](resource-pools-examples.md#multiple-pools-and-multi-key-requests). +**same** pool and policy that wins. See [Examples — Multiple pools and multi-key requests](resource-pools-examples.md#multiple-pools-and-multi-key-requests). {% endhint %} ### CLI: policies @@ -111,38 +98,28 @@ zenml resource-pool detach-policy training-gpus my-k8s-orch ## Resource requests -For eligible runs, the server builds a **resource request** from the step’s -`ResourceSettings`, records whether the step is preemptible, and tracks status: +For eligible runs, the server builds a **resource request** from the step’s `ResourceSettings`, records whether the step is preemptible, and tracks status: queued, allocated, rejected, preempted, or cancelled. {% hint style="info" %} -Only dynamic pipelines participate in resource queuing and allocation -waiting: the server creates resource requests and the client blocks until -allocation when the snapshot is dynamic. Static pipelines do not use this -path today. +Only dynamic pipelines participate in resource queuing and allocation waiting: the server creates resource requests and the client blocks until +allocation when the snapshot is dynamic. Static pipelines do not use this path today. {% endhint %} -What users set in `ResourceSettings` becomes a server-side resource request (for -example `gpu_count` → `gpu`, `cpu_count` → `mcpu`, `memory` → `memory_mb`, plus -an implicit concurrent `step_run` slot). The pool must define capacity for each -key requested by the step, except for three built-in types: if the pool has no -row for `mcpu`, `memory_mb`, or the implicit `step_run` key, ZenML Pro treats that -dimension as **effectively unbounded** at the pool layer, so missing rows there -do not by themselves cause rejection. For every other key (including -`gpu` and custom keys from `pool_resources`), a missing pool row means zero +What users set in `ResourceSettings` becomes a server-side resource request (for example `gpu_count` → `gpu`, `cpu_count` → `mcpu`, `memory` → `memory_mb`, plus +an implicit concurrent `step_run` slot). The pool must define capacity for each key requested by the step, except for three built-in types: if the pool has no +row for `mcpu`, `memory_mb`, or the implicit `step_run` key, ZenML Pro treats that dimension as **effectively unbounded** at the pool layer, so missing rows there +do not by themselves cause rejection. For every other key (including `gpu` and custom keys from `pool_resources`), a missing pool row means zero capacity: a positive request is rejected and the step run fails to start. {% hint style="warning" %} -If the pool does define a key but the subject policy omits that key, -limits fall back to the pool total and reserved defaults to zero for bounded -keys. Non-preemptible steps must stay within their reserved capacity, so a -positive request for a key not defined in the policy is rejected. +If the pool does define a key but the subject policy omits that key, limits fall back to the pool total and reserved defaults to zero for bounded +keys. Non-preemptible steps must stay within their reserved capacity, so a positive request for a key not defined in the policy is rejected. {% endhint %} ### Step decorators: `ResourceSettings` -Declare demand on the step; the server turns it into the resource request when -the pipeline is **dynamic** and pooling applies to the stack. +Declare demand on the step; the server turns it into the resource request when the pipeline is **dynamic** and pooling applies to the stack. **Typical GPU / CPU / memory (preemptible by default):** @@ -194,14 +171,12 @@ def infer() -> None: ... ``` -Typed fields override the same keys if both appear in `pool_resources`. See -[step configuration](https://docs.zenml.io/how-to/steps-pipelines/configuration) +Typed fields override the same keys if both appear in `pool_resources`. See [step configuration](https://docs.zenml.io/how-to/steps-pipelines/configuration) in the OSS docs for the full `ResourceSettings` model. ### CLI: resource requests -Requests are created when dynamic steps run; you **inspect** or **clean them up** -from the CLI (IDs come from list output or the dashboard). +Requests are created when dynamic steps run; you **inspect** or **clean them up** from the CLI (IDs come from list output or the dashboard). ```bash # All resource requests visible to your user (see --help for filters) diff --git a/docs/book/getting-started/zenml-pro/resource-pools-examples.md b/docs/book/getting-started/zenml-pro/resource-pools-examples.md index dcf11b19e71..f8d037b4bc1 100644 --- a/docs/book/getting-started/zenml-pro/resource-pools-examples.md +++ b/docs/book/getting-started/zenml-pro/resource-pools-examples.md @@ -6,24 +6,19 @@ description: >- # Resource pool examples (workbook) Read this page like a short course: each section is one self-contained scenario. -You will always see three things—the pool (shared capacity), the policy (how one -orchestrator or step operator may use that pool), and the step +You will always see three things—the pool (shared capacity), the policy (how one orchestrator or step operator may use that pool), and the step (`ResourceSettings`). Then we spell out what the server does. **Assumptions unless stated otherwise:** * Steps are preemptible by default if you omit `preemptible=False`. -* One step run at a time when we say “no other work is running,” so you can - focus on a single decision. -* Every key in a policy’s `reserved` and `limit` must exist on the pool’s - capacity. You cannot meter a resource in policy that the pool does not +* One step run at a time when we say “no other work is running,” so you can focus on a single decision. +* Every key in a policy’s `reserved` and `limit` must exist on the pool’s capacity. You cannot meter a resource in policy that the pool does not define. -* If one orchestrator or step operator has policies to several pools, the step - still receives at most one allocation from one pool. The whole request must be +* If one orchestrator or step operator has policies to several pools, the step still receives at most one allocation from one pool. The whole request must be eligible on that pool; resources are not split across pools for a single step. -For definitions of reserved, limit, and priority, see -[Core concepts](resource-pools-core-concepts.md). For preemption ordering, see +For definitions of reserved, limit, and priority, see [Core concepts](resource-pools-core-concepts.md). For preemption ordering, see [How preemption works](resource-pools-reconciliation.md#how-preemption-works). ## Primer: from `ResourceSettings` to the resource request @@ -52,12 +47,9 @@ ZenML turns that into one resource request. Roughly: | `pool_resources` | (your names) | Copied as-is, merged with typed fields | 1 | | Server | `step_run` | Always 1 per step | 1 | -The pool must define capacity for bounded keys such as `gpu` and -`tensorrt_sessions`. If the pool has no row for `mcpu`, `memory_mb`, or -`step_run`, that dimension is unbounded at the pool layer (see the examples -below). For everything else, a missing pool row means zero capacity. If you want -a policy to set `reserved` / `limit` on a key, that key must appear on the pool -first—policy keys are always a subset of pool keys. +The pool must define capacity for bounded keys such as `gpu` and `tensorrt_sessions`. If the pool has no row for `mcpu`, `memory_mb`, or +`step_run`, that dimension is unbounded at the pool layer (see the examples below). For everything else, a missing pool row means zero capacity. If you want +a policy to set `reserved` / `limit` on a key, that key must appear on the pool first—policy keys are always a subset of pool keys. --- @@ -115,16 +107,14 @@ def train() -> None: ... ``` -**Outcome:** Allocated immediately (no queue). Four GPUs count against the -policy reserved share; two are borrowed from free pool capacity (between +**Outcome:** Allocated immediately (no queue). Four GPUs count against the policy reserved share; two are borrowed from free pool capacity (between reserved and limit, and pool must still have free units). --- ### Non-preemptible step stays inside reserved -**Story:** Same pool and policy. Production wants two GPUs and opts out of -preemption. Two is within the four-GPU reservation. +**Story:** Same pool and policy. Production wants two GPUs and opts out of preemption. Two is within the four-GPU reservation. **Pool** @@ -169,15 +159,13 @@ def production_train() -> None: ... ``` -**Outcome:** Allocated (assuming no other contention). Non-preemptible work must -satisfy requested ≤ reserved per key; `2 ≤ 4` passes. +**Outcome:** Allocated (assuming no other contention). Non-preemptible work must satisfy requested ≤ reserved per key; `2 ≤ 4` passes. --- ### Non-preemptible step beyond reserved -**Story:** Same pool and policy. Production asks for six GPUs but refuses -preemption. Non-preemptible work cannot use the “borrow” band above reserved. +**Story:** Same pool and policy. Production asks for six GPUs but refuses preemption. Non-preemptible work cannot use the “borrow” band above reserved. **Pool** @@ -222,8 +210,7 @@ def too_large_production_train() -> None: ... ``` -**Outcome:** Rejected immediately (dynamic run fails fast). Six exceeds reserved -(4) for `gpu`; non-preemptible requests cannot borrow up to limit. +**Outcome:** Rejected immediately (dynamic run fails fast). Six exceeds reserved (4) for `gpu`; non-preemptible requests cannot borrow up to limit. --- @@ -231,10 +218,8 @@ def too_large_production_train() -> None: ### GPU-only pool—CPU and memory not quota’d -**Story:** You only modeled GPUs on the pool and policy. The step still sends -`mcpu` and `memory_mb` on the request, but those keys are unbounded at the pool -layer when omitted, and this policy omits them too—so they do not block -non-preemptible work. +**Story:** You only modeled GPUs on the pool and policy. The step still sends `mcpu` and `memory_mb` on the request, but those keys are unbounded at the pool +layer when omitted, and this policy omits them too—so they do not block non-preemptible work. **Pool** @@ -282,15 +267,13 @@ def hungry_but_ok_on_gpu() -> None: ``` **Outcome:** Allocated if nothing else is wrong. Only `gpu` is gated here; -`mcpu` / `memory_mb` / `step_run` are not limited by pool or policy in this -pattern. CPU and memory remain informational unless you add rows later. +`mcpu` / `memory_mb` / `step_run` are not limited by pool or policy in this pattern. CPU and memory remain informational unless you add rows later. --- ### Non-preemptible CPU inside policy reserved -**Story:** You cap milli-CPU on the pool, then split it with reserved / limit -on the policy. Non-preemptible CPU demand must fit reserved per key. +**Story:** You cap milli-CPU on the pool, then split it with reserved / limit on the policy. Non-preemptible CPU demand must fit reserved per key. **Pool** @@ -339,8 +322,7 @@ def fits_reserved_cpu() -> None: ... ``` -**Outcome:** Allocated. `cpu_count=2` → `mcpu` 2000 ≤ reserved 4000, and `gpu` is -valid. +**Outcome:** Allocated. `cpu_count=2` → `mcpu` 2000 ≤ reserved 4000, and `gpu` is valid. --- @@ -400,8 +382,7 @@ non-preemptible work cannot borrow toward limit on `mcpu`. ### Preemptible CPU burst with policy `mcpu` rows -**Pool** and **Policy:** same as *Non-preemptible CPU inside policy reserved* -(pool includes `gpu` and `mcpu`; policy sets both keys). +**Pool** and **Policy:** same as *Non-preemptible CPU inside policy reserved* (pool includes `gpu` and `mcpu`; policy sets both keys). **Step** @@ -419,15 +400,13 @@ def preemptible_cpu_burst() -> None: ... ``` -**Outcome:** May allocate using headroom up to limit on `mcpu` (and pool free -capacity), analogous to GPU borrowing. +**Outcome:** May allocate using headroom up to limit on `mcpu` (and pool free capacity), analogous to GPU borrowing. --- ### Preemptible when pool lists `mcpu` but policy omits it -**Story:** The pool caps total milli-CPU. With no `mcpu` on the policy, reserved -defaults to 0 and limit falls back to the pool total. +**Story:** The pool caps total milli-CPU. With no `mcpu` on the policy, reserved defaults to 0 and limit falls back to the pool total. **Pool** @@ -474,8 +453,7 @@ def preemptible_with_pool_mcpu() -> None: ... ``` -**Outcome:** Allocated or queued then allocated when possible. `mcpu` 4000 ≤ -effective limit 8000 (pool total). +**Outcome:** Allocated or queued then allocated when possible. `mcpu` 4000 ≤ effective limit 8000 (pool total). --- @@ -499,16 +477,14 @@ def non_preemptible_positive_mcpu_zero_reserved() -> None: ... ``` -**Outcome:** Rejected. Any positive `mcpu` with reserved 0 fails for -non-preemptible work. Fix: add `mcpu` to the policy with enough reserved, or +**Outcome:** Rejected. Any positive `mcpu` with reserved 0 fails for non-preemptible work. Fix: add `mcpu` to the policy with enough reserved, or remove `mcpu` from the pool if you wanted fully unbounded CPU at the pool layer. --- ### Capping concurrent steps with `step_run` -**Story:** You want both GPUs and a ceiling on how many steps from this -orchestrator run at once. Each step always requests one `step_run`. +**Story:** You want both GPUs and a ceiling on how many steps from this orchestrator run at once. Each step always requests one `step_run`. **Pool** @@ -556,8 +532,7 @@ def train() -> None: ... ``` -**Outcome:** The server grants only when both `gpu` and `step_run` have enough -free units. If GPUs are free but all `step_run` slots are taken, the request +**Outcome:** The server grants only when both `gpu` and `step_run` have enough free units. If GPUs are free but all `step_run` slots are taken, the request waits in the queue. --- @@ -615,8 +590,7 @@ def infer() -> None: ... ``` -**Outcome:** Allocated when `1 ≤ reserved` for both `gpu` and -`tensorrt_sessions`. Unbounded defaults do not apply to custom keys—the pool +**Outcome:** Allocated when `1 ≤ reserved` for both `gpu` and `tensorrt_sessions`. Unbounded defaults do not apply to custom keys—the pool must list them. --- @@ -671,8 +645,7 @@ def infer() -> None: ``` **Outcome:** Rejected. Missing policy row → reserved 0 for `tensorrt_sessions`; -non-preemptible cannot ask for a positive amount. Fix: add `tensorrt_sessions` to -the policy, or mark the step preemptible if borrowing is acceptable. +non-preemptible cannot ask for a positive amount. Fix: add `tensorrt_sessions` to the policy, or mark the step preemptible if borrowing is acceptable. --- @@ -723,8 +696,7 @@ def too_big_for_planet() -> None: ... ``` -**Outcome:** Rejected immediately. Ten exceeds the pool total for `gpu`; the -request does not join a queue. +**Outcome:** Rejected immediately. Ten exceeds the pool total for `gpu`; the request does not join a queue. --- @@ -773,8 +745,7 @@ def over_team_limit() -> None: ... ``` -**Outcome:** Rejected. Six exceeds this component’s limit (4) for `gpu`, even if -eight GPUs exist in the pool. +**Outcome:** Rejected. Six exceeds this component’s limit (4) for `gpu`, even if eight GPUs exist in the pool. --- @@ -782,8 +753,7 @@ eight GPUs exist in the pool. ### Two teams, same priority, not enough GPUs -**Story:** Red and Blue orchestrators share one pool. Policies use the same -priority. Many preemptible steps each want 2 GPUs; the pool cannot satisfy +**Story:** Red and Blue orchestrators share one pool. Policies use the same priority. Many preemptible steps each want 2 GPUs; the pool cannot satisfy everyone at once. **Pool** @@ -843,20 +813,16 @@ def train() -> None: ... ``` -**Outcome:** Requests wait in the pool queue until GPUs free up. Among the same -policy priority, ordering tends to favor older waiters (FIFO-style). The -allocator also prefers a request that still fits entirely in its unused reserved -slice over one that must borrow when both are waiting—so a team with reservation -headroom is not stuck behind another team that is already bursting, if the next -grant can be served from that reserved slice. No preemption until a +**Outcome:** Requests wait in the pool queue until GPUs free up. Among the same policy priority, ordering tends to favor older waiters (FIFO-style). The +allocator also prefers a request that still fits entirely in its unused reserved slice over one that must borrow when both are waiting—so a team with reservation +headroom is not stuck behind another team that is already bursting, if the next grant can be served from that reserved slice. No preemption until a higher-priority waiter or reclaim logic forces it. --- ### Higher priority wins; lower may be preempted -**Story:** Sandbox bursts with preemptible work. Production has higher policy -priority and needs GPUs when the pool is full. +**Story:** Sandbox bursts with preemptible work. Production has higher policy priority and needs GPUs when the pool is full. **Pool** @@ -930,17 +896,14 @@ def prod_train() -> None: ... ``` -**Outcome:** If four GPUs cannot be granted without reclaiming space, the -reconciler may preempt Sandbox’s preemptible runs (lower policy priority) so Prod -can proceed. See [How preemption works](resource-pools-reconciliation.md#how-preemption-works) for victim -ordering. +**Outcome:** If four GPUs cannot be granted without reclaiming space, the reconciler may preempt Sandbox’s preemptible runs (lower policy priority) so Prod +can proceed. See [How preemption works](resource-pools-reconciliation.md#how-preemption-works) for victim ordering. --- ### Production non-preemptible waits on reserved only -**Story:** Prod uses `preemptible=False` and asks only for what is reserved. If -another non-preemptible job on the same stack component already holds the reserved +**Story:** Prod uses `preemptible=False` and asks only for what is reserved. If another non-preemptible job on the same stack component already holds the reserved GPUs, this step does not borrow from Sandbox’s burst. **Pool** @@ -971,26 +934,21 @@ def prod_sla_job() -> None: ... ``` -**Outcome:** Waits in the queue if Prod’s reserved `gpu` (2) is already used by -other non-preemptible work on `prod-orch`. It will not take Sandbox’s borrowed -GPUs. Ways out: raise reserved for Prod, wait for the other job to finish, or -use preemptible Prod work if policy allows. +**Outcome:** Waits in the queue if Prod’s reserved `gpu` (2) is already used by other non-preemptible work on `prod-orch`. It will not take Sandbox’s borrowed +GPUs. Ways out: raise reserved for Prod, wait for the other job to finish, or use preemptible Prod work if policy allows. --- ## Multiple pools and multi-key requests {% hint style="warning" %} -Several policies on the same stack component mean several pools may try -to satisfy the same resource request, but only one pool can win. -Every key in the request must pass that pool’s checks; ZenML does not take -`gpu` from one pool and `mcpu` from another for one step. +Several policies on the same stack component mean several pools may try to satisfy the same resource request, but only one pool can win. +Every key in the request must pass that pool’s checks; ZenML does not take `gpu` from one pool and `mcpu` from another for one step. {% endhint %} ### Two pools on one orchestrator—primary pool wins -**Story:** You attach two policies to the same orchestrator pointing at -different pools. The step still produces one resource request, enqueued in every +**Story:** You attach two policies to the same orchestrator pointing at different pools. The step still produces one resource request, enqueued in every eligible pool; only one pool may win. **Pools** @@ -1058,17 +1016,14 @@ def train() -> None: ... ``` -**Outcome:** The server tries higher policy priority first—eu-west before -eu-north. Whichever pool grants first owns the allocation; the other queue entry -is dropped as stale. Use this for primary/fallback or regional capacity, not for -splitting one step across unrelated quotas. +**Outcome:** The server tries higher policy priority first—eu-west before eu-north. Whichever pool grants first owns the allocation; the other queue entry +is dropped as stale. Use this for primary/fallback or regional capacity, not for splitting one step across unrelated quotas. --- ### One step must satisfy every key in each pool -**Story:** Eligibility is checked per pool against all keys on the request. If a -pool lacks a key the step needs, that pool treats it as zero capacity—the request +**Story:** Eligibility is checked per pool against all keys on the request. If a pool lacks a key the step needs, that pool treats it as zero capacity—the request is not eligible there. **Pool A** (GPUs only) @@ -1112,26 +1067,19 @@ def infer() -> None: ... ``` -**Outcome:** A pool with only `gpu` cannot satisfy `tensorrt_sessions`—that -dimension is zero there, so the request does not enqueue on that pool. +**Outcome:** A pool with only `gpu` cannot satisfy `tensorrt_sessions`—that dimension is zero there, so the request does not enqueue on that pool. -**Lesson:** Model every scarce bounded dimension you care about on one pool (or -ensure every candidate pool defines the same key set for those keys). The next -section shows how `mcpu` and `memory_mb` differ: omitting them on one pool keeps -that path eligible even when another pool meters them strictly. +**Lesson:** Model every scarce bounded dimension you care about on one pool (or ensure every candidate pool defines the same key set for those keys). The next +section shows how `mcpu` and `memory_mb` differ: omitting them on one pool keeps that path eligible even when another pool meters them strictly. --- ### Two pools: higher-priority path meters CPU/RAM; GPU-only path still wins -**Story:** One orchestrator has two policies (same pattern as *Two pools on one -orchestrator—primary pool wins*). Pool B’s capacity and policy include `mcpu` and -`memory_mb`, with reserved amounts sized for small non-preemptible jobs. Pool A -only defines `gpu`; it does not list `mcpu` or `memory_mb`, so those dimensions -are unbounded at the pool layer and its policy does not reserve them. A -non-preemptible step asks for one GPU but more CPU and RAM than Pool B’s policy -allows. The higher-priority policy (Pool B) cannot grant that request; the -lower-priority policy (Pool A) can, because the request’s CPU and memory demand is +**Story:** One orchestrator has two policies (same pattern as *Two pools on one orchestrator—primary pool wins*). Pool B’s capacity and policy include `mcpu` and +`memory_mb`, with reserved amounts sized for small non-preemptible jobs. Pool A only defines `gpu`; it does not list `mcpu` or `memory_mb`, so those dimensions +are unbounded at the pool layer and its policy does not reserve them. A non-preemptible step asks for one GPU but more CPU and RAM than Pool B’s policy +allows. The higher-priority policy (Pool B) cannot grant that request; the lower-priority policy (Pool A) can, because the request’s CPU and memory demand is not quota’d on that path. The allocation is owned by Pool A. **Pool A** (GPUs only—no `mcpu` or `memory_mb` on the pool) @@ -1158,8 +1106,7 @@ not quota’d on that path. The allocation is owned by Pool A. } ``` -**Policies** (same component, different priorities—B is preferred when both can -grant) +**Policies** (same component, different priorities—B is preferred when both can grant) ```json [ @@ -1211,17 +1158,12 @@ def train() -> None: ... ``` -**Outcome:** The request maps to roughly `mcpu` 8000 and tens of thousands of -`memory_mb` for `32GiB`. For non-preemptible work, each key must be ≤ policy -reserved on the path you use. Pool B’s policy reserves only `mcpu` 4000 and -`memory_mb` 8192, so that path cannot satisfy the step. Pool A’s policy has no -`mcpu` or `memory_mb` rows; with those keys absent from the pool, they are not -treated as zero capacity, so the step remains eligible there on `gpu` alone. The +**Outcome:** The request maps to roughly `mcpu` 8000 and tens of thousands of `memory_mb` for `32GiB`. For non-preemptible work, each key must be ≤ policy +reserved on the path you use. Pool B’s policy reserves only `mcpu` 4000 and `memory_mb` 8192, so that path cannot satisfy the step. Pool A’s policy has no +`mcpu` or `memory_mb` rows; with those keys absent from the pool, they are not treated as zero capacity, so the step remains eligible there on `gpu` alone. The reconciler allocates from Pool A and drops the competing queue row for Pool B. -If you want large non-preemptible jobs to stay on the metered pool, raise -reserved (and capacity) on Pool B for `mcpu` and `memory_mb`, or reduce demand in -`ResourceSettings`—otherwise the GPU-only policy acts as an escape hatch for -heavy CPU/RAM asks. +If you want large non-preemptible jobs to stay on the metered pool, raise reserved (and capacity) on Pool B for `mcpu` and `memory_mb`, or reduce demand in +`ResourceSettings`—otherwise the GPU-only policy acts as an escape hatch for heavy CPU/RAM asks. --- @@ -1229,7 +1171,5 @@ heavy CPU/RAM asks. * [Resource pools](resource-pools.md) — overview * [Core concepts](resource-pools-core-concepts.md) — pools, policies, requests -* [How preemption works](resource-pools-reconciliation.md#how-preemption-works) — preemption - ordering -* [Step configuration](https://docs.zenml.io/how-to/steps-pipelines/configuration) - — full `ResourceSettings` reference +* [How preemption works](resource-pools-reconciliation.md#how-preemption-works) — preemption ordering +* [Step configuration](https://docs.zenml.io/how-to/steps-pipelines/configuration) — full `ResourceSettings` reference diff --git a/docs/book/getting-started/zenml-pro/resource-pools-reconciliation.md b/docs/book/getting-started/zenml-pro/resource-pools-reconciliation.md index c44e22216bf..4adb7f65084 100644 --- a/docs/book/getting-started/zenml-pro/resource-pools-reconciliation.md +++ b/docs/book/getting-started/zenml-pro/resource-pools-reconciliation.md @@ -6,66 +6,51 @@ description: >- ## Runtime flow (orchestration) -1. Request creation: For eligible runs, the server derives requested - resources from the step’s `ResourceSettings` (see below), adds `step_run: 1`, +1. Request creation: For eligible runs, the server derives requested resources from the step’s `ResourceSettings` (see below), adds `step_run: 1`, and stores `preemptible` from the same settings. The resource requester is the stack’s step operator if the step uses one, otherwise the orchestrator. -2. Queuing: If capacity is not available immediately, the step can remain - queued until the reconciler allocates it. -3. Client wait: The step launcher polls the resource request until it is allocated - (with backoff). If the request is not allocated, it is rejected, preempted, or cancelled, the client surfaces an error. When allocation succeeds, the step is published as running and execution proceeds. -4. Preemption: If the job at the front of the queue still cannot be granted, - the reconciler may stop other *preemptible* runs in that pool to free units +2. Queuing: If capacity is not available immediately, the step can remain queued until the reconciler allocates it. +3. Client wait: The step launcher polls the resource request until it is allocated (with backoff). If the request is not allocated, it is rejected, preempted, or cancelled, the client surfaces an error. When allocation succeeds, the step is published as running and execution proceeds. +4. Preemption: If the job at the front of the queue still cannot be granted, the reconciler may stop other *preemptible* runs in that pool to free units (see [How preemption works](#how-preemption-works)). Non-preemptible runs are never stopped this way. They are also constrained so each request’s per-key demand is **≤ policy reserved** for that key—even when **limit** is higher—so they never rely on borrowed capacity that could clash with other non-preemptible use on the same component. 5. Post-preemption retry: after preemption, if the step configuration allows -retries, the step goes back to the queue and is retried again. If the number of -retries is exhausted, the step fails. See [Automatic Step Retries](https://docs.zenml.io/how-to/steps-pipelines/advanced_features#automatic-step-retries) for more information. +retries, the step goes back to the queue and is retried again. If the number of retries is exhausted, the step fails. See [Automatic Step Retries](https://docs.zenml.io/how-to/steps-pipelines/advanced_features#automatic-step-retries) for more information. 6. Deallocation: When the step run completes, the resources are released back to the pool. If the step crashes unexpectedly, the resources are eventually released back to the pool. {% hint style="warning" %} -If the resource requester (orchestrator or step operator) has more than one -policy attached to more than one pool, the same logical request may appear in -several pool queues, but reconciliation still grants at most one active -allocation. Eligibility is computed separately per pool against the entire -resource request; the system never assigns part of a step’s demand to one pool -and the rest to another. +If the resource requester (orchestrator or step operator) has more than one policy attached to more than one pool, the same logical request may appear in +several pool queues, but reconciliation still grants at most one active allocation. Eligibility is computed separately per pool against the entire +resource request; the system never assigns part of a step’s demand to one pool and the rest to another. {% endhint %} ## How preemption works -**When.** Preemption runs only when the next queued request for a pool cannot be -allocated—there is not enough free capacity, or a policy rule blocks the -grant. The reconciler may then mark selected *already running* requests as -preempted, which cancels those step runs and returns their units to the pool. +**When.** Preemption runs only when the next queued request for a pool cannot be allocated—there is not enough free capacity, or a policy rule blocks the +grant. The reconciler may then mark selected *already running* requests as preempted, which cancels those step runs and returns their units to the pool. -**Who can be stopped.** Only steps with `preemptible=True` (the default in -`ResourceSettings`) are candidates. `preemptible=False` means “never pick this +**Who can be stopped.** Only steps with `preemptible=True` (the default in `ResourceSettings`) are candidates. `preemptible=False` means “never pick this run as the one to kill.” **Who gets stopped first (simple picture).** -1. Among preemptible runs in the same pool, **lower policy priority** is - considered before higher priority. If the waiting job’s priority is *strictly +1. Among preemptible runs in the same pool, **lower policy priority** is considered before higher priority. If the waiting job’s priority is *strictly higher* than a victim’s, that victim can be preempted to make room, as long as freeing it actually fixes the shortage. -2. **Reserved** adds a second idea: *reclaim*. If the waiting component still - has unused **reserved** headroom on this pool (reserved minus what it is +2. **Reserved** adds a second idea: *reclaim*. If the waiting component still has unused **reserved** headroom on this pool (reserved minus what it is already using here), the system may preempt preemptible runs that are using **borrowed** capacity—even when those runs have the same or higher priority than the waiter. Intuition: your reserved share is “yours to fill”; if someone else is on the spare capacity you could have used under your reservation, they can be moved out of the way. -3. **Limit** does not pick victims. It only caps how much a component may hold; - if the waiting request itself is over its own limit, killing other jobs will +3. **Limit** does not pick victims. It only caps how much a component may hold; if the waiting request itself is over its own limit, killing other jobs will not fix that—you need a higher limit or a smaller request. -Victims are ordered by ascending policy priority, then by allocation time as a -tie-break. +Victims are ordered by ascending policy priority, then by allocation time as a tie-break. ### Step-level: `preemptible` @@ -74,38 +59,30 @@ tie-break. | `preemptible=True` (default) | This run may be preempted to help another request. | | `preemptible=False` | This run is never preempted. Each requested amount per pool key must be ≤ that key’s **reserved** on the policy; **limit** above reserved does not increase what a non-preemptible step may request. | -Policies do not override `preemptible`; they only affect ordering and reclaim -among runs that are allowed to be preempted. +Policies do not override `preemptible`; they only affect ordering and reclaim among runs that are allowed to be preempted. ### After preemption Preempted step runs are stopped and the resources are released back to the pool. -The steps are put back into the queue and are retried again. If the number of -retries is exhausted or the step is not configured to allow retries, the step fails. See [Automatic Step Retries](https://docs.zenml.io/how-to/steps-pipelines/advanced_features#automatic-step-retries) for more information. +The steps are put back into the queue and are retried again. If the number of retries is exhausted or the step is not configured to allow retries, the step fails. See [Automatic Step Retries](https://docs.zenml.io/how-to/steps-pipelines/advanced_features#automatic-step-retries) for more information. ## Policy scenarios (how reserved, limit, and preemptible interact) -For the problems these patterns solve in everyday terms, see -[Resource pools](resource-pools.md). +For the problems these patterns solve in everyday terms, see [Resource pools](resource-pools.md). -* Fair share plus burst: set **reserved** to the slice you want to account as - “yours” and **limit** to the most that stack may ever hold. **Preemptible** +* Fair share plus burst: set **reserved** to the slice you want to account as “yours” and **limit** to the most that stack may ever hold. **Preemptible** steps can **borrow** idle capacity between reserved and limit (and up to the pool) when the pool has room. **Non-preemptible** steps only use up to **reserved** per requested key, regardless of a higher limit. -* Production vs experiments: higher **priority** on production policies; - experimental steps stay **preemptible** so production can take capacity or +* Production vs experiments: higher **priority** on production policies; experimental steps stay **preemptible** so production can take capacity or reclaim borrowed slack when it needs its reservation. -* Non-preemptible training: set `preemptible=False` and size **reserved** so - each step’s per-key request (for example `gpu_count`) is ≤ reserved for that +* Non-preemptible training: set `preemptible=False` and size **reserved** so each step’s per-key request (for example `gpu_count`) is ≤ reserved for that key. **Limit** can be higher for preemptible burst on the same policy, but it does not raise the ceiling for non-preemptible requests; raise **reserved** if those jobs need more per step. The reconciler also blocks non-preemptible grants that would sit on borrowed capacity in ways that conflict with other non-preemptible use on the component. -* Several pools for one component: multiple policies with different - **priority** values; higher priority is preferred when queuing and allocating, +* Several pools for one component: multiple policies with different **priority** values; higher priority is preferred when queuing and allocating, subject to each pool’s **limit**. -For preemption rules (priority vs reclaim), see -[How preemption works](#how-preemption-works). +For preemption rules (priority vs reclaim), see [How preemption works](#how-preemption-works). diff --git a/docs/book/getting-started/zenml-pro/resource-pools.md b/docs/book/getting-started/zenml-pro/resource-pools.md index 2bdf3ca9c6e..9da1f6d9f8a 100644 --- a/docs/book/getting-started/zenml-pro/resource-pools.md +++ b/docs/book/getting-started/zenml-pro/resource-pools.md @@ -8,137 +8,98 @@ icon: layer-group # Resource pools {% hint style="info" %} -Resource pools are part of ZenML's paid features. For availability and plans, -see the [pricing page](https://www.zenml.io/pricing). +Resource pools are part of ZenML's paid features. For availability and plans, see the [pricing page](https://www.zenml.io/pricing). {% endhint %} {% hint style="info" %} Resource pools are only available for [dynamic pipelines](https://docs.zenml.io/how-to/steps-pipelines/dynamic_pipelines). {% endhint %} -If you run AI or ML work in a shared environment, you have probably seen the -same problems: jobs fighting over GPUs, surprise slowdowns when another team -launches a big run, or expensive hardware sitting idle while someone waits in a -queue. Resource pools are ZenML Pro’s answer. They are aimed at -*platform and team leaders* who need clear rules, and at *practitioners* who -just want their steps to get the right compute without babysitting the cluster. +If you run AI or ML work in a shared environment, you have probably seen the same problems: jobs fighting over GPUs, surprise slowdowns when another team +launches a big run, or expensive hardware sitting idle while someone waits in a queue. Resource pools are ZenML Pro’s answer. They are aimed at +*platform and team leaders* who need clear rules, and at *practitioners* who just want their steps to get the right compute without babysitting the cluster. Here are typical situations this feature is built for: **“Our production jobs must finish—no surprises.”** -You have training, fine-tuning, or inference that cannot vanish because someone -else submitted a heavier workload. You want a clear agreement: this stack or -this team gets a dependable slice of capacity, and critical steps are not stopped -to make room for ad hoc work. +You have training, fine-tuning, or inference that cannot vanish because someone else submitted a heavier workload. You want a clear agreement: this stack or +this team gets a dependable slice of capacity, and critical steps are not stopped to make room for ad hoc work. **“We share one pool of GPUs across many teams.”** -You need one place that describes how much capacity exists, who may use it, and -what happens when everyone wants it at once—without maintaining a separate +You need one place that describes how much capacity exists, who may use it, and what happens when everyone wants it at once—without maintaining a separate spreadsheet or manual booking process for every pipeline. **“We paid for the hardware—we should use it when it’s free.”** -When one group is quiet, you want other teams to use spare capacity so machines -do not sit empty. You also want the original team to get their capacity back when -they return, without a long negotiation or a cluster reconfiguration every -time. +When one group is quiet, you want other teams to use spare capacity so machines do not sit empty. You also want the original team to get their capacity back when +they return, without a long negotiation or a cluster reconfiguration every time. **“Engineers describe needs; ops maps them to reality.”** -Pipeline authors say what each step requires (GPUs, memory, and so on). Platform -or DevOps ties those stacks to the right shared capacity. The same pipeline code +Pipeline authors say what each step requires (GPUs, memory, and so on). Platform or DevOps ties those stacks to the right shared capacity. The same pipeline code can run in different environment or stages without hard-coding cluster details. **“We need to ration more than just GPUs.”** -Alongside standard compute, you may need to track things like licenses, special -hardware, or how many pipeline steps may run at once. Pools let you treat those -as countable resources under the same workspace-level model, as long as your -organization agrees on names and units. - -None of this replaces your orchestrator or cloud provider—it coordinates -demand so teams see fair queuing, optional sharing of idle capacity, and explicit -rules for critical versus best-effort work. When you are ready for how ZenML -models that behavior, continue with [Introduction to capacity management](#introduction-to-capacity-management) +Alongside standard compute, you may need to track things like licenses, special hardware, or how many pipeline steps may run at once. Pools let you treat those +as countable resources under the same workspace-level model, as long as your organization agrees on names and units. + +None of this replaces your orchestrator or cloud provider—it coordinates demand so teams see fair queuing, optional sharing of idle capacity, and explicit +rules for critical versus best-effort work. When you are ready for how ZenML models that behavior, continue with [Introduction to capacity management](#introduction-to-capacity-management) and [Core concepts](#core-concepts). ## Introduction to capacity management -ZenML Pro resource pooling separates three concerns: what capacity exists, what -each pipeline step asks for, and which stack components may draw from which pools under +ZenML Pro resource pooling separates three concerns: what capacity exists, what each pipeline step asks for, and which stack components may draw from which pools under which rules. The subsections below describe the strategy; [Core concepts](#core-concepts) then defines terms and mechanics precisely. ### Supply: resource pools -On one side, **resource pools** describe resources that are available and -consumable when pipeline steps run. They often include standard compute—GPUs, -CPUs, and RAM—that orchestrators understand and that map to real infrastructure, -but pools are not limited to that. Teams can define **custom resource keys** as a +On one side, **resource pools** describe resources that are available and consumable when pipeline steps run. They often include standard compute—GPUs, +CPUs, and RAM—that orchestrators understand and that map to real infrastructure, but pools are not limited to that. Teams can define **custom resource keys** as a convention between whoever operates the platform and whoever writes pipelines: -capabilities, access to external services or tools, licenses, or any other -scarce thing you want to schedule in integer units. The same resource key names +capabilities, access to external services or tools, licenses, or any other scarce thing you want to schedule in integer units. The same resource key names must appear in pool capacity and in step requests for the convention to work. This is the contract between the platform and the users. -The number of **parallel step runs** is also modeled as a resource today so -concurrency can be capped alongside GPU or CPU demand. -A pool can hold **multiple resource types at once** (for example GPUs and a -`step_run` slot). +The number of **parallel step runs** is also modeled as a resource today so concurrency can be capped alongside GPU or CPU demand. +A pool can hold **multiple resource types at once** (for example GPUs and a `step_run` slot). -**Who defines pools:** the role that owns infrastructure access—IT ops, -platform engineering, DevOps, or similar — defines and maintains resource pools -at workspace scope so every team in that workspace shares the same catalog -of named capacity. +**Who defines pools:** the role that owns infrastructure access—IT ops, platform engineering, DevOps, or similar — defines and maintains resource pools +at workspace scope so every team in that workspace shares the same catalog of named capacity. ### Demand: `ResourceSettings` on steps -On the other side, **pipeline steps** declare what they need through -`ResourceSettings`. That is typically the ML engineer, AI engineer, or data -engineer annotating each step with GPUs, CPUs, memory, and optionally the same -custom keys the admins put in pools. Authors can align requests with published -pool capacity and follow whatever naming convention the platform agreed on for -non-standard resources. - -Steps also declare whether or not they are **preemptible**. Preemption is the -ability to interrupt a running step run before it completes, to free up resources for -other steps demanding them with a higher priority. Steps that are preemptible -are not guaranteed to complete if they go outside of their reserved capacity. -They may be interrupted, re-added to the queue and restarted again later, or -even cancelled if they are not configured to allow retries. - -That is the product-level tradeoff: **preemptible** steps may access more of the -pool — including capacity that others are not using right now — but another -workload can force them off the machine if priorities and policies require it, -so they can fail early when contention is high. **Non-preemptible** steps opt -into a smaller, dependable slice: they only consume what resources are reserved -for them, so they are not evicted for pool reasons, at the cost of not using the +On the other side, **pipeline steps** declare what they need through `ResourceSettings`. That is typically the ML engineer, AI engineer, or data +engineer annotating each step with GPUs, CPUs, memory, and optionally the same custom keys the admins put in pools. Authors can align requests with published +pool capacity and follow whatever naming convention the platform agreed on for non-standard resources. + +Steps also declare whether or not they are **preemptible**. Preemption is the ability to interrupt a running step run before it completes, to free up resources for +other steps demanding them with a higher priority. Steps that are preemptible are not guaranteed to complete if they go outside of their reserved capacity. +They may be interrupted, re-added to the queue and restarted again later, or even cancelled if they are not configured to allow retries. + +That is the product-level tradeoff: **preemptible** steps may access more of the pool — including capacity that others are not using right now — but another +workload can force them off the machine if priorities and policies require it, so they can fail early when contention is high. **Non-preemptible** steps opt +into a smaller, dependable slice: they only consume what resources are reserved for them, so they are not evicted for pool reasons, at the cost of not using the “burst” capacity above that reservation. ### The link: subject policies -**Subject policies** connect pools to execution. A policy binds a workload bearing **stack -component** — today an orchestrator or step operator — to a pool and states what -subset of that pool’s resources that component may use. Steps stay decoupled -from a specific pool name: resolution uses the component in the active stack, -so the same step definition can behave differently in different environments -without code changes. +**Subject policies** connect pools to execution. A policy binds a workload bearing **stack component** — today an orchestrator or step operator — to a pool and states what +subset of that pool’s resources that component may use. Steps stay decoupled from a specific pool name: resolution uses the component in the active stack, +so the same step definition can behave differently in different environments without code changes. -Policies are usually owned by the same platform or admin function, but ML -teams can share ownership where it makes sense. Each policy carries **priority**, +Policies are usually owned by the same platform or admin function, but ML teams can share ownership where it makes sense. Each policy carries **priority**, **reserved** amounts per key, and optionally **limits**: -* **Reservation** answers “what pool share is accounted exclusively to this component” - and is the only capacity that **non-preemptible** steps may rely on: they +* **Reservation** answers “what pool share is accounted exclusively to this component” and is the only capacity that **non-preemptible** steps may rely on: they cannot use the gap between reserved and limit. -* **Limits** (and the pool’s own maximum) bound how many resources **preemptible** work - can take when spare capacity exists; without a limit, the effective ceiling is +* **Limits** (and the pool’s own maximum) bound how many resources **preemptible** work can take when spare capacity exists; without a limit, the effective ceiling is what the pool still has free. A higher limit does not raise the ceiling for non-preemptible requests; you need to raise **reserved** if those jobs need more per step. -When thinking about subject policies, it's helpful to understand that multiple -steps in the same run or even in different runs will share the resources defined +When thinking about subject policies, it's helpful to understand that multiple steps in the same run or even in different runs will share the resources defined in the same subject policy if they run on the same stack component. So they will compete for the same reserved resources. If they go outside of their reserved capacity, they will also compete with other stack components that access the same pool. -Together, pools plus policies plus step annotations implement a **shared, -prioritized, optionally elastic** scheduling story: strict guarantees where +Together, pools plus policies plus step annotations implement a **shared, prioritized, optionally elastic** scheduling story: strict guarantees where needed, elastic sharing where teams accept preemption risk. ### What this looks like: three surfaces @@ -154,11 +115,9 @@ zenml resource-pool create datacenter-one \ ``` **2 — Policy (wiring a stack to a pool).** -They attach a **subject policy** so a specific stack component knows which pool -to use and what slice it may claim. If your pipeline runs on a stack called **prod-stack**, its orchestrator (or step operator) is the component named in +They attach a **subject policy** so a specific stack component knows which pool to use and what slice it may claim. If your pipeline runs on a stack called **prod-stack**, its orchestrator (or step operator) is the component named in the policy: “prod-stack’s orchestrator may pull from **datacenter-one**, with -*this much* reserved and *this much* limit,” and a priority -versus other stacks. +*this much* reserved and *this much* limit,” and a priority versus other stacks. ```shell zenml resource-pool attach-policy datacenter-one prod-stack \ @@ -194,19 +153,15 @@ if __name__ == "__main__": my_pipeline(input="Hello, World!") ``` -When they launch a **dynamic** run on **prod-stack**, ZenML turns that into a **resource -request**: three GPUs, matched against **datacenter-one** through prod-stack’s policy. -If three are free, the step proceeds; if not, it **waits** in line; if the ask -breaks the rules (too many GPUs, non-preemptible without reservation), it +When they launch a **dynamic** run on **prod-stack**, ZenML turns that into a **resource request**: three GPUs, matched against **datacenter-one** through prod-stack’s policy. +If three are free, the step proceeds; if not, it **waits** in line; if the ask breaks the rules (too many GPUs, non-preemptible without reservation), it **fails fast** with a clear status. Run and step views show **queued / allocated / rejected** so operators can compare ZenML to the real cluster. -The subsections explain how queues, borrowing, and preemption implement -that model in ZenML Pro. +The subsections explain how queues, borrowing, and preemption implement that model in ZenML Pro. ## See also * [Workspaces](./workspaces.md) — pools are scoped to the workspace. -* [Teams](./teams.md) — organizational context for who owns which stacks and - policies. +* [Teams](./teams.md) — organizational context for who owns which stacks and policies. * ZenML OSS: [step and pipeline configuration](https://docs.zenml.io/how-to/steps-pipelines/configuration). diff --git a/docs/book/getting-started/zenml-pro/roles.md b/docs/book/getting-started/zenml-pro/roles.md index 6b8f5fd7f06..e5ba9cc77b3 100644 --- a/docs/book/getting-started/zenml-pro/roles.md +++ b/docs/book/getting-started/zenml-pro/roles.md @@ -64,8 +64,7 @@ To assign organization roles: {% stepper %} {% step %} -Navigate to the **Organization** **Settings** page -{% endstep %} +Navigate to the **Organization** **Settings** page {% endstep %} {% step %} Click on the **Members** tab. Here you can update roles for existing members. diff --git a/docs/book/getting-started/zenml-pro/scenarios.md b/docs/book/getting-started/zenml-pro/scenarios.md index c78195ddafa..dae46f5cd43 100644 --- a/docs/book/getting-started/zenml-pro/scenarios.md +++ b/docs/book/getting-started/zenml-pro/scenarios.md @@ -22,8 +22,7 @@ ZenML Pro offers three flexible deployment options to match your organization's | **Best For** | Teams wanting minimal infrastructure overhead and fastest time-to-value | Organizations with security/compliance requirements but wanting simplified user management | Organizations requiring complete data isolation and on-premises control | {% hint style="info" %} -In all of these cases the client SDK that you pip install into your development environment is the same one found here: https://pypi.org/project/zenml/ -{% endhint %} +In all of these cases the client SDK that you pip install into your development environment is the same one found here: https://pypi.org/project/zenml/ {% endhint %} ## Which Scenario is Right for You? diff --git a/docs/book/getting-started/zenml-pro/sso.md b/docs/book/getting-started/zenml-pro/sso.md index a87a3190f8b..9c1774d9a23 100644 --- a/docs/book/getting-started/zenml-pro/sso.md +++ b/docs/book/getting-started/zenml-pro/sso.md @@ -342,8 +342,7 @@ We provide a complete example synchronization script that demonstrates how to: - Use the ZenML Pro API correctly for all operations {% file src="scripts/sync_sso_users.py" %} -Download the example SSO user synchronization script -{% endfile %} +Download the example SSO user synchronization script {% endfile %} **Script highlights:** diff --git a/docs/book/getting-started/zenml-pro/teams.md b/docs/book/getting-started/zenml-pro/teams.md index 80aadbdb0dd..c822c5cb731 100644 --- a/docs/book/getting-started/zenml-pro/teams.md +++ b/docs/book/getting-started/zenml-pro/teams.md @@ -30,8 +30,7 @@ To create a team: Click on the **Settings** tab from your **Organization** page. -
-{% endstep %} +
{% endstep %} {% step %} **Click on the Teams tab** @@ -62,16 +61,13 @@ To add users to an existing team: {% stepper %} {% step %} -Go to the **Teams** tab in **Organization** settings -{% endstep %} +Go to the **Teams** tab in **Organization** settings {% endstep %} {% step %} -Select the team you want to modify -{% endstep %} +Select the team you want to modify {% endstep %} {% step %} -Click on **Add Members** -{% endstep %} +Click on **Add Members** {% endstep %} {% step %} Choose users from your organization to add to the team @@ -86,16 +82,14 @@ Teams can be assigned to workspaces just like individual users. To add a team to {% stepper %} {% step %} -Go to the **Workspace Settings** page -{% endstep %} +Go to the **Workspace Settings** page {% endstep %} {% step %} Click on **Members** tab and click on the **Teams** tab. {% endstep %} {% step %} -Select **Add Team** -{% endstep %} +Select **Add Team** {% endstep %} {% step %} Choose the team and assign a role diff --git a/docs/book/getting-started/zenml-pro/triggers.md b/docs/book/getting-started/zenml-pro/triggers.md index a23e043c5e7..4c758e25641 100644 --- a/docs/book/getting-started/zenml-pro/triggers.md +++ b/docs/book/getting-started/zenml-pro/triggers.md @@ -9,23 +9,18 @@ icon: server Triggers are part of ZenML's paid features. For details on availability and supported plans, visit the [pricing page](https://www.zenml.io/pricing) {% endhint %} -In the [snapshots](./snapshots.md) section, you learned how to prepare snapshots and execute them on demand via -the dashboard, CLI, or SDK. In many cases, however, pipelines need to run automatically - either on a schedule or in +In the [snapshots](./snapshots.md) section, you learned how to prepare snapshots and execute them on demand via the dashboard, CLI, or SDK. In many cases, however, pipelines need to run automatically - either on a schedule or in response to an event. -Triggers enable this behavior. A trigger is a configuration that defines one or more conditions under -which a pipeline is automatically started. +Triggers enable this behavior. A trigger is a configuration that defines one or more conditions under which a pipeline is automatically started. ## Schedule Triggers -*Schedule Triggers* allow pipelines to run automatically based on time-based rules, such as fixed intervals -or cron expressions. They are ideal for recurring workflows, on a predictable timeline, like daily retraining, +*Schedule Triggers* allow pipelines to run automatically based on time-based rules, such as fixed intervals or cron expressions. They are ideal for recurring workflows, on a predictable timeline, like daily retraining, batch processing, or periodic data ingestion. -When defining a scheduled trigger, you can configure both when and how your pipeline runs. Choose between -one-off executions, interval-based schedules, or cron expressions for fine-grained timing control. -Additional options, such as time boundaries, concurrency limits, and activation settings, let you -tailor the trigger to your workflow requirements. +When defining a scheduled trigger, you can configure both when and how your pipeline runs. Choose between one-off executions, interval-based schedules, or cron expressions for fine-grained timing control. +Additional options, such as time boundaries, concurrency limits, and activation settings, let you tailor the trigger to your workflow requirements. | Attribute | Description | Notes | |---------------------|-------------------------------------------------------------------------|------------------------------| @@ -45,8 +40,7 @@ Let's start by creating a schedule. We can do so, via the SDK or the CLI. Via the SDK: -~~~python -from zenml.client import Client +~~~python from zenml.client import Client from zenml.enums import TriggerRunConcurrency client = Client() @@ -61,22 +55,18 @@ daily_schedule = client.create_schedule_trigger( Via the CLI: -~~~bash -zenml trigger schedule create daily-schedule-6-am --cron-expression "0 6 * * *" +~~~bash zenml trigger schedule create daily-schedule-6-am --cron-expression "0 6 * * *" ~~~ ### Attach/Detach schedules and snapshots -So far we have instructed our system with *when* to execute but not *what*. To do so, -we need to *attach* a schedule to a snapshot. +So far we have instructed our system with *when* to execute but not *what*. To do so, we need to *attach* a schedule to a snapshot. -For the CLI, commands that take a trigger or snapshot accept its name or ID -(exact match, not a prefix). Positional order is always trigger first, then snapshot. +For the CLI, commands that take a trigger or snapshot accept its name or ID (exact match, not a prefix). Positional order is always trigger first, then snapshot. Via the SDK: -~~~python -from uuid import UUID +~~~python from uuid import UUID from zenml.client import Client client = Client() @@ -88,19 +78,15 @@ client.attach_trigger_to_snapshot( Via the CLI: -~~~bash -zenml trigger schedule attach "" "" +~~~bash zenml trigger schedule attach "" "" ~~~ -Users can provide a configuration object to define the parameters of pipeline runs -triggered from this attachment. +Users can provide a configuration object to define the parameters of pipeline runs triggered from this attachment. Via the SDK: -~~~python -from uuid import UUID -from zenml.client import Client -from zenml.config.pipeline_run_configuration import PipelineRunConfiguration +~~~python from uuid import UUID +from zenml.client import Client from zenml.config.pipeline_run_configuration import PipelineRunConfiguration client = Client() client.attach_trigger_to_snapshot( @@ -115,18 +101,15 @@ client.attach_trigger_to_snapshot( Via the CLI (using a `PipelineRunConfiguration` YAML file): -~~~bash -zenml trigger schedule attach "" "" \ +~~~bash zenml trigger schedule attach "" "" \ --config=.yml ~~~ -To stop a trigger from launching runs for a specific snapshot, you can *detach* the -trigger from that snapshot. +To stop a trigger from launching runs for a specific snapshot, you can *detach* the trigger from that snapshot. Via the SDK: -~~~python -from uuid import UUID +~~~python from uuid import UUID from zenml.client import Client client = Client() @@ -138,12 +121,10 @@ client.detach_trigger_from_snapshot( Via the CLI: -~~~bash -zenml trigger schedule detach "" "" +~~~bash zenml trigger schedule detach "" "" ~~~ -The ability to detach and attach snapshots is particularly useful as pipelines evolve. When a new pipeline -version becomes available, you can update the schedule to use it by detaching the previous +The ability to detach and attach snapshots is particularly useful as pipelines evolve. When a new pipeline version becomes available, you can update the schedule to use it by detaching the previous snapshot and attaching the new one. {% hint style="warning" %} @@ -159,8 +140,7 @@ You can update a schedule's configuration at any point. In the example, we will Via the SDK: -~~~python -from zenml.client import Client +~~~python from zenml.client import Client client = Client() client.update_schedule_trigger( @@ -172,15 +152,13 @@ client.update_schedule_trigger( Via the CLI: -~~~bash -zenml trigger schedule update "daily-schedule-6-am" --active=false \ +~~~bash zenml trigger schedule update "daily-schedule-6-am" --active=false \ --name="daily-schedule-6-am-(dont-touch)" ~~~ ### View Schedules -Triggers are a first-level citizen of the ZenML platform. You can view detailed information in the -dashboard as well as via the SDK and CLI. +Triggers are a first-level citizen of the ZenML platform. You can view detailed information in the dashboard as well as via the SDK and CLI. Via the dashboard: @@ -199,32 +177,23 @@ Or its attached snapshots and executed pipeline runs: ![Image Showing Schedule Runs](../../.gitbook/assets/schedule_runs_dash.png) For each trigger–snapshot attachment, ZenML can persist the **dispatch state**. -This is the small status record that answers, "what happened the last time this -trigger tried to launch this snapshot?" The status can be: +This is the small status record that answers, "what happened the last time this trigger tried to launch this snapshot?" The status can be: - `SUCCESS`: a run was launched successfully. -- `SKIPPED_CONCURRENCY`: a run was skipped because the trigger's concurrency - rule said not to start another one yet. -- `SKIPPED_MAX_RUNS`: a run was skipped because the configured run limit for - this trigger-snapshot attachment has already been reached. -- `SKIPPED_TRIGGER_CYCLE`: a platform event trigger dispatch was skipped - because it would revisit a pipeline already present in the trigger chain. +- `SKIPPED_CONCURRENCY`: a run was skipped because the trigger's concurrency rule said not to start another one yet. +- `SKIPPED_MAX_RUNS`: a run was skipped because the configured run limit for this trigger-snapshot attachment has already been reached. +- `SKIPPED_TRIGGER_CYCLE`: a platform event trigger dispatch was skipped because it would revisit a pipeline already present in the trigger chain. - `ERROR`: ZenML tried to dispatch the run, but something failed. -When the last status is `ERROR`, the dispatch state can also include the last -error message, error type, severity, stack trace, first/last error timestamps, -and an error count. After you fix the underlying problem, you can acknowledge -the error and clear the stored dispatch state (see the next section). In the -normal failure case, clearing errors removes the `ERROR` status record for that -trigger-snapshot attachment, so the next dispatch can start from a clean state. -Use `get_schedule_trigger` with `trigger_name_id_or_prefix` to load a schedule -by name, full ID, or ID prefix; set `allow_name_prefix_match=False` if you need +When the last status is `ERROR`, the dispatch state can also include the last error message, error type, severity, stack trace, first/last error timestamps, +and an error count. After you fix the underlying problem, you can acknowledge the error and clear the stored dispatch state (see the next section). In the +normal failure case, clearing errors removes the `ERROR` status record for that trigger-snapshot attachment, so the next dispatch can start from a clean state. +Use `get_schedule_trigger` with `trigger_name_id_or_prefix` to load a schedule by name, full ID, or ID prefix; set `allow_name_prefix_match=False` if you need an exact name match. Via the SDK: -~~~python -from zenml.client import Client +~~~python from zenml.client import Client client = Client() @@ -242,22 +211,19 @@ for snapshot in schedule.snapshots: # iterate a schedule's attached snapshots Via the CLI: -~~~bash -zenml trigger schedule list --active=true +~~~bash zenml trigger schedule list --active=true ~~~ ### Stopping Criteria -You can control when a schedule stops by limiting it in time or by number of runs. This helps avoid unintended -long-running schedules and gives you tighter control over resource usage. +You can control when a schedule stops by limiting it in time or by number of runs. This helps avoid unintended long-running schedules and gives you tighter control over resource usage. #### `end_time` Defines the point in time after which the schedule will no longer trigger runs. The schedule remains visible but inactive for future executions. -~~~python -from datetime import datetime, timedelta +~~~python from datetime import datetime, timedelta from zenml.client import Client client = Client() @@ -271,17 +237,14 @@ client.create_schedule_trigger( #### `max_runs` -Limits how many times a schedule can trigger a pipeline. Once the limit is -reached for a trigger-snapshot attachment, no further runs are scheduled for -that attachment. Later dispatch attempts can show up as `SKIPPED_MAX_RUNS` in -the dispatch state. This is not an error that needs to be cleared; it is ZenML +Limits how many times a schedule can trigger a pipeline. Once the limit is reached for a trigger-snapshot attachment, no further runs are scheduled for +that attachment. Later dispatch attempts can show up as `SKIPPED_MAX_RUNS` in the dispatch state. This is not an error that needs to be cleared; it is ZenML saying, "the run limit you configured has been reached." Note that this limit applies per attached snapshot, for example, with a limit of 2 and two attached snapshots, you will see a total of 4 runs. -~~~python -from zenml.client import Client +~~~python from zenml.client import Client client = Client() @@ -298,13 +261,11 @@ If both are set, the schedule stops when the first condition is reached (time or #### Clear dispatch errors -To clear stored dispatch error details after the issue is resolved, use the SDK -or the `clear-errors` command under `zenml trigger schedule`. With no snapshot argument, errors are cleared for **all** snapshots attached to that trigger. +To clear stored dispatch error details after the issue is resolved, use the SDK or the `clear-errors` command under `zenml trigger schedule`. With no snapshot argument, errors are cleared for **all** snapshots attached to that trigger. Via the SDK: -~~~python -from uuid import UUID +~~~python from uuid import UUID from zenml.client import Client client = Client() @@ -319,11 +280,9 @@ client.clear_trigger_dispatch_error( ) ~~~ -To clear the error only for one attached snapshot, pass that snapshot’s UUID -(in addition to the `client` and `trigger` values from the example above): +To clear the error only for one attached snapshot, pass that snapshot’s UUID (in addition to the `client` and `trigger` values from the example above): -~~~python -from uuid import UUID +~~~python from uuid import UUID client.clear_trigger_dispatch_error( trigger_id=trigger.id, @@ -333,32 +292,27 @@ client.clear_trigger_dispatch_error( Via the CLI: -~~~bash -zenml trigger schedule clear-errors "my-schedule" +~~~bash zenml trigger schedule clear-errors "my-schedule" zenml trigger schedule clear-errors "my-schedule" "my-snapshot" ~~~ ### Delete schedules -Triggers in ZenML are archivable objects. When a Trigger is archived (soft-deleted), it is deactivated and can no -longer be used, but it remains in the system to preserve references for visibility and debugging. +Triggers in ZenML are archivable objects. When a Trigger is archived (soft-deleted), it is deactivated and can no longer be used, but it remains in the system to preserve references for visibility and debugging. -Archiving (soft deletion) is the default deletion mode. Triggers can also be permanently deleted. Neither -operation can be reversed. +Archiving (soft deletion) is the default deletion mode. Triggers can also be permanently deleted. Neither operation can be reversed. Via the dashboard: ![Image Showing Schedule Deletion](../../.gitbook/assets/delete_schedule_soft.png) -You can view archived schedules by setting the `Display archived` filter, where -you can also hard delete them. +You can view archived schedules by setting the `Display archived` filter, where you can also hard delete them. ![Image Showing Schedule Hard Deletion](../../.gitbook/assets/delete_schedule_hard.png) Via the SDK: -~~~python -from uuid import UUID +~~~python from uuid import UUID from zenml.client import Client client = Client() @@ -370,21 +324,16 @@ client.delete_trigger( Via the CLI: -Default behavior is **soft** deletion (the trigger is archived). Pass -`--hard` to remove the trigger and its associated references permanently. To -operate on an **archived** trigger, add `--archived` (for example, to hard -delete a trigger that is already archived). +Default behavior is **soft** deletion (the trigger is archived). Pass `--hard` to remove the trigger and its associated references permanently. To +operate on an **archived** trigger, add `--archived` (for example, to hard delete a trigger that is already archived). -~~~bash -zenml trigger schedule delete "my-schedule" -zenml trigger schedule delete "my-schedule" --hard -zenml trigger schedule delete "my-old-schedule" --archived --hard +~~~bash zenml trigger schedule delete "my-schedule" +zenml trigger schedule delete "my-schedule" --hard zenml trigger schedule delete "my-old-schedule" --archived --hard ~~~ ## Triggers vs OSS schedules -ZenML provides [scheduling](../../how-to/steps-pipelines/scheduling.md) as an open-source feature. This section -outlines the differences between open-source schedules and schedule-based Triggers, and explains why Triggers are +ZenML provides [scheduling](../../how-to/steps-pipelines/scheduling.md) as an open-source feature. This section outlines the differences between open-source schedules and schedule-based Triggers, and explains why Triggers are better suited for production workloads: * Lifecycle management @@ -405,17 +354,13 @@ better suited for production workloads: ## Platform Event Triggers -*Platform Event Triggers* extend ZenML’s trigger system by enabling pipelines to run automatically in response -to events occurring within the ZenML platform itself. Instead of relying on time-based schedules, these -triggers allow users to define downstream pipeline executions that react to lifecycle events, such as -the completion of another pipeline. This makes it easy to build event-driven workflows where pipelines are +*Platform Event Triggers* extend ZenML’s trigger system by enabling pipelines to run automatically in response to events occurring within the ZenML platform itself. Instead of relying on time-based schedules, these +triggers allow users to define downstream pipeline executions that react to lifecycle events, such as the completion of another pipeline. This makes it easy to build event-driven workflows where pipelines are seamlessly chained based on platform activity. -When defining a platform event trigger, you configure what resource to listen to and which events should -initiate a pipeline run. The source_type and source_id identify the ZenML entity you want to react +When defining a platform event trigger, you configure what resource to listen to and which events should initiate a pipeline run. The source_type and source_id identify the ZenML entity you want to react to (e.g., a specific pipeline), while target_events define the events of interest (e.g., completed or failed). -Additional options, such as activation state and concurrency behavior, allow you to control how the -trigger operates within your workflow. +Additional options, such as activation state and concurrency behavior, allow you to control how the trigger operates within your workflow. | Attribute | Description | Notes | |---------------|-----------------------------------------------------------|------------------------------------| @@ -427,8 +372,7 @@ trigger operates within your workflow. | concurrency | Option to control how concurrent runs should be handled | Skip is the default option | You can manage platform event triggers using the same set of commands (create, update, attach, detach, list, delete, clear-errors) -as for schedule triggers. While some parameters and responses differ slightly, additional utilities are -available to help you work more effectively with this trigger type. +as for schedule triggers. While some parameters and responses differ slightly, additional utilities are available to help you work more effectively with this trigger type. Supported target events depend on the source type: @@ -438,17 +382,14 @@ Supported target events depend on the source type: | `pipeline_run` | React to one specific pipeline run | `completed`, `failed` | | `pipeline_snapshot` | React to runs of a pipeline snapshot | `run_completed`, `run_failed` | -That distinction is easy to miss. If the source is a pipeline, the event name -includes the word `run` because the pipeline itself is not what completes; one -of its runs does. Pipeline snapshot events follow the same naming pattern. If -the source is already a pipeline run, the event is simply `completed` or +That distinction is easy to miss. If the source is a pipeline, the event name includes the word `run` because the pipeline itself is not what completes; one +of its runs does. Pipeline snapshot events follow the same naming pattern. If the source is already a pipeline run, the event is simply `completed` or `failed`. ### Create Platform Event Trigger -Let's start by creating a Platform Event Trigger. In this example, we want to react to the successful -completion of a specific pipeline. +Let's start by creating a Platform Event Trigger. In this example, we want to react to the successful completion of a specific pipeline. Via the SDK: @@ -468,8 +409,7 @@ trigger = create_platform_event_trigger( Via the CLI: -The SDK provides helpful overloads that guide you toward valid configurations by suggesting the supported target -events for a given source type as you type. To improve discoverability in the CLI, an additional helper +The SDK provides helpful overloads that guide you toward valid configurations by suggesting the supported target events for a given source type as you type. To improve discoverability in the CLI, an additional helper command is available: ```bash @@ -531,8 +471,7 @@ trigger = Client().get_platform_event_trigger( triggers = Client().list_platform_event_triggers() ``` -To attach, detach, and delete triggers via the SDK, the methods are shared across -all trigger types (IDs are UUIDs): +To attach, detach, and delete triggers via the SDK, the methods are shared across all trigger types (IDs are UUIDs): ```python from uuid import UUID @@ -552,8 +491,7 @@ c.delete_trigger(trigger_id=UUID(""), soft=True) ### Upstream information -When a downstream pipeline is executed, it can be useful to access information about the upstream run -that triggered it. The following example shows how to retrieve this information within a running pipeline step: +When a downstream pipeline is executed, it can be useful to access information about the upstream run that triggered it. The following example shows how to retrieve this information within a running pipeline step: ```python @@ -570,25 +508,19 @@ def my_step(): ### Chaining pipelines with triggers Platform Event Triggers can be used to build simple multi-pipeline workflows by chaining pipelines together. -For example, you can configure a trigger so that when Pipeline A completes, it starts Pipeline B, which in -turn can trigger Pipeline C. This enables lightweight orchestration patterns directly within ZenML, allowing you +For example, you can configure a trigger so that when Pipeline A completes, it starts Pipeline B, which in turn can trigger Pipeline C. This enables lightweight orchestration patterns directly within ZenML, allowing you to break down complex workflows into smaller, reusable pipeline components that execute in sequence based on platform events. {% hint style="warning" %} -ZenML detects trigger loops at the **pipeline level** while dispatching platform -event triggers. Snapshots, runs, trigger IDs, and trigger configuration do not -create separate nodes in the cycle: two different snapshots of the same -pipeline still refer to the same pipeline in the trigger chain. - -If a downstream snapshot would revisit a pipeline that is already in the -current chain, ZenML skips only that trigger-snapshot dispatch and records -`SKIPPED_TRIGGER_CYCLE`. Other non-cyclic snapshots attached to the same trigger -continue to dispatch normally. This protection does not disable or modify the +ZenML detects trigger loops at the **pipeline level** while dispatching platform event triggers. Snapshots, runs, trigger IDs, and trigger configuration do not +create separate nodes in the cycle: two different snapshots of the same pipeline still refer to the same pipeline in the trigger chain. + +If a downstream snapshot would revisit a pipeline that is already in the current chain, ZenML skips only that trigger-snapshot dispatch and records +`SKIPPED_TRIGGER_CYCLE`. Other non-cyclic snapshots attached to the same trigger continue to dispatch normally. This protection does not disable or modify the trigger configuration, and it does not cancel runs that have already started. {% endhint %} -Dispatch states are stored per attached snapshot, and -`cycle_pipeline_ids` contains the ordered closed cycle (the first and last IDs +Dispatch states are stored per attached snapshot, and `cycle_pipeline_ids` contains the ordered closed cycle (the first and last IDs are the pipeline that closes the loop): ```python diff --git a/docs/book/getting-started/zenml-pro/upgrades-control-plane.md b/docs/book/getting-started/zenml-pro/upgrades-control-plane.md index 3c03db22f0f..ca5c24a105b 100644 --- a/docs/book/getting-started/zenml-pro/upgrades-control-plane.md +++ b/docs/book/getting-started/zenml-pro/upgrades-control-plane.md @@ -40,8 +40,7 @@ For air-gapped environments: To upgrade the Control Plane in a self-hosted deployment: -1. **Update Helm Values:** - Change the Control Plane version in your `values.yaml` file to reference the new image tag. +1. **Update Helm Values:** Change the Control Plane version in your `values.yaml` file to reference the new image tag. 2. **Apply the Upgrade:** @@ -63,8 +62,7 @@ To upgrade the Control Plane in a self-hosted deployment: --values current-values.yaml ``` -3. **Monitor the Upgrade:** - Watch the logs and pod statuses to verify a healthy rollout: +3. **Monitor the Upgrade:** Watch the logs and pod statuses to verify a healthy rollout: ```bash kubectl -n get pods kubectl -n logs diff --git a/docs/book/getting-started/zenml-pro/upgrades-workspace-server.md b/docs/book/getting-started/zenml-pro/upgrades-workspace-server.md index 44eded77d9a..430eb8e3142 100644 --- a/docs/book/getting-started/zenml-pro/upgrades-workspace-server.md +++ b/docs/book/getting-started/zenml-pro/upgrades-workspace-server.md @@ -34,32 +34,26 @@ This provides a safe and reliable process to keep your workspaces up to date wit To upgrade workspace servers in a hybrid deployment: -1. **Update Helm Values:** - Change the Workspace Server version in your `values.yaml` file to reference the new image tag (the version you want to upgrade to). +1. **Update Helm Values:** Change the Workspace Server version in your `values.yaml` file to reference the new image tag (the version you want to upgrade to). -2. **Apply the Upgrade:** - Re-apply the Helm chart to perform the upgrade: +2. **Apply the Upgrade:** Re-apply the Helm chart to perform the upgrade: ```bash helm upgrade zenml/zenml \ --namespace \ --values values.yaml ``` -3. **Automatic Backup:** - As part of the upgrade process, the system takes a database backup automatically before proceeding. This ensures you can safely roll back if anything goes wrong. +3. **Automatic Backup:** As part of the upgrade process, the system takes a database backup automatically before proceeding. This ensures you can safely roll back if anything goes wrong. -4. **Monitor the Upgrade:** - Watch the logs and pod statuses to verify a healthy rollout: +4. **Monitor the Upgrade:** Watch the logs and pod statuses to verify a healthy rollout: ```bash kubectl -n get pods kubectl -n logs ``` -5. **Rollback on Failure:** - If the upgrade fails for any reason, the system will automatically roll back to the previous workspace server version using the backup. No manual intervention is required. +5. **Rollback on Failure:** If the upgrade fails for any reason, the system will automatically roll back to the previous workspace server version using the backup. No manual intervention is required. -6. **Zero Downtime:** - Workspace upgrades are orchestrated to be highly available—users should not experience downtime during the upgrade process. +6. **Zero Downtime:** Workspace upgrades are orchestrated to be highly available—users should not experience downtime during the upgrade process. {% hint style="info" %} **Workload Manager Updates:** When upgrading, check the [release notes](https://docs.zenml.io/changelog/server-sdk) for any changes to workload manager configuration. If you have configured a workload manager, you may need to update environment variables in your Helm values. See [Workspace Server Configuration](deploy-workspace-snapshots.md) for the full configuration reference. diff --git a/docs/book/getting-started/zenml-pro/workspaces.md b/docs/book/getting-started/zenml-pro/workspaces.md index d315328087c..a72397fb716 100644 --- a/docs/book/getting-started/zenml-pro/workspaces.md +++ b/docs/book/getting-started/zenml-pro/workspaces.md @@ -56,14 +56,12 @@ A workspace is a crucial part of your Organization and serves as a container for {% stepper %} {% step %} -**Go to your organization page** -{% endstep %} +**Go to your organization page** {% endstep %} {% step %} **Click on the "New Workspace" button** -

Image showing the "New Workspace" button

-{% endstep %} +

Image showing the "New Workspace" button

{% endstep %} {% step %} **Add a name and id** @@ -74,8 +72,7 @@ Give your workspace a name, an id, and click on the "**Create Workspace**" butto **Important**: The workspace ID must be globally unique across all ZenML instances and cannot be changed after creation. Choose carefully as this permanent identifier will be used in all future API calls and references. {% endhint %} -
-{% endstep %} +
{% endstep %} {% step %} **Your workspace is ready!** diff --git a/docs/book/how-to/artifacts/materializers.md b/docs/book/how-to/artifacts/materializers.md index e0e7c7d5ac8..d299ef5083c 100644 --- a/docs/book/how-to/artifacts/materializers.md +++ b/docs/book/how-to/artifacts/materializers.md @@ -30,16 +30,13 @@ ZenML includes built-in materializers for many common data types: ZenML also provides a CloudpickleMaterializer that can handle any object by saving it with [cloudpickle](https://github.com/cloudpipe/cloudpickle). However, this is not production-ready because the resulting artifacts cannot be loaded when running with a different Python version. For production use, you should implement a custom materializer for your specific data types. {% hint style="info" %} -Pydantic artifacts created by current ZenML versions are stored in -`data_v2.json`. ZenML can still load older Pydantic artifacts stored as -`data.json` by ZenML `<= 0.94.2`, so existing runs remain readable after an -upgrade. +Pydantic artifacts created by current ZenML versions are stored in `data_v2.json`. ZenML can still load older Pydantic artifacts stored as +`data.json` by ZenML `<= 0.94.2`, so existing runs remain readable after an upgrade. {% endhint %} ### Dataclass artifacts -The `DataclassMaterializer` handles JSON-serializable Python dataclasses -without requiring you to write a custom materializer. +The `DataclassMaterializer` handles JSON-serializable Python dataclasses without requiring you to write a custom materializer. ```python from dataclasses import dataclass @@ -56,10 +53,8 @@ def make_config() -> TrainingConfig: return TrainingConfig(learning_rate=0.01, epochs=10) ``` -This works for dataclasses that Pydantic can serialize to JSON. If your -dataclass contains objects such as open file handles, live model objects, -database connections, or other arbitrary Python objects, use a custom -materializer instead. +This works for dataclasses that Pydantic can serialize to JSON. If your dataclass contains objects such as open file handles, live model objects, +database connections, or other arbitrary Python objects, use a custom materializer instead. ### Passing Files and Directories Between Steps diff --git a/docs/book/how-to/containerization/containerization.md b/docs/book/how-to/containerization/containerization.md index 2648b6e8d93..ed3b782312e 100644 --- a/docs/book/how-to/containerization/containerization.md +++ b/docs/book/how-to/containerization/containerization.md @@ -267,8 +267,7 @@ ZenML offers several ways to specify dependencies for your Docker containers: By default, ZenML automatically installs all packages required by your active ZenML stack. {% hint style="warning" %} -In future versions, if none of the `replicate_local_python_environment`, `pyproject_path` or `requirements` attributes on `DockerSettings` are specified, ZenML will try to automatically find a `requirements.txt` and `pyproject.toml` files inside your current [source root](../steps-pipelines/sources.md#source-root) and install packages from the first one it finds. You can disable this behavior by setting `disable_automatic_requirements_detection=True`. If -you already want this automatic detection in current versions of ZenML, set `disable_automatic_requirements_detection=False`. +In future versions, if none of the `replicate_local_python_environment`, `pyproject_path` or `requirements` attributes on `DockerSettings` are specified, ZenML will try to automatically find a `requirements.txt` and `pyproject.toml` files inside your current [source root](../steps-pipelines/sources.md#source-root) and install packages from the first one it finds. You can disable this behavior by setting `disable_automatic_requirements_detection=True`. If you already want this automatic detection in current versions of ZenML, set `disable_automatic_requirements_detection=False`. {% endhint %} 1. **Replicate Local Environment**: @@ -421,8 +420,7 @@ The available package installers are: Full documentation for how `uv` works with PyTorch can be found on the Astral Docs website [here](https://docs.astral.sh/uv/guides/integration/pytorch/). It covers some of the particular gotchas and details you might need to know. {% hint style="info" %} -If you're using `uv` and specify a custom parent image or Dockerfile that does not have an activated virtual environment, you need to pass `python_package_installer_args={"system": None}` in your DockerSettings so that -`uv` installs the packages for the Python system installation. Depending on the parent image, you might also need to include `"break-system-packages": None` in the installer args as well to make it work. +If you're using `uv` and specify a custom parent image or Dockerfile that does not have an activated virtual environment, you need to pass `python_package_installer_args={"system": None}` in your DockerSettings so that `uv` installs the packages for the Python system installation. Depending on the parent image, you might also need to include `"break-system-packages": None` in the installer args as well to make it work. {% endhint %} To speed up repeated image builds, set `python_package_installer_cache_mount` to a BuildKit `--mount` spec. ZenML emits `RUN --mount= ...` on the install steps and drops the default `--no-cache-dir` flag so the mount is actually used. The value is passed through verbatim, so any valid `--mount` spec works (e.g. `type=cache` or `type=bind`). Requires BuildKit to be enabled on the builder. @@ -442,8 +440,7 @@ docker_settings = DockerSettings( ### Using custom python executable -To use a custom python executable, instead of a standard `python` you can use the `ZENML_CONTAINER_PYTHON_EXECUTABLE` environment -variable to control the python executable being used in the entrypoint command, for example `ZENML_CONTAINER_PYTHON_EXECUTABLE=/home/user1/custom-venv/bin/python` can be used. +To use a custom python executable, instead of a standard `python` you can use the `ZENML_CONTAINER_PYTHON_EXECUTABLE` environment variable to control the python executable being used in the entrypoint command, for example `ZENML_CONTAINER_PYTHON_EXECUTABLE=/home/user1/custom-venv/bin/python` can be used. ## Private PyPI Repositories @@ -591,8 +588,7 @@ from zenml.config import DockerSettings docker_settings = DockerSettings(image_tag="1.0.0") ``` -Keep in mind that this will be applied to all images built using the DockerSettings object. If there are multiple -such images, only one of them will keep the tag while the rest will be untagged. +Keep in mind that this will be applied to all images built using the DockerSettings object. If there are multiple such images, only one of them will keep the tag while the rest will be untagged. ### Decoupling Code from Builds diff --git a/docs/book/how-to/dashboard/dashboard-features.md b/docs/book/how-to/dashboard/dashboard-features.md index e4434f4fe3c..859d6f1805d 100644 --- a/docs/book/how-to/dashboard/dashboard-features.md +++ b/docs/book/how-to/dashboard/dashboard-features.md @@ -112,10 +112,8 @@ When building a stack, the dashboard helps you browse available components by ca The dashboard streamlines the process of registering individual components like orchestrators, artifact stores, and container registries. Instead of writing configuration code, you can use form-based interfaces to set up each component. -The UI helps connect components to appropriate service connectors and validates -settings before saving. This visual approach to component management reduces -configuration errors and simplifies the setup process, especially for team -members who may not be familiar with the underlying infrastructure details. +The UI helps connect components to appropriate service connectors and validates settings before saving. This visual approach to component management reduces +configuration errors and simplifies the setup process, especially for team members who may not be familiar with the underlying infrastructure details. ![Component registration](../../.gitbook/assets/dashboard-v2-component-registration.png) diff --git a/docs/book/how-to/deployment/deployment.md b/docs/book/how-to/deployment/deployment.md index 02308f8d55d..3bb17853483 100644 --- a/docs/book/how-to/deployment/deployment.md +++ b/docs/book/how-to/deployment/deployment.md @@ -45,37 +45,27 @@ Pipeline deployments are ideal for scenarios requiring real-time, on-demand exec ## Traditional Model Serving vs. Deployed Pipelines -If you're reaching for tools like Seldon or KServe, consider this: deployed -pipelines give you all the core serving primitives, plus the power of a full +If you're reaching for tools like Seldon or KServe, consider this: deployed pipelines give you all the core serving primitives, plus the power of a full application runtime. -- Equivalent functionality: A pipeline handles the end-to-end inference path - out of the box — request validation, feature pre-processing, model loading +- Equivalent functionality: A pipeline handles the end-to-end inference path out of the box — request validation, feature pre-processing, model loading and inference, post-processing, and response shaping. -- More flexible: Deployed pipelines are unopinionated, so you can layer in - retrieval, guardrails, rules, A/B routing, canary logic, human-in-the-loop, +- More flexible: Deployed pipelines are unopinionated, so you can layer in retrieval, guardrails, rules, A/B routing, canary logic, human-in-the-loop, or any custom orchestration. You're not constrained by a model-server template. -- More customizable: The deployment is a real ASGI app. Tailor endpoints, - authentication, authorization, rate limiting, structured logging, tracing, +- More customizable: The deployment is a real ASGI app. Tailor endpoints, authentication, authorization, rate limiting, structured logging, tracing, correlation IDs, or SSO/OIDC — all with first-class middleware and framework-level hooks. -- More features: Serve single-page apps alongside the API. Ship admin/ops - dashboards, experiment playgrounds, model cards, or customer-facing UIs +- More features: Serve single-page apps alongside the API. Ship admin/ops dashboards, experiment playgrounds, model cards, or customer-facing UIs from the very same deployment for tighter operational feedback loops. This approach aligns better with production realities: inference is rarely -"just call a model." There are policies, data dependencies, and integrations -that need a programmable, evolvable surface. Deployed pipelines give you that -without sacrificing the convenience of a managed deployer and a clean HTTP -contract. +"just call a model." There are policies, data dependencies, and integrations that need a programmable, evolvable surface. Deployed pipelines give you that +without sacrificing the convenience of a managed deployer and a clean HTTP contract. {% hint style="info" %} -Deprecation notice: ZenML is phasing out the Model Deployer stack components -in favor of pipeline deployments. Pipeline deployments are the strategic -direction for real-time serving: they are more dynamic, more extensible, and -offer deeper integration points with your security, observability, and product -requirements. Existing model deployers will continue to function during the -transition period, but new investments will focus on pipeline deployments. +Deprecation notice: ZenML is phasing out the Model Deployer stack components in favor of pipeline deployments. Pipeline deployments are the strategic +direction for real-time serving: they are more dynamic, more extensible, and offer deeper integration points with your security, observability, and product +requirements. Existing model deployers will continue to function during the transition period, but new investments will focus on pipeline deployments. {% endhint %} ## How Deployments Work diff --git a/docs/book/how-to/deployment/deployment_settings.md b/docs/book/how-to/deployment/deployment_settings.md index 1989691a792..82d6cf971ea 100644 --- a/docs/book/how-to/deployment/deployment_settings.md +++ b/docs/book/how-to/deployment/deployment_settings.md @@ -5,50 +5,37 @@ description: Customize the pipeline deployment ASGI application with DeploymentS ## Deployment servers and ASGI apps -ZenML pipeline deployments run an ASGI application under a production-grade -`uvicorn` server. This makes your pipelines callable over HTTP for online -workloads like real-time ML inference, LLM agents/workflows, and even full -web apps co-located with pipelines. +ZenML pipeline deployments run an ASGI application under a production-grade `uvicorn` server. This makes your pipelines callable over HTTP for online +workloads like real-time ML inference, LLM agents/workflows, and even full web apps co-located with pipelines. At runtime, three core components work together: -- the ASGI application: the HTTP surface that exposes endpoints (health, invoke, - metrics, docs) and any custom routes or middleware you configure. This is powered by an ASGI framework like FastAPI, Starlette, Django, Flask, etc. +- the ASGI application: the HTTP surface that exposes endpoints (health, invoke, metrics, docs) and any custom routes or middleware you configure. This is powered by an ASGI framework like FastAPI, Starlette, Django, Flask, etc. - the ASGI application factory (aka the Deployment App Runner): this component is responsible for constructing the ASGI application piece by piece based on the instructions provided by users via runtime configuration. -- the Deployment Service: the component responsible for the business logic that - backs the pipeline deployment and its invocation lifecycle. +- the Deployment Service: the component responsible for the business logic that backs the pipeline deployment and its invocation lifecycle. Both the Deployment App Runner and the Deployment Service are customizable at runtime, through the `DeploymentSettings` configuration mechanism. They can also be extended via inheritance to support different ASGI frameworks or to tweak existing functionality. The `DeploymentSettings` class lets you shape both server behavior and the -ASGI app composition without changing framework code. Typical reasons to -customize include: - -- Tight security posture: CORS controls, strict headers, authentication, - API surface minimization. -- Observability: request/response logging, tracing, metrics, correlation - identifiers. -- Enterprise integration: policy gateways, SSO/OIDC/OAuth, audit logging, - routing and network architecture constraints. -- Product UX: single-page application (SPA) static files served alongside - deployment APIs or custom docs paths. -- Performance/SRE: thread pool sizing, uvicorn worker settings, log levels, - max request sizes and platform-specific fine-tuning. - -All `DeploymentSettings` are pipeline-level settings. They apply to the -deployment that serves the pipeline as a whole. They are not available at +ASGI app composition without changing framework code. Typical reasons to customize include: + +- Tight security posture: CORS controls, strict headers, authentication, API surface minimization. +- Observability: request/response logging, tracing, metrics, correlation identifiers. +- Enterprise integration: policy gateways, SSO/OIDC/OAuth, audit logging, routing and network architecture constraints. +- Product UX: single-page application (SPA) static files served alongside deployment APIs or custom docs paths. +- Performance/SRE: thread pool sizing, uvicorn worker settings, log levels, max request sizes and platform-specific fine-tuning. + +All `DeploymentSettings` are pipeline-level settings. They apply to the deployment that serves the pipeline as a whole. They are not available at step-level. ## Configuration overview -You can configure `DeploymentSettings` in Python or via YAML, the same way as -other settings classes. The settings can be attached to a pipeline decorator +You can configure `DeploymentSettings` in Python or via YAML, the same way as other settings classes. The settings can be attached to a pipeline decorator or via `with_options`. These settings are only valid at pipeline level. ### Python configuration -Use the `DeploymentSettings` class to configure the deployment settings for your -pipeline in-code +Use the `DeploymentSettings` class to configure the deployment settings for your pipeline in-code ```python from zenml import pipeline @@ -124,8 +111,7 @@ Check out [this page](https://docs.zenml.io/concepts/steps_and_pipelines/configu ## Basic customization options -`DeploymentSettings` expose the following basic customization options. The sections below provide -short examples and guidance. +`DeploymentSettings` expose the following basic customization options. The sections below provide short examples and guidance. - application metadata and paths - built-in endpoints and middleware toggles @@ -209,8 +195,7 @@ With the above settings, the ASGI application will only expose the following end ### Static files (single-page applications) Deployed pipelines can serve full single-page applications (React/Vue/Svelte) -from the same origin as your inference API. This eliminates CORS/auth/routing -friction and lets you ship user-facing UI components alongside +from the same origin as your inference API. This eliminates CORS/auth/routing friction and lets you ship user-facing UI components alongside your endpoints, such as: * operator dashboards @@ -222,11 +207,9 @@ your endpoints, such as: * observability dashboards * customer-facing playgrounds -Co-locating UI and API streamlines delivery (one image, one URL, one CI/CD), -improves latency, and keeps telemetry and auth consistent. +Co-locating UI and API streamlines delivery (one image, one URL, one CI/CD), improves latency, and keeps telemetry and auth consistent. -To enable this, point `dashboard_files_path` to a directory containing an -`index.html` and any static assets. The path must be relative to the +To enable this, point `dashboard_files_path` to a directory containing an `index.html` and any static assets. The path must be relative to the [source root](../steps-pipelines/sources.md#source-root): ```python @@ -281,8 +264,7 @@ settings = DeploymentSettings( ### Secure headers -Harden responses with strict headers. Each field supports either a boolean or -string. Using `True` selects a safe default, `False` disables the header, and +Harden responses with strict headers. Each field supports either a boolean or string. Using `True` selects a safe default, `False` disables the header, and custom strings allow fully custom policies: ```python @@ -308,8 +290,7 @@ settings = DeploymentSettings( ) ``` -Set any field to `False` to omit that header. Set to a string for a custom -value. The defaults are strong, production-safe policies. +Set any field to `False` to omit that header. Set to a string for a custom value. The defaults are strong, production-safe policies. ### Startup and shutdown hooks @@ -423,8 +404,7 @@ When the built-in ASGI application, endpoints and middleware are not enough, you ### Custom endpoints -In production, custom endpoints are often required alongside the main -pipeline invoke route. Common use-cases include: +In production, custom endpoints are often required alongside the main pipeline invoke route. Common use-cases include: - Online inference controls - model (re)load, warm-up, and cache priming @@ -678,10 +658,8 @@ deploy_settings = DeploymentSettings( ) ``` -And here is a minimal ZenML inference pipeline that uses the globally loaded -model. The prediction step reads the model from the global variable set -by the FastAPI router above. You can invoke this pipeline via the built-in -`/invoke` endpoint once a model has been loaded through `/model/load`. +And here is a minimal ZenML inference pipeline that uses the globally loaded model. The prediction step reads the model from the global variable set +by the FastAPI router above. You can invoke this pipeline via the built-in `/invoke` endpoint once a model has been loaded through `/model/load`. ```python from typing import List @@ -950,8 +928,7 @@ Definitions can be provided as Python objects or as loadable source path strings The extensions are summoned to take part in the ASGI application building process near the end of the initialization - after the ASGI app has been built according to the deployment configuration settings. -The example below installs API key authentication at the FastAPI application -level, attaches the dependency to selected routes, registers an auth error +The example below installs API key authentication at the FastAPI application level, attaches the dependency to selected routes, registers an auth error handler, and augments the OpenAPI schema with the security scheme. ```python @@ -1066,20 +1043,16 @@ settings = DeploymentSettings( ## Implementation customizations for advanced use cases -For cases where you need deeper control over how the ASGI app is created or -how the deployment logic is implemented, you can swap/extend the core +For cases where you need deeper control over how the ASGI app is created or how the deployment logic is implemented, you can swap/extend the core components using the following `DeploymentSettings` fields: -- `deployment_app_runner_flavor` and `deployment_app_runner_kwargs` let you - choose or extend the app runner that constructs and runs the ASGI app. This +- `deployment_app_runner_flavor` and `deployment_app_runner_kwargs` let you choose or extend the app runner that constructs and runs the ASGI app. This needs to be set to a subclass of `BaseDeploymentAppRunnerFlavor`, which is basically a descriptor of an app runner implementation that itself is a subclass of `BaseDeploymentAppRunner`. -- `deployment_service_class` and `deployment_service_kwargs` let you provide - your own deployment service to customize the pipeline deployment logic. This +- `deployment_service_class` and `deployment_service_kwargs` let you provide your own deployment service to customize the pipeline deployment logic. This needs to be set to a subclass of `BasePipelineDeploymentService`. -Both accept loadable sources or objects. We cover how to implement custom -runner flavors and services in a dedicated guide. +Both accept loadable sources or objects. We cover how to implement custom runner flavors and services in a dedicated guide.
ZenML Scarf
diff --git a/docs/book/how-to/environment-variables/environment-variables.md b/docs/book/how-to/environment-variables/environment-variables.md index d4aff11d31e..edff81a8cad 100644 --- a/docs/book/how-to/environment-variables/environment-variables.md +++ b/docs/book/how-to/environment-variables/environment-variables.md @@ -30,11 +30,9 @@ Environment variables and secrets can be configured at different levels with inc ## ZenML repository directory name By default, `zenml init` creates a `.zen` directory that marks the root of a -ZenML repository. ZenML uses that marker later when it resolves local code, for -example custom stack component flavors that live in your repository. +ZenML repository. ZenML uses that marker later when it resolves local code, for example custom stack component flavors that live in your repository. -If your workspace cannot use `.zen` as the marker directory name, set -`ZENML_REPOSITORY_DIRECTORY_NAME` before initializing or discovering the +If your workspace cannot use `.zen` as the marker directory name, set `ZENML_REPOSITORY_DIRECTORY_NAME` before initializing or discovering the repository: ```bash @@ -42,8 +40,7 @@ export ZENML_REPOSITORY_DIRECTORY_NAME=.my-zenml zenml init ``` -After that, ZenML looks for `.my-zenml` instead of `.zen`. Keep this setting -consistent for everyone and every process that works with the repository; if one +After that, ZenML looks for `.my-zenml` instead of `.zen`. Keep this setting consistent for everyone and every process that works with the repository; if one shell uses `.zen` and another uses `.my-zenml`, they will not agree on where the ZenML repository root is. diff --git a/docs/book/how-to/infrastructure-deployment/infrastructure-as-code/README.md b/docs/book/how-to/infrastructure-deployment/infrastructure-as-code/README.md index 928d6c92da6..6fb9881da86 100644 --- a/docs/book/how-to/infrastructure-deployment/infrastructure-as-code/README.md +++ b/docs/book/how-to/infrastructure-deployment/infrastructure-as-code/README.md @@ -5,8 +5,7 @@ icon: network-wired # Infrastructure as code with Terraform -[Infrastructure as Code (IaC)](https://aws.amazon.com/what-is/iac) is\ -the practice of managing and provisioning infrastructure through code\ +[Infrastructure as Code (IaC)](https://aws.amazon.com/what-is/iac) is\ the practice of managing and provisioning infrastructure through code\ instead of through manual processes. In this section, we will show you how to integrate ZenML with popular\ @@ -14,17 +13,14 @@ IaC tools such as [Terraform](https://developer.hashicorp.com/terraform). ![Screenshot of ZenML stack on Terraform Registry](../../../.gitbook/assets/terraform_providers_screenshot.png) -Terraform is a powerful tool for managing infrastructure as code, and is by far the -most popular IaC tool. Many companies already have existing Terraform setups, +Terraform is a powerful tool for managing infrastructure as code, and is by far the most popular IaC tool. Many companies already have existing Terraform setups, and it is often desirable to integrate ZenML with this setup. We already got a glimpse on how to [deploy a cloud stack with Terraform](../stack-deployment/deploy-a-cloud-stack-with-terraform.md) -using existing Terraform modules that are maintained by the ZenML team. While this -is a great solution for quickly getting started, it might not always be suitable for +using existing Terraform modules that are maintained by the ZenML team. While this is a great solution for quickly getting started, it might not always be suitable for your use case. -This guide is for advanced users who want to manage their own custom Terraform code but -want to use ZenML to manage their stacks. For this, the +This guide is for advanced users who want to manage their own custom Terraform code but want to use ZenML to manage their stacks. For this, the [ZenML provider](https://registry.terraform.io/providers/zenml-io/zenml/latest) is a better choice. ## Understanding the Two-Phase Approach @@ -90,8 +86,7 @@ This will create a service account and generate an API key that you can use to a The API key is shown only once during creation. Make sure to save it securely, as you cannot retrieve it later. If you lose it, you'll need to create a new key. {% endhint %} -You can learn more about how to generate a `ZENML_API_KEY` via service accounts -[here](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account). +You can learn more about how to generate a `ZENML_API_KEY` via service accounts [here](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account). If you're using a ZenML Pro server, you will need to create a Personal Access Token or an organization-level service account and an API key for it. You can find more about Personal Access Tokens [here](https://docs.zenml.io/pro/access-management/personal-access-tokens) and organization-level service accounts and API keys [here](https://docs.zenml.io/pro/access-management/service-accounts). @@ -453,7 +448,6 @@ Remember to: - Regular backup of your Terraform state files - Version control your Terraform configurations (excluding sensitive files) -To learn more about the ZenML terraform provider, visit the -[ZenML provider](https://registry.terraform.io/providers/zenml-io/zenml/latest). +To learn more about the ZenML terraform provider, visit the [ZenML provider](https://registry.terraform.io/providers/zenml-io/zenml/latest).
ZenML Scarf
diff --git a/docs/book/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform.md b/docs/book/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform.md index 29ede3245ca..9f1765bcb57 100644 --- a/docs/book/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform.md +++ b/docs/book/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform.md @@ -7,8 +7,7 @@ icon: globe ZenML maintains a collection of [Terraform modules](https://registry.terraform.io/modules/zenml-io/zenml-stack) designed to streamline the provisioning of cloud resources and seamlessly integrate them with ZenML Stacks. These modules simplify the setup process, allowing users to quickly provision cloud resources as well as configure and authorize ZenML to utilize them for running pipelines and other AI/ML operations. -By leveraging these Terraform modules, users can ensure a more efficient and scalable deployment of their machine learning infrastructure, ultimately enhancing their development and operational workflows. The modules' implementation can also be used as a reference for creating custom Terraform\ -configurations tailored to specific cloud environments and requirements. +By leveraging these Terraform modules, users can ensure a more efficient and scalable deployment of their machine learning infrastructure, ultimately enhancing their development and operational workflows. The modules' implementation can also be used as a reference for creating custom Terraform\ configurations tailored to specific cloud environments and requirements. {% hint style="info" %} Terraform requires you to manage your infrastructure as code yourself. Among other things, this means that you will need to have Terraform installed on your machine, and you will need to manually manage the state of your infrastructure. @@ -139,8 +138,7 @@ zenml integration install zenml stack set ``` -You can find more details specific to the cloud provider of your choice in the\ -next section: +You can find more details specific to the cloud provider of your choice in the\ next section: {% tabs %} {% tab title="AWS" %} @@ -194,8 +192,7 @@ output "zenml_stack_name" { **Stack Components** -The Terraform module will create a ZenML stack configuration with the\ -following components: +The Terraform module will create a ZenML stack configuration with the\ following components: 1. An S3 Artifact Store linked to an S3 bucket via an AWS Service Connector configured with IAM role credentials 2. An ECR Container Registry linked to an ECR repository via an AWS Service Connector configured with IAM role credentials @@ -274,8 +271,7 @@ output "zenml_stack_name" { **Stack Components** -The Terraform module will create a ZenML stack configuration with the\ -following components: +The Terraform module will create a ZenML stack configuration with the\ following components: 1. An GCP Artifact Store linked to a GCS bucket via a GCP Service Connector configured with the GCP service account credentials 2. An GCP Container Registry linked to a Google Artifact Registry via a GCP Service Connector configured with the GCP service account credentials @@ -368,8 +364,7 @@ output "zenml_stack_name" { **Stack Components** -The Terraform module will create a ZenML stack configuration with the\ -following components: +The Terraform module will create a ZenML stack configuration with the\ following components: 1. An Azure Artifact Store linked to an Azure Storage Account and Blob Container via an Azure Service Connector configured with Azure Service Principal credentials 2. An ACR Container Registry linked to an Azure Container Registry via an Azure Service Connector configured with Azure Service Principal credentials diff --git a/docs/book/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack.md b/docs/book/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack.md index c1ad394b68d..1ab5e3f8914 100644 --- a/docs/book/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack.md +++ b/docs/book/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack.md @@ -23,13 +23,11 @@ Once you are connected to your deployed ZenML instance, you can use the 1-click {% tabs %} {% tab title="Dashboard" %} -In order to create a remote stack over the dashboard, go to the stacks page\ -on the dashboard and click "+ New Stack". +In order to create a remote stack over the dashboard, go to the stacks page\ on the dashboard and click "+ New Stack". ![The new stacks page](../../../.gitbook/assets/register_stack_button.png) -Since we will be deploying it from scratch, select "New Infrastructure" on the\ -next page: +Since we will be deploying it from scratch, select "New Infrastructure" on the\ next page: ![Options for registering a stack](../../../.gitbook/assets/register_stack_page.png) @@ -135,12 +133,10 @@ Once the Terraform deployment is complete, you may close the Cloud Shell session Azure Stack Dashboard output -
-{% endtab %} + {% endtab %} {% tab title="CLI" %} -In order to create a remote stack over the CLI, you can use the following\ -command: +In order to create a remote stack over the CLI, you can use the following\ command: ```shell zenml stack deploy -p {aws|gcp|azure} @@ -161,13 +157,11 @@ You will have to log in to your AWS account, have permission to deploy an AWS Cl ![Finalizing the new stack](../../../.gitbook/assets/deploy_stack_aws_cloudformation.png) The Cloud Formation stack will provision the necessary resources for your new\ -AWS stack and automatically register the stack with your ZenML server. You can\ -monitor the progress of the stack in your AWS console: +AWS stack and automatically register the stack with your ZenML server. You can\ monitor the progress of the stack in your AWS console: ![AWS Cloud Formation progress](../../../.gitbook/assets/deploy_stack_aws_cf_progress.png) -Once the provisioning is complete, you may close the AWS Cloud Formation page\ -and return to the ZenML CLI to view the newly created stack: +Once the provisioning is complete, you may close the AWS Cloud Formation page\ and return to the ZenML CLI to view the newly created stack: ![AWS Stack CLI output](../../../.gitbook/assets/deploy_stack_aws_cli_output.png) diff --git a/docs/book/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack.md b/docs/book/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack.md index ce579d2c0b6..b607dde64ab 100644 --- a/docs/book/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack.md +++ b/docs/book/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack.md @@ -5,8 +5,7 @@ icon: cloud # Register a cloud stack -In ZenML, the [stack](https://docs.zenml.io/user-guides/production-guide/understand-stacks) is a fundamental concept that represents the configuration of your\ -infrastructure. In a normal workflow, creating a stack requires you to first deploy the necessary pieces of infrastructure and then define them as stack components in ZenML with proper authentication. +In ZenML, the [stack](https://docs.zenml.io/user-guides/production-guide/understand-stacks) is a fundamental concept that represents the configuration of your\ infrastructure. In a normal workflow, creating a stack requires you to first deploy the necessary pieces of infrastructure and then define them as stack components in ZenML with proper authentication. Especially in a remote setting, this process can be challenging and time-consuming, and it may create multi-faceted problems. This is why we implemented a feature called the stack wizard, which allows you to **browse through your existing infrastructure and use it to register a ZenML cloud stack**. @@ -22,8 +21,7 @@ The stack wizard is available to you through both our CLI and our dashboard. {% tabs %} {% tab title="Dashboard" %} -If you are using the dashboard, the stack wizard is available through\ -the stacks page. +If you are using the dashboard, the stack wizard is available through\ the stacks page. ![The new stacks page](../../../.gitbook/assets/stack-wizard-new-stack.png) @@ -43,8 +41,7 @@ Choose one of the possible authentication methods based on your provider and fil AWS: Authentication methods -If you select `aws` as your cloud provider, and you haven't selected a connector\ -or declined auto-configuration, you will be prompted to select an authentication method for your cloud connector. +If you select `aws` as your cloud provider, and you haven't selected a connector\ or declined auto-configuration, you will be prompted to select an authentication method for your cloud connector. {% code title="Available authentication methods for AWS" %} ``` @@ -99,8 +96,7 @@ or declined auto-configuration, you will be prompted to select an authentication GCP: Authentication methods -If you select `gcp` as your cloud provider, and you haven't selected a connector\ -or declined auto-configuration, you will be prompted to select an authentication\ +If you select `gcp` as your cloud provider, and you haven't selected a connector\ or declined auto-configuration, you will be prompted to select an authentication\ method for your cloud connector. {% code title="Available authentication methods for GCP" %} @@ -152,8 +148,7 @@ method for your cloud connector. Azure: Authentication methods -If you select `azure` as your cloud provider, and you haven't selected a\ -connector or declined auto-configuration, you will be prompted to select an\ +If you select `azure` as your cloud provider, and you haven't selected a\ connector or declined auto-configuration, you will be prompted to select an\ authentication method for your cloud connector. {% code title="Available authentication methods for Azure" %} @@ -189,18 +184,14 @@ zenml stack register -p {aws|gcp|azure} To register the cloud stack, the first thing that the wizard needs is a [service connector](https://docs.zenml.io/stacks/service-connectors/auth-management). You can either use an existing connector by providing its ID or name`-sc ` (CLI-Only), or the wizard will create one for you. {% hint style="info" %} -Similar to the service connector, if you use the CLI, you can also use existing\ -stack components. However, this is only possible if these components are already\ -configured with the same service connector that you provided through the\ -parameter described above. +Similar to the service connector, if you use the CLI, you can also use existing\ stack components. However, this is only possible if these components are already\ +configured with the same service connector that you provided through the\ parameter described above. {% endhint %} **Define Service Connector** -As the very first step, the configuration wizard will check if the selected\ -cloud provider credentials can be acquired automatically from the local environment.\ -If the credentials are found, you will be offered to use them or proceed to\ -manual configuration. +As the very first step, the configuration wizard will check if the selected\ cloud provider credentials can be acquired automatically from the local environment.\ +If the credentials are found, you will be offered to use them or proceed to\ manual configuration. {% code title="Example prompt for AWS auto-configuration" %} ``` @@ -217,8 +208,7 @@ If you decline auto-configuration next you might be offered the list of already AWS: Authentication methods -If you select `aws` as your cloud provider, and you haven't selected a connector\ -or declined auto-configuration, you will be prompted to select an authentication\ +If you select `aws` as your cloud provider, and you haven't selected a connector\ or declined auto-configuration, you will be prompted to select an authentication\ method for your cloud connector. {% code title="Available authentication methods for AWS" %} @@ -274,8 +264,7 @@ method for your cloud connector. GCP: Authentication methods -If you select `gcp` as your cloud provider, and you haven't selected a connector\ -or declined auto-configuration, you will be prompted to select an authentication\ +If you select `gcp` as your cloud provider, and you haven't selected a connector\ or declined auto-configuration, you will be prompted to select an authentication\ method for your cloud connector. {% code title="Available authentication methods for GCP" %} @@ -327,8 +316,7 @@ method for your cloud connector. Azure: Authentication methods -If you select `azure` as your cloud provider, and you haven't selected a\ -connector or declined auto-configuration, you will be prompted to select an\ +If you select `azure` as your cloud provider, and you haven't selected a\ connector or declined auto-configuration, you will be prompted to select an\ authentication method for your cloud connector. {% code title="Available authentication methods for Azure" %} @@ -363,8 +351,7 @@ All three are crucial for a basic cloud stack. Extra components can be added lat For each component, you will be asked: -* if you would like to reuse one of the existing components connected via a defined\ - service connector (if any) +* if you would like to reuse one of the existing components connected via a defined\ service connector (if any) {% code title="Example Command Output for available orchestrator" %} ``` diff --git a/docs/book/how-to/manage-zenml-server/best-practices-upgrading-zenml.md b/docs/book/how-to/manage-zenml-server/best-practices-upgrading-zenml.md index fc6500d7519..07fd658e1a5 100644 --- a/docs/book/how-to/manage-zenml-server/best-practices-upgrading-zenml.md +++ b/docs/book/how-to/manage-zenml-server/best-practices-upgrading-zenml.md @@ -112,8 +112,7 @@ Read about [how snapshots work](https://docs.zenml.io/user-guides/tutorial/trigg {% endhint %} After building, execute all snapshots end-to-end as a smoke test. -Ideally, your data science teams have a "smoke test" parameter in the pipeline -to load mock data just for this scenario! +Ideally, your data science teams have a "smoke test" parameter in the pipeline to load mock data just for this scenario! ## 🚀 Step #6: Upgrade Production and Go Live diff --git a/docs/book/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account.md b/docs/book/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account.md index 4c3a1516635..b333fa83236 100644 --- a/docs/book/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account.md +++ b/docs/book/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account.md @@ -28,8 +28,7 @@ export ZENML_STORE_API_KEY= ``` {% hint style="info" %} -You don't need to run `zenml login` after setting these two environment -variables and can start interacting with your server right away. +You don't need to run `zenml login` after setting these two environment variables and can start interacting with your server right away. {% endhint %} {% hint style="info" %} diff --git a/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-sixty.md b/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-sixty.md index e9338af3511..6f500045a7d 100644 --- a/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-sixty.md +++ b/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-sixty.md @@ -6,23 +6,17 @@ description: How to migrate from ZenML 0.58.2 to 0.60.0 (Pydantic 2 edition). ZenML now uses Pydantic v2. 🥳 -This upgrade comes with a set of critical updates. While your user experience -mostly remains unaffected, you might see unexpected behavior due to the -changes in our dependencies. Moreover, since Pydantic v2 provides a slightly -stricter validation process, you might end up bumping into some validation -errors which was not caught before, but it is all for the better 🙂 If -you run into any other errors, please let us know either on -[GitHub](https://github.com/zenml-io/zenml) or on -our [Slack](https://zenml.io/slack-invite). +This upgrade comes with a set of critical updates. While your user experience mostly remains unaffected, you might see unexpected behavior due to the +changes in our dependencies. Moreover, since Pydantic v2 provides a slightly stricter validation process, you might end up bumping into some validation +errors which was not caught before, but it is all for the better 🙂 If you run into any other errors, please let us know either on +[GitHub](https://github.com/zenml-io/zenml) or on our [Slack](https://zenml.io/slack-invite). ## Changes in some of the critical dependencies -- SQLModel is one of the core dependencies of ZenML and prior to this upgrade, - we were utilizing version `0.0.8`. However, this version is relatively +- SQLModel is one of the core dependencies of ZenML and prior to this upgrade, we were utilizing version `0.0.8`. However, this version is relatively outdated and incompatible with Pydantic v2. Within the scope of this upgrade, we upgraded it to `0.0.18`. -- Due to the change in the SQLModel version, we also had to upgrade our - SQLAlchemy dependency from V1 to v2. While this does not affect the way +- Due to the change in the SQLModel version, we also had to upgrade our SQLAlchemy dependency from V1 to v2. While this does not affect the way that you are using ZenML, if you are using SQLAlchemy in your environment, you might have to migrate your code as well. For a detailed list of changes, feel free to @@ -30,139 +24,97 @@ our [Slack](https://zenml.io/slack-invite). ## Changes in `pydantic` -Pydantic v2 brings a lot of new and exciting changes to the table. The core -logic now uses Rust and it is much faster and more efficient in terms of -performance. On top of it, the main concepts like model design, configuration, -validation, or serialization now include a lot of new cool features. If you are -using `pydantic` in your workflow and are interested in the new changes, you can -check [the brilliant migration guide](https://docs.pydantic.dev/2.7/migration/) +Pydantic v2 brings a lot of new and exciting changes to the table. The core logic now uses Rust and it is much faster and more efficient in terms of +performance. On top of it, the main concepts like model design, configuration, validation, or serialization now include a lot of new cool features. If you are +using `pydantic` in your workflow and are interested in the new changes, you can check [the brilliant migration guide](https://docs.pydantic.dev/2.7/migration/) provided by the `pydantic` team to see the full list of changes. ## Changes in our integrations changes -Much like ZenML, `pydantic` is an important dependency in many other Python -packages. That’s why conducting this upgrade helped us unlock a new version for -several ZenML integration dependencies. Additionally, in some instances, we had -to adapt the functionality of the integration to keep it compatible -with `pydantic`. So, if you are using any of these integrations, please go -through the changes. +Much like ZenML, `pydantic` is an important dependency in many other Python packages. That’s why conducting this upgrade helped us unlock a new version for +several ZenML integration dependencies. Additionally, in some instances, we had to adapt the functionality of the integration to keep it compatible +with `pydantic`. So, if you are using any of these integrations, please go through the changes. ### Airflow -As mentioned above upgrading our `pydantic` dependency meant we had to upgrade -our `sqlmodel` dependency. Upgrading our `sqlmodel` dependency meant we had to -upgrade our `sqlalchemy` dependency as well. Unfortunately, `apache-airflow` -is still using `sqlalchemy` v1 and is incompatible with pydantic v2. As a -solution, we have removed the dependencies of the `airflow` integration. Now, -you can use ZenML to create your Airflow pipelines and use a separate -environment to run them with Airflow. You can check the updated docs -[right here](https://docs.zenml.io/stacks/orchestrators/airflow). +As mentioned above upgrading our `pydantic` dependency meant we had to upgrade our `sqlmodel` dependency. Upgrading our `sqlmodel` dependency meant we had to +upgrade our `sqlalchemy` dependency as well. Unfortunately, `apache-airflow` is still using `sqlalchemy` v1 and is incompatible with pydantic v2. As a +solution, we have removed the dependencies of the `airflow` integration. Now, you can use ZenML to create your Airflow pipelines and use a separate +environment to run them with Airflow. You can check the updated docs [right here](https://docs.zenml.io/stacks/orchestrators/airflow). ### AWS -Some of our integrations now require `protobuf` 4. Since our -previous `sagemaker` version (`2.117.0`) did not support `protobof` 4, we could -not pair it with these new integrations. Thankfully `sagemaker` started -supporting `protobuf` 4 with version `2.172.0` and relaxing its dependency +Some of our integrations now require `protobuf` 4. Since our previous `sagemaker` version (`2.117.0`) did not support `protobof` 4, we could +not pair it with these new integrations. Thankfully `sagemaker` started supporting `protobuf` 4 with version `2.172.0` and relaxing its dependency solved the compatibility issue. ### Evidently -The old version of our `evidently` integration was not compatible with Pydantic -v2. They started supporting it starting from version `0.4.16`. As their latest -version is `0.4.22`, the new dependency of the integration is limited between -these two versions. +The old version of our `evidently` integration was not compatible with Pydantic v2. They started supporting it starting from version `0.4.16`. As their latest +version is `0.4.22`, the new dependency of the integration is limited between these two versions. ### Feast Our previous implementation of the `feast` integration was not compatible with -Pydantic v2 due to the extra `redis` dependency we were using. This extra -dependency is now removed and the `feast` integration is working as intended. +Pydantic v2 due to the extra `redis` dependency we were using. This extra dependency is now removed and the `feast` integration is working as intended. ### GCP -The previous version of the Kubeflow dependency (`kfp==1.8.22`) in our GCP -integration required Pydantic V1 to be installed. While we were upgrading our -Pydantic dependency, we saw this as an opportunity and wanted to use this chance -to upgrade the `kfp` dependency to v2 (which has no dependencies on the Pydantic -library). This is why you may see some functional changes in the vertex step -operator and orchestrator. If you would like to go through the changes in -the `kfp` library, you can -find [the migration guide here](https://www.kubeflow.org/docs/components/pipelines/v2/migration/). +The previous version of the Kubeflow dependency (`kfp==1.8.22`) in our GCP integration required Pydantic V1 to be installed. While we were upgrading our +Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the `kfp` dependency to v2 (which has no dependencies on the Pydantic +library). This is why you may see some functional changes in the vertex step operator and orchestrator. If you would like to go through the changes in +the `kfp` library, you can find [the migration guide here](https://www.kubeflow.org/docs/components/pipelines/v2/migration/). ### Great Expectations -Great Expectations started supporting Pydantic v2 starting from -version `0.17.15` and they are closing in on their `1.0` release. Since this -release might include a lot of big changes, we adjusted the dependency in our -integration to `great-expectations>=0.17.15,<1.0`. We will try to keep it +Great Expectations started supporting Pydantic v2 starting from version `0.17.15` and they are closing in on their `1.0` release. Since this +release might include a lot of big changes, we adjusted the dependency in our integration to `great-expectations>=0.17.15,<1.0`. We will try to keep it updated in the future once they release the `1.0` version ### Kubeflow -Similar to the GCP integration, the previous version of the kubeflow -dependency (`kfp==1.8.22`) in our `kubeflow` integration required Pydantic V1 to -be installed. While we were upgrading our Pydantic dependency, we saw this as an -opportunity and wanted to use this chance to upgrade the `kfp` dependency to -v2 (which has no dependencies on the Pydantic library). If you would like to go -through the changes in the `kfp` library, you can +Similar to the GCP integration, the previous version of the kubeflow dependency (`kfp==1.8.22`) in our `kubeflow` integration required Pydantic V1 to +be installed. While we were upgrading our Pydantic dependency, we saw this as an opportunity and wanted to use this chance to upgrade the `kfp` dependency to +v2 (which has no dependencies on the Pydantic library). If you would like to go through the changes in the `kfp` library, you can find [the migration guide here](https://www.kubeflow.org/docs/components/pipelines/v2/migration/). ( -We also are considering adding an alternative version of this integration so our -users can keep using `kfp` V1 in their environment. Stay tuned for any updates.) +We also are considering adding an alternative version of this integration so our users can keep using `kfp` V1 in their environment. Stay tuned for any updates.) ### MLflow -`mlflow` is compatible with both Pydantic V1 and v2. However, due to a known -issue, if you install `zenml` first and then -do `zenml integration install mlflow -y`, it downgrades `pydantic` to V1. This -is why we manually added the same duplicated `pydantic` requirement in the -integration definition as well. Keep in mind that the `mlflow` library is still -using some features of `pydantic` V1 which are deprecated. So, if the -integration is installed in your environment, you might run into some -deprecation warnings. +`mlflow` is compatible with both Pydantic V1 and v2. However, due to a known issue, if you install `zenml` first and then +do `zenml integration install mlflow -y`, it downgrades `pydantic` to V1. This is why we manually added the same duplicated `pydantic` requirement in the +integration definition as well. Keep in mind that the `mlflow` library is still using some features of `pydantic` V1 which are deprecated. So, if the +integration is installed in your environment, you might run into some deprecation warnings. ### Label Studio -While we were working on updating our `pydantic` dependency, -the `label-studio-sdk` has released its 1.0 version. In this new -version, `pydantic` v2 is also supported. The implementation and documentation -of our Label Studio integration have been updated accordingly. +While we were working on updating our `pydantic` dependency, the `label-studio-sdk` has released its 1.0 version. In this new +version, `pydantic` v2 is also supported. The implementation and documentation of our Label Studio integration have been updated accordingly. ### Skypilot -With the switch to `pydantic` v2, the implementation of our `skypilot` -integration mostly remained untouched. However, due to an incompatibility -between the new version `pydantic` and the `azurecli`, the `skypilot[azure]` -flavor can not be installed at the same time, thus our `skypilot_azure` -integration is currently deactivated. We are working on fixing this issue and if -you are using this integration in your workflows, we recommend staying on the +With the switch to `pydantic` v2, the implementation of our `skypilot` integration mostly remained untouched. However, due to an incompatibility +between the new version `pydantic` and the `azurecli`, the `skypilot[azure]` flavor can not be installed at the same time, thus our `skypilot_azure` +integration is currently deactivated. We are working on fixing this issue and if you are using this integration in your workflows, we recommend staying on the previous version of ZenML until we can solve this issue. ### Tensorflow -The new version of `pydantic` creates a drift between `tensorflow` -and `typing_extensions` packages and relaxing the dependencies here resolves -the issue. At the same time, the upgrade to `kfp` v2 (in integrations -like `kubeflow`, `tekton`, or `gcp`) bumps our `protobuf` dependency from `3.X` -to `4.X`. To stay compatible with this requirement, the installed version -of `tensorflow` needs to be `>=2.12.0`. While this change solves the dependency -issues in most settings, we have bumped into some errors while -using `tensorflow` 2.12.0 on Python 3.8 on Ubuntu. If you would like to use this +The new version of `pydantic` creates a drift between `tensorflow` and `typing_extensions` packages and relaxing the dependencies here resolves +the issue. At the same time, the upgrade to `kfp` v2 (in integrations like `kubeflow`, `tekton`, or `gcp`) bumps our `protobuf` dependency from `3.X` +to `4.X`. To stay compatible with this requirement, the installed version of `tensorflow` needs to be `>=2.12.0`. While this change solves the dependency +issues in most settings, we have bumped into some errors while using `tensorflow` 2.12.0 on Python 3.8 on Ubuntu. If you would like to use this integration, please consider using a higher Python version. ### Tekton -Similar to the `gcp` and `kubeflow` integrations, the old version of -our `tekton` integration was not compatible with `pydantic` V1 due to its `kfp` -dependency. With the switch from `kfp` V1 to v2, we have adapted our -implementation to use the new version of `kfp` library and updated our +Similar to the `gcp` and `kubeflow` integrations, the old version of our `tekton` integration was not compatible with `pydantic` V1 due to its `kfp` +dependency. With the switch from `kfp` V1 to v2, we have adapted our implementation to use the new version of `kfp` library and updated our documentation accordingly. {% hint style="warning" %} -Due to all aforementioned changes, when you upgrade ZenML to 0.60.0, you might -run into some dependency issues, especially if you were previously using an -integration which was not supporting Pydantic v2 before. In such cases, we -highly recommend setting up a fresh Python environment. +Due to all aforementioned changes, when you upgrade ZenML to 0.60.0, you might run into some dependency issues, especially if you were previously using an +integration which was not supporting Pydantic v2 before. In such cases, we highly recommend setting up a fresh Python environment. {% endhint %}
ZenML Scarf
diff --git a/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-thirty.md b/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-thirty.md index 2fd54ac5a8e..42e357ec35b 100644 --- a/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-thirty.md +++ b/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-thirty.md @@ -3,17 +3,13 @@ description: How to migrate from ZenML 0.20.0-0.23.0 to 0.30.0-0.39.1. --- {% hint style="warning" %} -Migrating to `0.30.0` performs non-reversible database changes so downgrading -to `<=0.23.0` is not possible afterwards. If you are running on an older ZenML -version, please follow the -[0.20.0 Migration Guide](migration-zero-twenty.md) first to prevent unexpected +Migrating to `0.30.0` performs non-reversible database changes so downgrading to `<=0.23.0` is not possible afterwards. If you are running on an older ZenML +version, please follow the [0.20.0 Migration Guide](migration-zero-twenty.md) first to prevent unexpected database migration failures. {% endhint %} -The ZenML 0.30.0 release removed the `ml-pipelines-sdk` dependency in favor of -natively storing pipeline runs and artifacts in the ZenML database. The -corresponding database migration will happen automatically as soon as you run -any `zenml ...` CLI command after installing the new ZenML version, e.g.: +The ZenML 0.30.0 release removed the `ml-pipelines-sdk` dependency in favor of natively storing pipeline runs and artifacts in the ZenML database. The +corresponding database migration will happen automatically as soon as you run any `zenml ...` CLI command after installing the new ZenML version, e.g.: ```bash pip install zenml==0.30.0 diff --git a/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-twenty.md b/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-twenty.md index 7b4ec14329c..99d89e5c0b1 100644 --- a/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-twenty.md +++ b/docs/book/how-to/manage-zenml-server/migration-guide/migration-zero-twenty.md @@ -321,8 +321,7 @@ This is part of a broader configuration rehaul which is discussed next. Alongside the architectural shift, Pipeline configuration has been completely rethought. This video gives an overview of how configuration has changed with ZenML in the post ZenML 0.20.0 world. {% embed url="https://www.youtube.com/embed/hI-UNV7uoNI" %} -Configuring pipelines, steps, and stack components in ZenML -{% endembed %} +Configuring pipelines, steps, and stack components in ZenML {% endembed %} **What changed?** diff --git a/docs/book/how-to/manage-zenml-server/upgrade-zenml-server.md b/docs/book/how-to/manage-zenml-server/upgrade-zenml-server.md index c200d5fe815..93c89745807 100644 --- a/docs/book/how-to/manage-zenml-server/upgrade-zenml-server.md +++ b/docs/book/how-to/manage-zenml-server/upgrade-zenml-server.md @@ -70,8 +70,7 @@ If you need to modify your ZenML server configuration during the upgrade, follow ``` {% hint style="info" %} -It is not recommended to change the container image tag in the Helm chart to custom values, since every Helm chart\ -version is tested to work only with the default image tag. However, if you know what you're doing you can change\ +It is not recommended to change the container image tag in the Helm chart to custom values, since every Helm chart\ version is tested to work only with the default image tag. However, if you know what you're doing you can change\ the `zenml.image.tag` value in your `custom-values.yaml` file to the desired ZenML version (e.g. `0.32.0`). {% endhint %} {% endtab %} diff --git a/docs/book/how-to/metadata/metadata.md b/docs/book/how-to/metadata/metadata.md index 348515b6d6d..9c7b5b9e30d 100644 --- a/docs/book/how-to/metadata/metadata.md +++ b/docs/book/how-to/metadata/metadata.md @@ -203,8 +203,7 @@ log_metadata( The `log_metadata` function does not support logging the same metadata for multiple entities simultaneously. To achieve this, you can use the `bulk_log_metadata` function: -~~~python -from zenml.models import ( +~~~python from zenml.models import ( ArtifactVersionIdentifier, ModelVersionIdentifier, PipelineRunIdentifier, @@ -265,8 +264,7 @@ def get_train_test_datasets(): ``` Keep in mind that when using the `infer_artifacts` option, the `bulk_log_metadata` function logs metadata to all output artifacts of the step. -When logging metadata, you may need the option to use `infer` options in combination with identifier references. For instance, you may want -to log metadata to a step's outputs but also to its inputs. The `bulk_log_metadata` function enables you to use both options in one go: +When logging metadata, you may need the option to use `infer` options in combination with identifier references. For instance, you may want to log metadata to a step's outputs but also to its inputs. The `bulk_log_metadata` function enables you to use both options in one go: ```python from zenml import bulk_log_metadata, get_step_context, step @@ -309,10 +307,8 @@ To improve performance, prefer using the entity's ID directly instead of its nam If the `log_metadata` or `bulk_log_metadata` functions are too restrictive for your use case, you can use the ZenML Client directly to create run metadata for resources: -~~~python -from zenml.client import Client -from zenml.enums import MetadataResourceTypes -from zenml.models import RunMetadataResource +~~~python from zenml.client import Client +from zenml.enums import MetadataResourceTypes from zenml.models import RunMetadataResource client = Client() @@ -401,8 +397,7 @@ The Experiment Comparison tool offers two complementary views for analyzing your ![Parallel Coordinates](../../../book/.gitbook/assets/coordinates-view.png) -The tool lets you compare up to 20 pipeline runs simultaneously and supports any -numerical metadata (`float` or `int`) that you've logged in your pipelines. +The tool lets you compare up to 20 pipeline runs simultaneously and supports any numerical metadata (`float` or `int`) that you've logged in your pipelines. ## Fetching Metadata diff --git a/docs/book/how-to/popular-integrations/aws-guide.md b/docs/book/how-to/popular-integrations/aws-guide.md index cf122d42fd6..b20998379e5 100644 --- a/docs/book/how-to/popular-integrations/aws-guide.md +++ b/docs/book/how-to/popular-integrations/aws-guide.md @@ -10,10 +10,8 @@ This page aims to quickly set up a minimal production stack on AWS. With just a {% hint style="info" %} Would you like to skip ahead and deploy a full AWS ZenML cloud stack already? -Check out the [in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack),\ -the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack),\ -or [the ZenML AWS Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform)\ -for a shortcut on how to deploy & register this stack. +Check out the [in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack),\ the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack),\ +or [the ZenML AWS Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform)\ for a shortcut on how to deploy & register this stack. {% endhint %} ## 1) Set up credentials and local environment diff --git a/docs/book/how-to/stack-components/service_connectors.md b/docs/book/how-to/stack-components/service_connectors.md index 7ab630c2e2f..0799e7f8af0 100644 --- a/docs/book/how-to/stack-components/service_connectors.md +++ b/docs/book/how-to/stack-components/service_connectors.md @@ -62,8 +62,7 @@ Each connector type supports authentication methods specific to that service. ### Creating and Managing Connectors -Service connectors can be created with different authentication methods -depending on your cloud provider and security requirements. +Service connectors can be created with different authentication methods depending on your cloud provider and security requirements. ![Authentication with Service Connectors](../../.gitbook/assets/authentication_with_connectors.png) diff --git a/docs/book/how-to/stack-components/stack_components.md b/docs/book/how-to/stack-components/stack_components.md index f35dd5b0a44..c37c6553987 100644 --- a/docs/book/how-to/stack-components/stack_components.md +++ b/docs/book/how-to/stack-components/stack_components.md @@ -33,12 +33,9 @@ Stacks may also include these optional components: * **Alerter**: Sends notifications about pipeline events * **Annotator**: Manages data labeling workflows -Most component types appear at most once in a stack. Three component types are -repeatable: **step operators**, **experiment trackers**, and **alerters**. If a -stack has more than one component of one of these types, the first attached -component is the default. You can still choose a non-default component by name -in step or pipeline configuration, and you can change the default later with -`zenml stack set-default`. +Most component types appear at most once in a stack. Three component types are repeatable: **step operators**, **experiment trackers**, and **alerters**. If a +stack has more than one component of one of these types, the first attached component is the default. You can still choose a non-default component by name +in step or pipeline configuration, and you can change the default later with `zenml stack set-default`. ## Working with Stacks @@ -87,11 +84,9 @@ zenml stack set-default training-stack --step_operator cpu-step-operator ### Discovering flavor-specific configuration -Stack component flavors have different configuration fields. A local -orchestrator needs very little information; a Kubernetes orchestrator needs +Stack component flavors have different configuration fields. A local orchestrator needs very little information; a Kubernetes orchestrator needs cluster and namespace details; an S3 artifact store needs bucket information. -Once you pass a flavor with `-f` / `--flavor`, the CLI can show the concrete -configuration fields for that flavor: +Once you pass a flavor with `-f` / `--flavor`, the CLI can show the concrete configuration fields for that flavor: ```bash zenml orchestrator register -f kubernetes --help @@ -99,8 +94,7 @@ zenml artifact-store register -f s3 --help zenml step-operator update my-runai-step-operator --help ``` -The help output includes a **Flavor configuration** section. For register and -update commands, pass those fields as `--name=value` arguments. This is often +The help output includes a **Flavor configuration** section. For register and update commands, pass those fields as `--name=value` arguments. This is often the fastest way to answer, "what exactly does this flavor need from me?" Or through the Python API: diff --git a/docs/book/how-to/steps-pipelines/advanced_features.md b/docs/book/how-to/steps-pipelines/advanced_features.md index 7b3fb483169..10302ee8ce9 100644 --- a/docs/book/how-to/steps-pipelines/advanced_features.md +++ b/docs/book/how-to/steps-pipelines/advanced_features.md @@ -65,16 +65,12 @@ def train_model(dataset_path: str, *, epochs: int = 10) -> None: ... ``` -Keyword-only arguments, like `epochs` in the example above, are supported and -are treated as normal step inputs. This is useful when you want a call site to be -explicit about important values. Variadic signatures such as `*args` and -`**kwargs` are not valid step inputs because ZenML needs to know the step +Keyword-only arguments, like `epochs` in the example above, are supported and are treated as normal step inputs. This is useful when you want a call site to be +explicit about important values. Variadic signatures such as `*args` and `**kwargs` are not valid step inputs because ZenML needs to know the step interface before the pipeline runs. -For production code, define the underlying Python functions for your steps and -pipelines at module level. They are easier to import, package, test, and review -that way. If you need a factory-style pattern, you can wrap those top-level -functions with `@step` or `@pipeline(dynamic=True)` inside another function, but +For production code, define the underlying Python functions for your steps and pipelines at module level. They are easier to import, package, test, and review +that way. If you need a factory-style pattern, you can wrap those top-level functions with `@step` or `@pipeline(dynamic=True)` inside another function, but the decorated function itself should still be importable from a module. You can also add semantic metadata to a step with `step_type`: @@ -88,10 +84,8 @@ def summarize_prompt(prompt: str) -> str: ... ``` -The current step types are `StepType.TOOL_CALL`, `StepType.LLM_CALL`, and -`StepType.MEMORY_CALL`. Think of this as a label on the step run: it does not -change how your Python function executes, but it gives the dashboard, DAG -metadata, and downstream consumers a clearer story about what kind of work the +The current step types are `StepType.TOOL_CALL`, `StepType.LLM_CALL`, and `StepType.MEMORY_CALL`. Think of this as a label on the step run: it does not +change how your Python function executes, but it gives the dashboard, DAG metadata, and downstream consumers a clearer story about what kind of work the step represents. ### Asynchronous Pipeline Execution diff --git a/docs/book/how-to/steps-pipelines/configuration.md b/docs/book/how-to/steps-pipelines/configuration.md index 79d4650e5ad..103ba07c070 100644 --- a/docs/book/how-to/steps-pipelines/configuration.md +++ b/docs/book/how-to/steps-pipelines/configuration.md @@ -102,8 +102,7 @@ def train_model(): ... ``` -If you want to make sure a step can only run with a specific experiment tracker/step operator, you can also specify the -component names like this: +If you want to make sure a step can only run with a specific experiment tracker/step operator, you can also specify the component names like this: ```python from zenml import step @@ -143,8 +142,7 @@ Settings in ZenML are categorized into three main types: * **General settings** that can be used on all ZenML pipelines: * `DockerSettings` for container configuration - * `ResourceSettings` for CPU, memory, and GPU allocation (on ZenML Pro, the same - fields drive [resource pools](https://docs.zenml.io/pro/core-concepts/resource-pools) + * `ResourceSettings` for CPU, memory, and GPU allocation (on ZenML Pro, the same fields drive [resource pools](https://docs.zenml.io/pro/core-concepts/resource-pools) for workspace quotas, queuing, and preemption) * `DeploymentSettings` for pipeline deployment configuration - can only be set at the pipeline level @@ -197,10 +195,8 @@ simple_ml_pipeline.configuration.settings["resources"] Resource settings allow you to specify the CPU, memory, and GPU requirements for your steps. -On **ZenML Pro**, those declarations are also what the **resource pool** feature -uses: for eligible dynamic pipelines, the server builds resource -requests from your merged `ResourceSettings` (including `pool_resources` and -`preemptible`), matches them against workspace pools and policies on your stack’s +On **ZenML Pro**, those declarations are also what the **resource pool** feature uses: for eligible dynamic pipelines, the server builds resource +requests from your merged `ResourceSettings` (including `pool_resources` and `preemptible`), matches them against workspace pools and policies on your stack’s orchestrator or step operator, and may queue or preempt work accordingly. For more information on how this feature works, see [ZenML Pro Resource Pools](https://docs.zenml.io/pro/core-concepts/resource-pools). ```python diff --git a/docs/book/how-to/steps-pipelines/dynamic_pipelines.md b/docs/book/how-to/steps-pipelines/dynamic_pipelines.md index 090f976394c..7d054acfb7b 100644 --- a/docs/book/how-to/steps-pipelines/dynamic_pipelines.md +++ b/docs/book/how-to/steps-pipelines/dynamic_pipelines.md @@ -65,10 +65,8 @@ This allows you to modify step behavior based on runtime conditions or data. ### Artifact name substitutions in dynamic pipelines -Dynamic pipelines support the same artifact name substitutions as regular -pipelines. This matters when a dynamically generated step has outputs whose -names include runtime-friendly placeholders. The substituted artifact name is -still a real output that you can pass to downstream steps. +Dynamic pipelines support the same artifact name substitutions as regular pipelines. This matters when a dynamically generated step has outputs whose +names include runtime-friendly placeholders. The substituted artifact name is still a real output that you can pass to downstream steps. ```python from typing import Annotated @@ -89,23 +87,18 @@ def dynamic_pipeline() -> None: consume(score) ``` -One caveat: when you use `child_pipeline.embed(...)`, the child pipeline's own -configuration is not applied. That includes child-level `substitutions`; the +One caveat: when you use `child_pipeline.embed(...)`, the child pipeline's own configuration is not applied. That includes child-level `substitutions`; the parent run's configuration controls the steps that execute inline. ### Step inputs as parameters -Any value you pass to a step that is not the output of another step is uploaded -to the artifact store as an external artifact, even a small `int`. Each upload -costs a write to the artifact store and a request to the server, which adds up -when a pipeline calls steps in a loop. +Any value you pass to a step that is not the output of another step is uploaded to the artifact store as an external artifact, even a small `int`. Each upload +costs a write to the artifact store and a request to the server, which adds up when a pipeline calls steps in a loop. Set the `ZENML_PARAMETER_SIZE_THRESHOLD` environment variable to pass JSON-serializable inputs as step parameters instead, which skips the upload. -The variable must be set in the environment in which the pipeline executes, -not on the client that starts the run. See -[this page](../environment-variables/environment-variables.md) for how to -configure environment variables for pipeline execution. +The variable must be set in the environment in which the pipeline executes, not on the client that starts the run. See +[this page](../environment-variables/environment-variables.md) for how to configure environment variables for pipeline execution. | Value | Behavior | | --- | --- | @@ -113,8 +106,7 @@ configure environment variables for pipeline execution. | a positive number | JSON-serializable inputs up to that many bytes become step parameters. Larger inputs are uploaded. | | `-1` | Every JSON-serializable input becomes a step parameter. | -Inputs that cannot be parameters are still uploaded, so raising the threshold is -safe. Wrap an input with `ExternalArtifact` to keep uploading it. +Inputs that cannot be parameters are still uploaded, so raising the threshold is safe. Wrap an input with `ExternalArtifact` to keep uploading it. ### Step Runtime Configuration @@ -171,8 +163,7 @@ def map_reduce(): reducer(results) # pass list of artifacts directly ``` -For a complete agentic workflow that combines dynamic mapping, reduction, and a -human approval gate, see the +For a complete agentic workflow that combines dynamic mapping, reduction, and a human approval gate, see the [`agentic_hitl_pipeline` example](https://github.com/zenml-io/zenml/tree/main/examples/agentic_hitl_pipeline). Key points: @@ -368,20 +359,16 @@ Using `after=server` here would deadlock, since the dependent would wait for the ### Child pipelines inside dynamic pipelines -Dynamic pipelines can call other dynamic pipelines from their `@pipeline` -body. This is useful for composing larger workflows out of reusable dynamic +Dynamic pipelines can call other dynamic pipelines from their `@pipeline` body. This is useful for composing larger workflows out of reusable dynamic building blocks. Key behavior: - Only dynamic pipelines can be called as child pipelines. - Child pipelines run on the same stack as the parent run. -- Child pipelines can run synchronously (`child(...)`) or concurrently - (`child.submit(...)`). -- Child pipeline calls are only allowed in pipeline bodies, not inside step - functions. -- Child pipelines reuse the parent run's Docker image — they don't trigger a - new build. The child snapshot inherits the parent's build, code reference, +- Child pipelines can run synchronously (`child(...)`) or concurrently (`child.submit(...)`). +- Child pipeline calls are only allowed in pipeline bodies, not inside step functions. +- Child pipelines reuse the parent run's Docker image — they don't trigger a new build. The child snapshot inherits the parent's build, code reference, and code path so the child runs against the exact same image and source bundle as the parent. @@ -426,8 +413,7 @@ def parent_pipeline_concurrent(): ### Inline child pipelines with `embed(...)` -Use `child_pipeline.embed(...)` if you want to reuse another -dynamic pipeline's body without creating a child pipeline run. +Use `child_pipeline.embed(...)` if you want to reuse another dynamic pipeline's body without creating a child pipeline run. ```python @pipeline(dynamic=True) @@ -445,23 +431,16 @@ def parent_pipeline_inline(): - It is not allowed inside `@step` functions. {% hint style="warning" %} -**Limitations of `embed(...)`.** Unlike `child_pipeline(...)` -and `child_pipeline.submit(...)`, the inline form does not apply the child -pipeline's own configuration. The parent run's configuration governs every -step that runs inline: +**Limitations of `embed(...)`.** Unlike `child_pipeline(...)` and `child_pipeline.submit(...)`, the inline form does not apply the child +pipeline's own configuration. The parent run's configuration governs every step that runs inline: -- Child-level `settings`, `retry`, `enable_cache`, `enable_step_logs`, - `environment`, `secrets`, `tags`, `substitutions`, `model`, and +- Child-level `settings`, `retry`, `enable_cache`, `enable_step_logs`, `environment`, `secrets`, `tags`, `substitutions`, `model`, and `on_init` / `on_success` / `on_failure` / `on_cleanup` hooks are ignored. -- Per-step Docker overrides on the child pipeline are also ignored — the - parent's image is used for any inline isolated step. -- `depends_on` config templates declared on the child pipeline are not - picked up. -- There is no failure isolation: an exception inside the inline body - aborts the parent run. - -If any of these matter to your use case, call the child as -`child_pipeline(...)` (sync) or `child_pipeline.submit(...)` (concurrent) +- Per-step Docker overrides on the child pipeline are also ignored — the parent's image is used for any inline isolated step. +- `depends_on` config templates declared on the child pipeline are not picked up. +- There is no failure isolation: an exception inside the inline body aborts the parent run. + +If any of these matter to your use case, call the child as `child_pipeline(...)` (sync) or `child_pipeline.submit(...)` (concurrent) instead. Both create a real child run with its own configuration applied. {% endhint %} @@ -469,52 +448,39 @@ In short, use: - `child_pipeline(...)` for a synchronous child run - `child_pipeline.submit(...)` for a concurrent child run -- `child_pipeline.embed(...)` for embedded execution in the parent - run +- `child_pipeline.embed(...)` for embedded execution in the parent run {% hint style="warning" %} -**Resume idempotency depends on submit order.** Child pipeline child runs are -identified by the order of `child_pipeline(...)` / `child_pipeline.submit(...)` -calls in the parent body: the first call to `my_pipeline` becomes -`pipeline:my_pipeline`, the second becomes `pipeline:my_pipeline_2`, and so on. -On resume, ZenML reuses an existing child run only if the same call appears in -the same position. If you reorder, insert, or remove child pipeline calls before -existing ones, every subsequent ID shifts and previously completed children are -re-executed. Same caveat applies to step invocation IDs. +**Resume idempotency depends on submit order.** Child pipeline child runs are identified by the order of `child_pipeline(...)` / `child_pipeline.submit(...)` +calls in the parent body: the first call to `my_pipeline` becomes `pipeline:my_pipeline`, the second becomes `pipeline:my_pipeline_2`, and so on. +On resume, ZenML reuses an existing child run only if the same call appears in the same position. If you reorder, insert, or remove child pipeline calls before +existing ones, every subsequent ID shifts and previously completed children are re-executed. Same caveat applies to step invocation IDs. {% endhint %} ### Build, code, and Docker settings inheritance -Child runs share the parent's orchestration environment, image, and code -bundle. This has two consequences worth knowing: +Child runs share the parent's orchestration environment, image, and code bundle. This has two consequences worth knowing: -- **No new Docker build.** The child snapshot inherits the parent's `build`, - `code_reference`, and `code_path`. The child runs against the exact same +- **No new Docker build.** The child snapshot inherits the parent's `build`, `code_reference`, and `code_path`. The child runs against the exact same image and source bundle as the parent — there is no separate build step, and the child's code/dependencies must already be installed in the parent's image. -- **Pipeline-level Docker settings on the child are ignored.** When a child - pipeline (or a child step) declares non-default `docker_settings`, those +- **Pipeline-level Docker settings on the child are ignored.** When a child pipeline (or a child step) declares non-default `docker_settings`, those settings are silently overridden by the parent's. If you need a different image for a step inside a child pipeline, configure that step with a `step_operator` or use `runtime="isolated"` together with stack-level resource configuration on the parent. -This applies to all three call modes (`child(...)`, `child.submit(...)`, -and `child.embed(...)`). +This applies to all three call modes (`child(...)`, `child.submit(...)`, and `child.embed(...)`). ### Permissions and authentication for nested runs -Nested runs orchestrate from the parent's environment, so they share the -parent's API token. The token must be scoped to the **root** run of the -nesting tree — the root orchestrator can mint per-child-run tokens for any -descendant. Child runs cannot mint tokens for their siblings; only +Nested runs orchestrate from the parent's environment, so they share the parent's API token. The token must be scoped to the **root** run of the +nesting tree — the root orchestrator can mint per-child-run tokens for any descendant. Child runs cannot mint tokens for their siblings; only descendants of the same root tree are reachable from a given parent token. -This is transparent for the default flow (the root orchestrator launches -everything in the same environment). It matters if you build automation on -top of `ZENML_PIPELINE_RUN_ID` tokens — those tokens give you read/update -access to the run they were minted for and any of its descendants, but not +This is transparent for the default flow (the root orchestrator launches everything in the same environment). It matters if you build automation on +top of `ZENML_PIPELINE_RUN_ID` tokens — those tokens give you read/update access to the run they were minted for and any of its descendants, but not to siblings or unrelated runs. ### Config Templates with `depends_on` diff --git a/docs/book/how-to/steps-pipelines/hooks.md b/docs/book/how-to/steps-pipelines/hooks.md index b01e3ebd952..e808e0c4f5b 100644 --- a/docs/book/how-to/steps-pipelines/hooks.md +++ b/docs/book/how-to/steps-pipelines/hooks.md @@ -4,16 +4,13 @@ description: Run custom code on pipeline and step lifecycle events and record ho # Hooks -Hooks let you run custom code at lifecycle points of a run or a step, and record -those invocations as queryable `HookInvocation` records. The same machinery powers -the built-in lifecycle hooks ZenML fires for you and the public API you can call -from your own code. Common uses include sending notifications on success or +Hooks let you run custom code at lifecycle points of a run or a step, and record those invocations as queryable `HookInvocation` records. The same machinery powers +the built-in lifecycle hooks ZenML fires for you and the public API you can call from your own code. Common uses include sending notifications on success or failure, logging run details, and triggering external workflows. ## Lifecycle hooks -Lifecycle hooks fire automatically and each fire creates one `HookInvocation` -row. +Lifecycle hooks fire automatically and each fire creates one `HookInvocation` row. | Hook | Step scope | Pipeline scope (dynamic) | |---|---|---| @@ -28,8 +25,7 @@ Step-level hooks fire for both static and dynamic pipelines, uniformly. ### Static pipelines propagate, dynamic pipelines fire -The same `@pipeline(on_*=...)` kwarg has two different runtime meanings depending -on whether the pipeline is dynamic. +The same `@pipeline(on_*=...)` kwarg has two different runtime meanings depending on whether the pipeline is dynamic. | `@pipeline(on_*=X)` | Static pipeline | Dynamic pipeline | |---|---|---| @@ -40,16 +36,13 @@ on whether the pipeline is dynamic. | `on_pause` | Ignored | Fires once at the pipeline level | | `on_resume` | Ignored | Fires once at the pipeline level | -For a static pipeline, a pipeline-level hook is a default that every step inherits -where it has not set its own. No pipeline-level hooks are run. For a dynamic -pipeline, the hook fires once at the run level and produces `RUN_*` entries. It fires -on every run, including each invocation of a deployed pipeline. Dynamic pipeline +For a static pipeline, a pipeline-level hook is a default that every step inherits where it has not set its own. No pipeline-level hooks are run. For a dynamic +pipeline, the hook fires once at the run level and produces `RUN_*` entries. It fires on every run, including each invocation of a deployed pipeline. Dynamic pipeline users who want per-step defaults wire each `@step` directly. ## Registering hooks -Pass a callable or a source string to the decorator, `.configure(...)`, or -`.with_options(...)`. +Pass a callable or a source string to the decorator, `.configure(...)`, or `.with_options(...)`. ```python from zenml import step, pipeline @@ -84,16 +77,13 @@ def on_end(): ... def on_end(exception: Optional[BaseException] = None): ... ``` -`exception` is set only when the attempt or run failed. Details about the -current step or run are available through the step or run context (see +`exception` is set only when the attempt or run failed. Details about the current step or run are available through the step or run context (see below). ### Accessing step/run information in hooks -Step-scope hooks read the current step run and pipeline run through the step -context. Run-scope hooks on a dynamic pipeline fire outside any step and read -the run from the run context instead, via -`DynamicPipelineRunContext.get().run`. +Step-scope hooks read the current step run and pipeline run through the step context. Run-scope hooks on a dynamic pipeline fire outside any step and read +the run from the run context instead, via `DynamicPipelineRunContext.get().run`. ```python from zenml import get_step_context, step @@ -135,53 +125,41 @@ def my_step(): * **Retries.** A retried step fires one `on_start` / `on_end` pair per attempt. `on_success` and `on_failure` fire exactly once, at the terminal outcome. -* **Cache hits.** A cached step fires no step-level hooks. Pipeline-level hooks on - a dynamic run still fire even when every step was cached. -* **Hook failures are swallowed.** When a lifecycle hook raises, the run or step is - not aborted. The exception is captured into the `HookInvocation` record with +* **Cache hits.** A cached step fires no step-level hooks. Pipeline-level hooks on a dynamic run still fire even when every step was cached. +* **Hook failures are swallowed.** When a lifecycle hook raises, the run or step is not aborted. The exception is captured into the `HookInvocation` record with `status=FAILED` and execution proceeds. -* **Async hooks.** Hook functions can be defined with `async def`. When the hook - fires, ZenML runs the coroutine to completion and blocks until it finishes. -* **Return values are discarded.** Set `ZENML_TRACK_LIFECYCLE_HOOK_OUTPUTS=true` - in the execution environment to instead materialize lifecycle hook return +* **Async hooks.** Hook functions can be defined with `async def`. When the hook fires, ZenML runs the coroutine to completion and blocks until it finishes. +* **Return values are discarded.** Set `ZENML_TRACK_LIFECYCLE_HOOK_OUTPUTS=true` in the execution environment to instead materialize lifecycle hook return values as output artifacts of the invocation, following the same rules as `store_return=True` on `run_hook` (see below). ## Init and cleanup hooks -`on_init` and `on_cleanup` are pipeline setup and teardown hooks. They initialize -and tear down shared run state rather than reacting to a single run or step +`on_init` and `on_cleanup` are pipeline setup and teardown hooks. They initialize and tear down shared run state rather than reacting to a single run or step outcome, so they follow different rules from the lifecycle hooks above. They are -**not** recorded as `HookInvocation` records. When `on_init` fails, the run still -records `RUN_START`, `RUN_END`, and `RUN_FAILURE`, but never a row for `on_init` +**not** recorded as `HookInvocation` records. When `on_init` fails, the run still records `RUN_START`, `RUN_END`, and `RUN_FAILURE`, but never a row for `on_init` itself. Find the root cause on `pipeline_run.exception_info`. -ZenML runs `on_init` **once per execution environment**, before any step body runs -in that environment, and `on_cleanup` once when that environment is torn down. +ZenML runs `on_init` **once per execution environment**, before any step body runs in that environment, and `on_cleanup` once when that environment is torn down. Where that lands depends on how the pipeline runs. ### Deployments -`on_init` runs once per deployment replica when the replica starts, and -`on_cleanup` once when it shuts down. Individual invocations of the deployed -pipeline reuse the initialized state and do not re-run either hook. Lifecycle -hooks like `on_start` and `on_end` still fire on every invocation. +`on_init` runs once per deployment replica when the replica starts, and `on_cleanup` once when it shuts down. Individual invocations of the deployed +pipeline reuse the initialized state and do not re-run either hook. Lifecycle hooks like `on_start` and `on_end` still fire on every invocation. ### Regular runs For a run that is not a deployment, `on_init` runs once per execution environment: -* **Dynamic pipeline:** once in the orchestrator environment, where the pipeline - function executes. -* **Any step that runs outside the orchestration environment (static or - dynamic):** once ahead of that step body, the first time its environment is +* **Dynamic pipeline:** once in the orchestrator environment, where the pipeline function executes. +* **Any step that runs outside the orchestration environment (static or dynamic):** once ahead of that step body, the first time its environment is used. A step that shares the orchestration environment skips the hook, because the run context is already initialized there. ## Recording custom invocations -Beyond the built-in lifecycle hooks, you can record arbitrary invocations from -inside a step or a dynamic pipeline function. This is useful for instrumenting +Beyond the built-in lifecycle hooks, you can record arbitrary invocations from inside a step or a dynamic pipeline function. This is useful for instrumenting third-party callbacks such as the tool and model calls of an agent framework. ### `run_hook` @@ -201,8 +179,7 @@ def agent_step(): result = run_hook(call_tool, "search") ``` -Pass `store_return=True` to materialize the return value as an output artifact. A -single unannotated return becomes one artifact named `output`. An annotated tuple +Pass `store_return=True` to materialize the return value as an output artifact. A single unannotated return becomes one artifact named `output`. An annotated tuple return unpacks into one artifact per element. ```python diff --git a/docs/book/how-to/steps-pipelines/logging.md b/docs/book/how-to/steps-pipelines/logging.md index fc6c23e0f74..60caba9ffbe 100644 --- a/docs/book/how-to/steps-pipelines/logging.md +++ b/docs/book/how-to/steps-pipelines/logging.md @@ -292,8 +292,7 @@ ZENML_DISABLE_STEP_NAMES_IN_LOGS=true ### on Steps and pipelines -When running steps and pipelines, ZenML only captures logs emitted from the -thread that executes the corresponding function. If your step code spawns additional +When running steps and pipelines, ZenML only captures logs emitted from the thread that executes the corresponding function. If your step code spawns additional threads or runs async code, logs from those execution contexts may not be captured. For instance, only the log emitted directly in the step function is captured: @@ -344,14 +343,11 @@ def async_step() -> None: ### on the Dashboard -When viewing logs in the dashboard, ZenML currently loads logs **in bulk** and -pagination/filtering happens on the client side. To keep the response size and -server memory usage bounded (especially when logs are stored in remote artifact -stores), the dashboard is limited to **500 pages** (**100 log entries per +When viewing logs in the dashboard, ZenML currently loads logs **in bulk** and pagination/filtering happens on the client side. To keep the response size and +server memory usage bounded (especially when logs are stored in remote artifact stores), the dashboard is limited to **500 pages** (**100 log entries per page**, i.e. **50,000 entries** total) by default. -You can adjust this limit by setting `ZENML_LOGS_MAX_ENTRIES_PER_REQUEST` in the -environment when you are deploying your ZenML workspace. +You can adjust this limit by setting `ZENML_LOGS_MAX_ENTRIES_PER_REQUEST` in the environment when you are deploying your ZenML workspace. Downloading logs from the dashboard will also only include up to this limit. diff --git a/docs/book/how-to/steps-pipelines/scheduling.md b/docs/book/how-to/steps-pipelines/scheduling.md index f3631e871fb..7141efd37ff 100644 --- a/docs/book/how-to/steps-pipelines/scheduling.md +++ b/docs/book/how-to/steps-pipelines/scheduling.md @@ -30,8 +30,7 @@ Schedules don't work for all orchestrators. Here is a list of all supported orch **Native Schedule Management** means the orchestrator supports updating and deleting schedules directly through ZenML commands. When supported, commands like `zenml pipeline schedule update` and `zenml pipeline schedule delete` will automatically update/delete the schedule on the orchestrator platform (e.g., Kubernetes CronJobs). For orchestrators without this support, you'll need to manually manage schedules on the orchestrator side. {% endhint %} -Check out [our tutorial on -scheduling](https://docs.zenml.io/user-guides/tutorial/managing-scheduled-pipelines) +Check out [our tutorial on scheduling](https://docs.zenml.io/user-guides/tutorial/managing-scheduled-pipelines) for a practical guide on how to schedule a pipeline. ### Set a schedule @@ -120,8 +119,7 @@ archived_schedules = Client().list_schedules(is_archived=True) #### Archived Object Naming After archival, you may notice that the schedule name has changed. In ZenML, schedule names act as unique identifiers. -To prevent naming conflicts and allow reuse of the original name, archived schedules are automatically renamed by -appending a random hash to the original name. +To prevent naming conflicts and allow reuse of the original name, archived schedules are automatically renamed by appending a random hash to the original name. #### Permanent Deletion (Hard Delete) @@ -132,8 +130,7 @@ Archived objects can be permanently deleted using one of the following identifie * Name (updated/archived name only) {% hint style="warning" %} -Deletion using the original (pre-archival) name will not work, since the object has been renamed. Deletion by name -prefix is not supported for schedules (and most other entities), as it is considered unsafe. +Deletion using the original (pre-archival) name will not work, since the object has been renamed. Deletion by name prefix is not supported for schedules (and most other entities), as it is considered unsafe. {% endhint %} ### Kubernetes CronJob advanced configuration diff --git a/docs/book/how-to/steps-pipelines/streaming_events.md b/docs/book/how-to/steps-pipelines/streaming_events.md index 0fc3aec0b15..10a08d1c5a7 100644 --- a/docs/book/how-to/steps-pipelines/streaming_events.md +++ b/docs/book/how-to/steps-pipelines/streaming_events.md @@ -4,31 +4,22 @@ description: Publish live events from inside a step to subscribed clients. # Streaming events -ZenML pipelines can send live events from inside a running step to any -subscriber listening on the server. Use it for LLM token streaming, -progress updates from long-running steps, real-time dashboards, and -similar cases where you need to surface intermediate output before the +ZenML pipelines can send live events from inside a running step to any subscriber listening on the server. Use it for LLM token streaming, +progress updates from long-running steps, real-time dashboards, and similar cases where you need to surface intermediate output before the step returns. -This page covers the producer-side Python API — calling -`zenml.streaming.publish()` from inside a step. For enabling streaming on -the server and the HTTP/SSE wire contract, see -[Live event streaming](../../getting-started/deploying-zenml/live-event-streaming.md). +This page covers the producer-side Python API — calling `zenml.streaming.publish()` from inside a step. For enabling streaming on +the server and the HTTP/SSE wire contract, see [Live event streaming](../../getting-started/deploying-zenml/live-event-streaming.md). {% hint style="info" %} -Streaming is off by default and must be enabled on the server. Pipelines -that don't call `zenml.streaming.publish()` are unaffected. Once a server -returns `501 Not Implemented`, the producer self-mutes for the rest of -the process and further `publish()` calls return without sending HTTP. +Streaming is off by default and must be enabled on the server. Pipelines that don't call `zenml.streaming.publish()` are unaffected. Once a server +returns `501 Not Implemented`, the producer self-mutes for the rest of the process and further `publish()` calls return without sending HTTP. {% endhint %} {% hint style="warning" %} -Streaming is best-effort, not persistent storage. Events are size-capped, -can be dropped under load, and disappear when the broker's retention -window elapses. **Once an event is lost it's gone — ZenML keeps no -secondary copy.** If you need to keep something, write it as -[run metadata](../metadata/metadata.md) or an -[artifact](../artifacts/artifacts.md). +Streaming is best-effort, not persistent storage. Events are size-capped, can be dropped under load, and disappear when the broker's retention +window elapses. **Once an event is lost it's gone — ZenML keeps no secondary copy.** If you need to keep something, write it as +[run metadata](../metadata/metadata.md) or an [artifact](../artifacts/artifacts.md). {% endhint %} ## Publish from inside a step @@ -46,10 +37,8 @@ def my_streaming_step() -> str: return "ok" ``` -`publish()` reads the current pipeline run and step from the step -context — inside `@step`-decorated functions you don't need to pass any -handle. The call does not block; events are queued and a background -thread sends them to the server in small batches. +`publish()` reads the current pipeline run and step from the step context — inside `@step`-decorated functions you don't need to pass any +handle. The call does not block; events are queued and a background thread sends them to the server in small batches. ## The `publish()` API @@ -65,38 +54,31 @@ publish( ) -> None ``` -- `payload`: any JSON-serializable dict. Each event is limited to 64 KiB - on the wire envelope; the check runs inside `publish()` so oversize +- `payload`: any JSON-serializable dict. Each event is limited to 64 KiB on the wire envelope; the check runs inside `publish()` so oversize payloads fail locally rather than after an HTTP round-trip. - `kind`: a free-form label that clients can filter on. -- `correlation_id` / `index`: opaque to ZenML — passed through to - consumers unchanged so clients can group or order events that belong +- `correlation_id` / `index`: opaque to ZenML — passed through to consumers unchanged so clients can group or order events that belong to the same logical sub-flow (for example, one `correlation_id` per LLM generation, or one per tool call). Consumers can filter by `correlation_id` on the SSE endpoint. ### Picking a `kind` -`kind` is also the SSE `event:` field on the wire — clients subscribe -with `addEventListener("token", ...)`. A small set of stable names that +`kind` is also the SSE `event:` field on the wire — clients subscribe with `addEventListener("token", ...)`. A small set of stable names that your consumers code against is easier to maintain than ad-hoc labels. -Common choices: `token` (LLM streaming), `progress` (step progress), -`status` (state changes), `log` (free-form log lines). +Common choices: `token` (LLM streaming), `progress` (step progress), `status` (state changes), `log` (free-form log lines). ### Grouping with `correlation_id` -When a single step emits events for multiple parallel sub-flows — for -example, an agent that issues several LLM generations in parallel — use -`correlation_id` to tag each event so consumers can separate them. The -field is opaque to ZenML (no validation, no interpretation); the only +When a single step emits events for multiple parallel sub-flows — for example, an agent that issues several LLM generations in parallel — use +`correlation_id` to tag each event so consumers can separate them. The field is opaque to ZenML (no validation, no interpretation); the only contract is that consumers can filter on equality. ```python publish({"text": chunk}, kind="token", correlation_id="gen-42", index=i) ``` -A consumer can then subscribe with `?correlation_ids=gen-42` to receive -only that generation's tokens. +A consumer can then subscribe with `?correlation_ids=gen-42` to receive only that generation's tokens. ## `flush()` — wait for delivery @@ -106,21 +88,16 @@ from zenml.streaming import flush published = flush(timeout=2.0) # True if the queue drained ``` -A background worker drains the queue continuously, and `atexit` catches -process termination, so most users don't need to call `flush()`. Call it -when you need to be sure a specific event has reached the server before -doing something else — for example, posting a "ready" event right -before sending an external webhook whose target will consume the -stream. +A background worker drains the queue continuously, and `atexit` catches process termination, so most users don't need to call `flush()`. Call it +when you need to be sure a specific event has reached the server before doing something else — for example, posting a "ready" event right +before sending an external webhook whose target will consume the stream. ## Inside dynamic pipelines -Inside the body of a `@pipeline(dynamic=True)` (outside any `@step`), -`publish()` attributes events to the pipeline run with no `step_run_id` +Inside the body of a `@pipeline(dynamic=True)` (outside any `@step`), `publish()` attributes events to the pipeline run with no `step_run_id` or `step_name`. See [Dynamic Pipelines](dynamic_pipelines.md). -Calls to `publish()` made outside any pipeline or step context (for -example, module-level code, a REPL, scripts) are dropped after a +Calls to `publish()` made outside any pipeline or step context (for example, module-level code, a REPL, scripts) are dropped after a `debug`-level log line — there is no run to attribute the event to. ## Producer-side limits @@ -135,6 +112,5 @@ example, module-level code, a REPL, scripts) are dropped after a larger values cause every send to fail validation server-side. - Publishing requires `UPDATE` permission on the run. -For the wire format, delivery semantics, the consumer protocol, and -server configuration, see +For the wire format, delivery semantics, the consumer protocol, and server configuration, see [Live event streaming](../../getting-started/deploying-zenml/live-event-streaming.md). diff --git a/docs/book/how-to/steps-pipelines/wait_resume.md b/docs/book/how-to/steps-pipelines/wait_resume.md index 6300f88651d..181301f4593 100644 --- a/docs/book/how-to/steps-pipelines/wait_resume.md +++ b/docs/book/how-to/steps-pipelines/wait_resume.md @@ -7,8 +7,7 @@ icon: hourglass-half Use `zenml.wait(...)` when a dynamic pipeline needs a human or external system to provide input before it can continue. -{% hint style="info" %} -`zenml.wait(...)` only works inside [dynamic pipelines](./dynamic_pipelines.md). It does not work in static pipelines or inside a step. +{% hint style="info" %} `zenml.wait(...)` only works inside [dynamic pipelines](./dynamic_pipelines.md). It does not work in static pipelines or inside a step. {% endhint %} ## Basic pattern @@ -81,10 +80,8 @@ def deployment_pipeline() -> None: ## Agentic approval workflow -`wait(...)` is useful when an agentic workflow should prepare a recommendation -but a human should approve the final action. The example below plans multiple -agent tasks, runs them with dynamic mapping, summarizes the results, and then -continues only if the approval wait condition resolves to `true`. +`wait(...)` is useful when an agentic workflow should prepare a recommendation but a human should approve the final action. The example below plans multiple +agent tasks, runs them with dynamic mapping, summarizes the results, and then continues only if the approval wait condition resolves to `true`. ```python from zenml import pipeline, step, wait @@ -130,41 +127,30 @@ def agentic_approval_pipeline(goal: str) -> None: take_final_action(summary) ``` -For a runnable version that also logs metadata and returns tabular artifacts, -see the +For a runnable version that also logs metadata and returns tabular artifacts, see the [`agentic_hitl_pipeline` example](https://github.com/zenml-io/zenml/tree/main/examples/agentic_hitl_pipeline). ## Timeouts and pausing -`wait(...)` accepts a `timeout` (default: 600 seconds). When the timeout -elapses without a resolution, ZenML transitions the run to `PAUSED` so the -orchestration process can be torn down — picking it back up later via -[resume](#resolve-and-resume). +`wait(...)` accepts a `timeout` (default: 600 seconds). When the timeout elapses without a resolution, ZenML transitions the run to `PAUSED` so the +orchestration process can be torn down — picking it back up later via [resume](#resolve-and-resume). -There is one subtlety to know about for nested or concurrent dynamic -pipelines: the run only pauses once all *tree-wide* work has settled. +There is one subtlety to know about for nested or concurrent dynamic pipelines: the run only pauses once all *tree-wide* work has settled. -- A run with concurrent steps (`step.submit(...)`), maps - (`step.map(...)`), or child pipelines (`child(...)`, +- A run with concurrent steps (`step.submit(...)`), maps (`step.map(...)`), or child pipelines (`child(...)`, `child.submit(...)`) keeps the orchestration process up so it can monitor that work. -- While that process is up anyway, pausing the run wouldn't free any - resources — the wait keeps polling and refreshing its lease even past +- While that process is up anyway, pausing the run wouldn't free any resources — the wait keeps polling and refreshing its lease even past `timeout`. -- Once the tree quiesces (concurrent steps and child runs have all - finished, or themselves paused), the wait gives up and publishes +- Once the tree quiesces (concurrent steps and child runs have all finished, or themselves paused), the wait gives up and publishes `PAUSED`. -In other words, `timeout` is the earliest moment the run can pause, not -a hard upper bound on how long the wait blocks. For a run with no -concurrent or nested work it behaves as the simple bound it appears to -be. For richer pipelines, treat it as a "stop polling once everything +In other words, `timeout` is the earliest moment the run can pause, not a hard upper bound on how long the wait blocks. For a run with no +concurrent or nested work it behaves as the simple bound it appears to be. For richer pipelines, treat it as a "stop polling once everything else is also idle" hint. -If you want the run to pause as soon as the timeout elapses regardless of -sibling work, design the pipeline so the `wait(...)` call is the only -in-flight work at that point — for example, by `.wait()`-ing on the -relevant futures before calling `wait(...)`. +If you want the run to pause as soon as the timeout elapses regardless of sibling work, design the pipeline so the `wait(...)` call is the only +in-flight work at that point — for example, by `.wait()`-ing on the relevant futures before calling `wait(...)`. ## Resolve and resume @@ -189,18 +175,13 @@ In ZenML Pro, the run should start resuming automatically after the wait conditi zenml pipeline runs resume ``` -If manual resume fails, check the error message before retrying. The common -causes are concrete: +If manual resume fails, check the error message before retrying. The common causes are concrete: -- The run is a child run. Resume the parent run ID shown in the error instead; - the parent is the run that can safely continue the whole nested execution +- The run is a child run. Resume the parent run ID shown in the error instead; the parent is the run that can safely continue the whole nested execution tree. -- The run is not currently `PAUSED`. A run that is still running, already - completed, failed, or stopped cannot be resumed with this command. -- The run still has an active wait condition. Resolve the wait condition first, - then resume the run. -- ZenML can no longer find the snapshot or stack that the paused run needs in - order to continue. In that case, the run cannot be resumed until the missing +- The run is not currently `PAUSED`. A run that is still running, already completed, failed, or stopped cannot be resumed with this command. +- The run still has an active wait condition. Resolve the wait condition first, then resume the run. +- ZenML can no longer find the snapshot or stack that the paused run needs in order to continue. In that case, the run cannot be resumed until the missing reference is restored or recreated.
ZenML Scarf
diff --git a/docs/book/how-to/tags/tags.md b/docs/book/how-to/tags/tags.md index 8b9b992cb38..9fa3086ff53 100644 --- a/docs/book/how-to/tags/tags.md +++ b/docs/book/how-to/tags/tags.md @@ -5,8 +5,7 @@ icon: tag # Tags -Organizing and categorizing your machine learning artifacts and models can\ -streamline your workflow and enhance discoverability. ZenML enables the use of\ +Organizing and categorizing your machine learning artifacts and models can\ streamline your workflow and enhance discoverability. ZenML enables the use of\ tags as a flexible tool to classify and filter your ML assets. ![Tags are visible in the ZenML Dashboard](../../.gitbook/assets/tags-in-dashboard.png) @@ -31,8 +30,7 @@ zenml artifact update my_artifact -t my_tag ### Assigning tags to artifact versions -In order to tag an artifact through the Python SDK, you can use either use\ -the `ArtifactConfig` object: +In order to tag an artifact through the Python SDK, you can use either use\ the `ArtifactConfig` object: ```python from typing import Annotated diff --git a/docs/book/reference/environment-variables.md b/docs/book/reference/environment-variables.md index 08225599d2e..c56187b6ac6 100644 --- a/docs/book/reference/environment-variables.md +++ b/docs/book/reference/environment-variables.md @@ -97,8 +97,7 @@ export ZENML_ENABLE_RICH_TRACEBACK=true ## Disable colorful logging -Console logs use colors by default. If you wish to disable colorful logging, -set the following environment variable: +Console logs use colors by default. If you wish to disable colorful logging, set the following environment variable: ```bash ZENML_LOGGING_COLORS_DISABLED=true @@ -171,10 +170,8 @@ ZENML_SKIP_STACK_VALIDATION=true ## Ignore untracked code repository files When using [code repositories](https://docs.zenml.io/concepts/code-repositories), -ZenML will by default require the local checkout to have no uncommitted or untracked files -in order to use the code repository to track the commit and download files. If you want to ignore untracked files, you can set -the `ZENML_CODE_REPOSITORY_IGNORE_UNTRACKED_FILES` environment variable to `True`. When doing this, you're responsible that -the files committed to the repository includes everything necessary to run your pipeline. +ZenML will by default require the local checkout to have no uncommitted or untracked files in order to use the code repository to track the commit and download files. If you want to ignore untracked files, you can set +the `ZENML_CODE_REPOSITORY_IGNORE_UNTRACKED_FILES` environment variable to `True`. When doing this, you're responsible that the files committed to the repository includes everything necessary to run your pipeline. ## ZenML global config path diff --git a/docs/book/reference/llms-txt.md b/docs/book/reference/llms-txt.md index 43c481d28f5..4551c6a8772 100644 --- a/docs/book/reference/llms-txt.md +++ b/docs/book/reference/llms-txt.md @@ -24,8 +24,7 @@ ZenML's documentation is now made available to LLMs at the following link: https://docs.zenml.io/llms.txt ``` -This file contains a comprehensive summary of the ZenML documentation -(containing links and descriptions) that LLMs can use to answer questions about +This file contains a comprehensive summary of the ZenML documentation (containing links and descriptions) that LLMs can use to answer questions about ZenML's features, functionality, and usage. ## How to use the llms.txt file @@ -82,8 +81,7 @@ The MCP server indexes the latest released documentation, not the develop branch Prefer the native GitBook MCP server above for the best experience; if you prefer working directly with llms.txt or need alternative workflows, the following tools are helpful: -To use the llms.txt file in partnership with an MCP client, you can use the -following tools: +To use the llms.txt file in partnership with an MCP client, you can use the following tools: - [GitMCP](https://gitmcp.io/) - A way to quickly create an MCP server for a github repository (e.g. for `zenml-io/zenml`) - [mcp-llms](https://github.com/parlance-labs/mcp-llms.txt/) - This shows how to use an MCP server to iteratively explore the llms.txt file with your MCP client diff --git a/docs/book/user-guide/README.md b/docs/book/user-guide/README.md index 4653c1e6782..752142c6c31 100644 --- a/docs/book/user-guide/README.md +++ b/docs/book/user-guide/README.md @@ -16,15 +16,13 @@ Step-by-step instructions to help you master ZenML concepts and features. ## Projects -Complete end-to-end implementations that showcase ZenML in real-world scenarios.\ -[See all projects in our website →](https://www.zenml.io/projects) +Complete end-to-end implementations that showcase ZenML in real-world scenarios.\ [See all projects in our website →](https://www.zenml.io/projects)
FloraCastA production-ready MLOps pipeline for time series forecasting using ZenML and Darts, featuring TFT-based training and scheduled batch inference.zencoder.jpghttps://www.zenml.io/projects/floracast
LLM-Complete GuideProduction-ready RAG pipelines from basic retrieval to advanced LLMOps with embeddings finetuning and evals.llm-complete-guide.jpghttps://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide
Retail ForecastA robust MLOps pipeline for retail sales forecasting designed for retail data scientists and ML engineers.nightwatch.jpghttps://www.zenml.io/projects/retail-forecast
Research RadarAutomates research paper discovery and classification for specialized research domains.researchradar.jpg
OncoClearA production-ready MLOps pipeline for accurate breast cancer classification using machine learning.magicphoto.jpghttps://www.zenml.io/projects/oncoclear
Sign Language Detection with YOLOv5End-to-end computer vision pipelineyolo.jpghttps://www.zenml.io/projects/sign-language-detection-with-yolov5
ZenML Support AgentA production-ready agent that can help you with your ZenML questions.support.jpghttps://www.zenml.io/projects/zenml-support-agent
OmniReaderA scalable multi-model OCR workflow framework for batch document processing and model evaluation.gamesense.jpghttps://www.zenml.io/projects/omnireader
EuroRate PredictorTurn European Central Bank data into actionable interest rate forecasts with this comprehensive MLOps solution.eurorate.jpghttps://www.zenml.io/projects/eurorate-predictor
## Examples -Focused code snippets and templates that address specific ML workflow challenges.\ -[See all examples in GitHub →](https://github.com/zenml-io/zenml-projects) +Focused code snippets and templates that address specific ML workflow challenges.\ [See all examples in GitHub →](https://github.com/zenml-io/zenml-projects)
QuickstartBridging Local Development and Cloud Deploymentexample-01.pnghttps://github.com/zenml-io/zenml/blob/main/examples/quickstart
End-to-End Batch InferenceSupervised ML project built with the ZenML framework and its integration.example-02.pnghttps://github.com/zenml-io/zenml/tree/main/examples/e2e
Agent Architecture ComparisonCompare AI agents with LangGraph workflows, LiteLLM integration, and automatic visualizations.example-06.pnghttps://github.com/zenml-io/zenml/blob/main/examples/agent_comparison
Agent Framework IntegrationsProduction-ready integrations for 11 popular agent frameworks including LangChain, CrewAI, AutoGen, and more.example-06.pnghttps://github.com/zenml-io/zenml/tree/main/examples/agent_framework_integrations
Deploying AgentsDocument analysis service with pipelines, evaluation, and embedded web UI.example-07.pnghttps://github.com/zenml-io/zenml/blob/main/examples/deploying_agent
Agent Outer LoopAgent training and evaluation loop: evolve a generic agent into a specialized support system through intent classification and model training.example-07.pnghttps://github.com/zenml-io/zenml/blob/main/examples/agent_outer_loop
Basic NLP with BERTBuild NLP models with production-ready ML pipeline frameworkexample-03.pnghttps://github.com/zenml-io/zenml/tree/main/examples/e2e_nlp
Computer Vision with YoloV8End-to-end computer vision pipeline with modular designexample-04.pnghttps://github.com/zenml-io/zenml/tree/main/examples/computer_vision
LLM FinetuningLLM fine-tuning pipeline with PEFT approachexample-05.pnghttps://github.com/zenml-io/zenml/tree/main/examples/llm_finetuning
Durable Agent Platform (Kitaru)The runnable code behind the Agents guide: an internal agent harness platform with Kitaru and PydanticAI.https://github.com/zenml-io/kitaru/tree/develop/examples/end_to_end/agent_harness_platform
diff --git a/docs/book/user-guide/agents-guide/03-skills.md b/docs/book/user-guide/agents-guide/03-skills.md index 7880fe32d0d..eb3912286e0 100644 --- a/docs/book/user-guide/agents-guide/03-skills.md +++ b/docs/book/user-guide/agents-guide/03-skills.md @@ -111,8 +111,7 @@ The `SkillSource` seam lives in `agent_harness_platform/profile.py`, with these {% hint style="info" %} Related: the [`zenml-io/kitaru-skills`](https://github.com/zenml-io/kitaru-skills) -package provides reusable Markdown skills for coding agents, including Kitaru -quickstart, scoping, authoring, and adapter migration skills. See +package provides reusable Markdown skills for coding agents, including Kitaru quickstart, scoping, authoring, and adapter migration skills. See [Agent Skills](https://docs.zenml.io/kitaru/agent-native/claude-code-skill) for install details. {% endhint %} diff --git a/docs/book/user-guide/agents-guide/replay-and-improve.md b/docs/book/user-guide/agents-guide/replay-and-improve.md index ab2a7cf81fe..081fc3fdad8 100644 --- a/docs/book/user-guide/agents-guide/replay-and-improve.md +++ b/docs/book/user-guide/agents-guide/replay-and-improve.md @@ -114,8 +114,7 @@ report.summary() # per-metric baseline-vs-variant deltas + an "is it better report.regressions() # the metrics (and decision changes) that got worse ``` -{% hint style="info" %} -`run_cohort` and the metric callables live in the example, not in the `kitaru` package — they're a pattern built on the real `flow.replay` and `KitaruClient`. Copy or adapt them; metrics are just `metric(baseline, variant) -> MetricDelta` callables, so you bring your own. +{% hint style="info" %} `run_cohort` and the metric callables live in the example, not in the `kitaru` package — they're a pattern built on the real `flow.replay` and `KitaruClient`. Copy or adapt them; metrics are just `metric(baseline, variant) -> MetricDelta` callables, so you bring your own. {% endhint %} For each run, the cohort reproduces and replays the same way, then aggregates: total cost, latency, and a quality score from an LLM judge. You get the answer to the real question — does the cheaper model hold quality across realistic traffic, or only on the one run you happened to look at? diff --git a/docs/book/user-guide/best-practices/configure-python-environments.md b/docs/book/user-guide/best-practices/configure-python-environments.md index 2f6832b4f7c..a046da12989 100644 --- a/docs/book/user-guide/best-practices/configure-python-environments.md +++ b/docs/book/user-guide/best-practices/configure-python-environments.md @@ -56,10 +56,8 @@ ZenML allows you to install dependencies required by integrations through the `z ### Use a tool like `pip-compile` for reproducibility -Consider using a tool like `pip-compile` (available through [the `pip-tools` -package](https://pip-tools.readthedocs.io/)) to compile your dependencies into a -static `requirements.txt` file that can be used across environments. (If you are -using [`uv`](https://github.com/astral-sh/uv), you might want to use `uv pip compile` as an alternative.) +Consider using a tool like `pip-compile` (available through [the `pip-tools` package](https://pip-tools.readthedocs.io/)) to compile your dependencies into a +static `requirements.txt` file that can be used across environments. (If you are using [`uv`](https://github.com/astral-sh/uv), you might want to use `uv pip compile` as an alternative.) For a practical example and explanation of using `pip-compile` to address exactly this need, see [our 'gitflow' repository and workflow](https://github.com/zenml-io/zenml-gitflow#-software-requirements-management) to learn more. @@ -69,10 +67,8 @@ Running [`pip check`](https://pip.pypa.io/en/stable/cli/pip\_check/) will verify ### Well-known dependency resolution issues -Some of ZenML's integrations come with strict dependency and package version -requirements. We try to keep these dependency requirements ranges as wide as -possible for the integrations developed by ZenML, but it is not always possible -to make this work completely smoothly. Here is one of the known issues: +Some of ZenML's integrations come with strict dependency and package version requirements. We try to keep these dependency requirements ranges as wide as +possible for the integrations developed by ZenML, but it is not always possible to make this work completely smoothly. Here is one of the known issues: * `click`: ZenML currently requires `click~=8.0.3` for its CLI. This is on account of another dependency of ZenML. Using versions of `click` in your own project that are greater than 8.0.3 may cause unanticipated behaviors. @@ -84,8 +80,7 @@ It is possible to skip ZenML's integration installation process and install depe Note that the `zenml integration install ...` command runs a `pip install ...` under the hood as part of its implementation, taking the dependencies listed in the integration object and installing them. For example, `zenml integration install gcp` will run `pip install "kfp==1.8.16" "gcsfs" "google-cloud-secret-manager" ...` and so on, since they are [specified in the integration definition](https://github.com/zenml-io/zenml/blob/main/src/zenml/integrations/gcp/__init__.py#L46). {% endhint %} -To do this, you will need to install the dependencies for the integration you -want to use manually. You can find the dependencies for the integrations by +To do this, you will need to install the dependencies for the integration you want to use manually. You can find the dependencies for the integrations by running the following: ```bash diff --git a/docs/book/user-guide/best-practices/debug-and-solve-issues.md b/docs/book/user-guide/best-practices/debug-and-solve-issues.md index 921e3201287..50bc0beb778 100644 --- a/docs/book/user-guide/best-practices/debug-and-solve-issues.md +++ b/docs/book/user-guide/best-practices/debug-and-solve-issues.md @@ -83,10 +83,8 @@ Walk us through how to reproduce the same error you had step-by-step, whenever p #### 4. Relevant log output -As a general rule of thumb, always attach relevant log outputs and the full -error traceback to help us understand what happened under the hood. If the full -error traceback does not fit into a text message, attach a file or use a service -like Pastebin or [Github's Gist](https://gist.github.com/). +As a general rule of thumb, always attach relevant log outputs and the full error traceback to help us understand what happened under the hood. If the full +error traceback does not fit into a text message, attach a file or use a service like Pastebin or [Github's Gist](https://gist.github.com/). Along with the error traceback, we recommend to always share the output of the following commands: diff --git a/docs/book/user-guide/best-practices/iac.md b/docs/book/user-guide/best-practices/iac.md index 76973c7bb33..89527faf6c6 100644 --- a/docs/book/user-guide/best-practices/iac.md +++ b/docs/book/user-guide/best-practices/iac.md @@ -182,8 +182,7 @@ Different environments (dev, staging, prod) require: ### The Solution: Environment Configuration Pattern with Smart Authentication -Create a flexible [service connector](https://docs.zenml.io/stacks/service-connectors/auth-management) setup that adapts to your environment. For example, -in development, a service account might be the more flexible pattern, while in production we go through +Create a flexible [service connector](https://docs.zenml.io/stacks/service-connectors/auth-management) setup that adapts to your environment. For example, in development, a service account might be the more flexible pattern, while in production we go through workload identity. Combine environment-specific configurations with appropriate authentication methods: ```hcl diff --git a/docs/book/user-guide/best-practices/keep-your-dashboard-server-clean.md b/docs/book/user-guide/best-practices/keep-your-dashboard-server-clean.md index 9da4485089d..9c1aa550db9 100644 --- a/docs/book/user-guide/best-practices/keep-your-dashboard-server-clean.md +++ b/docs/book/user-guide/best-practices/keep-your-dashboard-server-clean.md @@ -27,8 +27,7 @@ If you want to delete a specific pipeline run, you can use a script like this: zenml pipeline runs delete ``` -If you want to delete all pipeline runs in the last 24 hours, for example, you\ -could run a script like this: +If you want to delete all pipeline runs in the last 24 hours, for example, you\ could run a script like this: ``` #!/usr/bin/env python3 @@ -78,8 +77,7 @@ This allows you to start fresh with a new pipeline, removing all previous runs a Pipelines can be given unique names each time they are run to uniquely identify them. This helps differentiate between multiple iterations of the same pipeline during development. -By default ZenML generates names automatically based on the current date and\ -time, but you can pass in a `run_name` when defining the pipeline: +By default ZenML generates names automatically based on the current date and\ time, but you can pass in a `run_name` when defining the pipeline: ```python training_pipeline = training_pipeline.with_options( @@ -106,8 +104,7 @@ See the full documentation on [how to delete models](https://docs.zenml.io/how-t ### Pruning artifacts -If you want to delete artifacts that are no longer referenced by any pipeline\ -runs, you can use the following CLI command: +If you want to delete artifacts that are no longer referenced by any pipeline\ runs, you can use the following CLI command: ```bash zenml artifact prune @@ -119,8 +116,7 @@ For more information, see the [documentation for this artifact pruning feature]( ## Cleaning your environment -As a more drastic measure, the `zenml clean` command can be used to start from\ -scratch on your local machine. This will: +As a more drastic measure, the `zenml clean` command can be used to start from\ scratch on your local machine. This will: * delete all pipelines, pipeline runs and associated metadata * delete all artifacts diff --git a/docs/book/user-guide/best-practices/mcp-chat-with-server.md b/docs/book/user-guide/best-practices/mcp-chat-with-server.md index 459a3130805..a78cf500d36 100644 --- a/docs/book/user-guide/best-practices/mcp-chat-with-server.md +++ b/docs/book/user-guide/best-practices/mcp-chat-with-server.md @@ -16,8 +16,7 @@ The Model Context Protocol (MCP) is an open protocol that standardizes how appli MCP follows a client-server architecture where: * **MCP Clients**: Programs like Claude Desktop or IDEs (Cursor, Windsurf, etc.) that want to access data through MCP -* **MCP Servers**: Lightweight programs that expose specific capabilities\ - through the standardized protocol. Our implementation is of an MCP server that connects to your ZenML server. +* **MCP Servers**: Lightweight programs that expose specific capabilities\ through the standardized protocol. Our implementation is of an MCP server that connects to your ZenML server. ## Why use MCP with ZenML? diff --git a/docs/book/user-guide/best-practices/project-templates.md b/docs/book/user-guide/best-practices/project-templates.md index 9c1a7075a7c..b2fad396f89 100644 --- a/docs/book/user-guide/best-practices/project-templates.md +++ b/docs/book/user-guide/best-practices/project-templates.md @@ -24,10 +24,8 @@ pip install 'zenml[templates]' ``` {% hint style="warning" %} -Note that these templates are not the same thing as the templates used for -triggering a pipeline (from the dashboard or via the Python SDK). Those are -known as 'Run Templates' and you can read more about them here. +Note that these templates are not the same thing as the templates used for triggering a pipeline (from the dashboard or via the Python SDK). Those are +known as 'Run Templates' and you can read more about them here. {% endhint %} Now, you can generate a project from one of the existing templates by using the `--template` flag with the `zenml init` command: diff --git a/docs/book/user-guide/best-practices/quick-wins.md b/docs/book/user-guide/best-practices/quick-wins.md index b4496095160..91037a9bc07 100644 --- a/docs/book/user-guide/best-practices/quick-wins.md +++ b/docs/book/user-guide/best-practices/quick-wins.md @@ -5,8 +5,7 @@ icon: bolt # Quick Wins -Below is a menu of 5-minute quick wins you can sprinkle into an existing ZenML -project with almost no code changes. Each entry explains why it matters, the +Below is a menu of 5-minute quick wins you can sprinkle into an existing ZenML project with almost no code changes. Each entry explains why it matters, the micro-setup (under 5 minutes) and any tips or gotchas to anticipate. {% hint style="info" %} @@ -160,10 +159,8 @@ def train_model(data): ## 4 Instant **alerter notifications** for successes/failures -**Why** -- get immediate notifications when pipelines succeed or fail, enabling -faster response times and improved collaboration. Alerter notifications ensure -your team is always aware of critical model training status, data drift alerts, -and deployment changes without constantly checking dashboards. +**Why** -- get immediate notifications when pipelines succeed or fail, enabling faster response times and improved collaboration. Alerter notifications ensure +your team is always aware of critical model training status, data drift alerts, and deployment changes without constantly checking dashboards. {% hint style="info" %} ZenML supports multiple alerter flavors including Slack and Discord. The example below uses Slack, but the pattern is similar for other alerters. diff --git a/docs/book/user-guide/best-practices/shared-components-for-teams.md b/docs/book/user-guide/best-practices/shared-components-for-teams.md index 92e155d3b3d..fe37f04f94d 100644 --- a/docs/book/user-guide/best-practices/shared-components-for-teams.md +++ b/docs/book/user-guide/best-practices/shared-components-for-teams.md @@ -97,8 +97,7 @@ def my_pipeline(...): ... ``` -The `requirements.txt` file would specify the private index URL in the following\ -way, for example: +The `requirements.txt` file would specify the private index URL in the following\ way, for example: ``` --extra-index-url https://YOURTOKEN@my-private-pypi-server.com/YOURUSERNAME/ @@ -131,8 +130,7 @@ As the project evolves, it's important to keep shared libraries updated with the Continuous integration (CI) helps maintain the stability and reliability of shared components. By automatically running tests and checks on each code change, CI can catch potential issues early and ensure compatibility across different environments and dependencies. -By leveraging these methods for sharing code and libraries, teams can\ -collaborate more effectively, maintain consistency across projects, and\ +By leveraging these methods for sharing code and libraries, teams can\ collaborate more effectively, maintain consistency across projects, and\ accelerate development processes within the ZenML framework.
ZenML Scarf
diff --git a/docs/book/user-guide/best-practices/vscode-extension.md b/docs/book/user-guide/best-practices/vscode-extension.md index 6e35c4886ed..f4c23478ac5 100644 --- a/docs/book/user-guide/best-practices/vscode-extension.md +++ b/docs/book/user-guide/best-practices/vscode-extension.md @@ -5,10 +5,8 @@ icon: computer-mouse # Using VS Code extension -The ZenML VSCode extension is a tool that allows you to manage your ZenML server\ -from within VSCode. It provides features for stack management, pipeline\ -visualization, and project management capabilities. You can use it in any IDE\ -which allows the installation of extensions from the VSCode Marketplace, which\ +The ZenML VSCode extension is a tool that allows you to manage your ZenML server\ from within VSCode. It provides features for stack management, pipeline\ +visualization, and project management capabilities. You can use it in any IDE\ which allows the installation of extensions from the VSCode Marketplace, which\ means that Cursor also supports this extension. ![ZenML VSCode Extension](../../.gitbook/assets/vscode-extension.gif) @@ -90,8 +88,7 @@ If you encounter issues with the extension: * Verify that your authentication credentials are correct * Try restarting VSCode -For more help, visit the [ZenML GitHub\ -repository](https://github.com/zenml-io/vscode-zenml) or send us a message on\ +For more help, visit the [ZenML GitHub\ repository](https://github.com/zenml-io/vscode-zenml) or send us a message on\ our [Slack community](https://zenml.io/slack).
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/evaluation/generation.md b/docs/book/user-guide/llmops-guide/evaluation/generation.md index 193bf7cfa15..bbde553f9b9 100644 --- a/docs/book/user-guide/llmops-guide/evaluation/generation.md +++ b/docs/book/user-guide/llmops-guide/evaluation/generation.md @@ -4,39 +4,28 @@ description: Evaluate the generation component of your RAG pipeline. # Generation evaluation -Now that we have a sense of how to evaluate the retrieval component of our RAG -pipeline, let's move on to the generation component. The generation component is -responsible for generating the answer to the question based on the retrieved -context. At this point, our evaluation starts to move into more subjective -territory. It's harder to come up with metrics that can accurately capture the -quality of the generated answers. However, there are some things we can do. +Now that we have a sense of how to evaluate the retrieval component of our RAG pipeline, let's move on to the generation component. The generation component is +responsible for generating the answer to the question based on the retrieved context. At this point, our evaluation starts to move into more subjective +territory. It's harder to come up with metrics that can accurately capture the quality of the generated answers. However, there are some things we can do. -As with the [retrieval evaluation](retrieval.md), we can start with a simple -approach and then move on to more sophisticated methods. +As with the [retrieval evaluation](retrieval.md), we can start with a simple approach and then move on to more sophisticated methods. ## Handcrafted evaluation tests -As in the retrieval evaluation, we can start by putting together a set of -examples where we know that our generated output should or shouldn't include -certain terms. For example, if we're generating answers to questions about -which orchestrators ZenML supports, we can check that the generated answers -include terms like "Airflow" and "Kubeflow" (since we do support them) and -exclude terms like "Flyte" or "Prefect" (since we don't (yet!) support them). -These handcrafted tests should be driven by mistakes that you've already seen in -the RAG output. The negative example of "Flyte" and "Prefect" showing up in the +As in the retrieval evaluation, we can start by putting together a set of examples where we know that our generated output should or shouldn't include +certain terms. For example, if we're generating answers to questions about which orchestrators ZenML supports, we can check that the generated answers +include terms like "Airflow" and "Kubeflow" (since we do support them) and exclude terms like "Flyte" or "Prefect" (since we don't (yet!) support them). +These handcrafted tests should be driven by mistakes that you've already seen in the RAG output. The negative example of "Flyte" and "Prefect" showing up in the list of supported orchestrators, for example, shows up sometimes when you use GPT 3.5 as the LLM. ![](../../../.gitbook/assets/generation-eval-manual.png) -As another example, when you make a query asking 'what is the default -orchestrator in ZenML?' you would expect that the answer would include the word +As another example, when you make a query asking 'what is the default orchestrator in ZenML?' you would expect that the answer would include the word 'local', so we can make a test case to confirm that. -You can view our starter set of these tests -[here](https://github.com/zenml-io/zenml-projects/blob/main/llm-complete-guide/steps/eval_e2e.py#L28-L55). -It's better to start with something small and simple and then expand as is -needed. There's no need for complicated harnesses or frameworks at this stage. +You can view our starter set of these tests [here](https://github.com/zenml-io/zenml-projects/blob/main/llm-complete-guide/steps/eval_e2e.py#L28-L55). +It's better to start with something small and simple and then expand as is needed. There's no need for complicated harnesses or frameworks at this stage. **`bad_answers` table:** @@ -87,14 +76,11 @@ def test_content_for_bad_words( return TestResult(success=True, question=question, response=response) ``` -Here we're testing that a particular word doesn't show up in the generated -response. If we find the word, then we return a failure, otherwise we return a -success. This is a simple example, but you can imagine more complex tests that -check for the presence of multiple words, or the presence of a word in a +Here we're testing that a particular word doesn't show up in the generated response. If we find the word, then we return a failure, otherwise we return a +success. This is a simple example, but you can imagine more complex tests that check for the presence of multiple words, or the presence of a word in a particular context. -We pass these custom tests into a test runner that keeps track of how many are -failing and also logs those to the console when they do: +We pass these custom tests into a test runner that keeps track of how many are failing and also logs those to the console when they do: ```python def run_tests(test_data: list, test_function: Callable) -> float: @@ -114,8 +100,7 @@ def run_tests(test_data: list, test_function: Callable) -> float: return round(failure_rate, 2) ``` -Our end-to-end evaluation of the generation component is then a combination of -these tests: +Our end-to-end evaluation of the generation component is then a combination of these tests: ```python @step @@ -153,37 +138,28 @@ def e2e_evaluation() -> ( ``` Running the tests using different LLMs will give different results. Here our -Ollama Mixtral did worse than GPT 3.5, for example, but there were still some -failures with GPT 3.5. This is a good way to get a sense of how well your +Ollama Mixtral did worse than GPT 3.5, for example, but there were still some failures with GPT 3.5. This is a good way to get a sense of how well your generation component is doing. -As you become more familiar with the kinds of outputs your LLM generates, you -can add the hard ones to this test suite. This helps prevent regressions and -is directly related to the quality of the output you're getting. This way you -can optimize for your specific use case. +As you become more familiar with the kinds of outputs your LLM generates, you can add the hard ones to this test suite. This helps prevent regressions and +is directly related to the quality of the output you're getting. This way you can optimize for your specific use case. ## Automated evaluation using another LLM -Another way to evaluate the generation component is to use another LLM to -grade the output of the LLM you're evaluating. This is a more sophisticated -approach and requires a bit more setup. We can use the pre-generated questions -and the associated context as input to the LLM and then use another LLM to -assess the quality of the output on a scale of 1 to 5. This is a more -quantitative approach and since it's automated it can run across a larger set of +Another way to evaluate the generation component is to use another LLM to grade the output of the LLM you're evaluating. This is a more sophisticated +approach and requires a bit more setup. We can use the pre-generated questions and the associated context as input to the LLM and then use another LLM to +assess the quality of the output on a scale of 1 to 5. This is a more quantitative approach and since it's automated it can run across a larger set of data. ![](../../../.gitbook/assets/generation-eval-automated.png) {% hint style="warning" %} LLMs don't always do well on this kind of evaluation where numbers are involved. -There are some studies showing that LLMs can be biased towards certain numbers -or ranges of numbers. This is something to keep in mind when using this -approach. Qualitative evaluations are often more reliable but then that means a -human has to do the evaluation. +There are some studies showing that LLMs can be biased towards certain numbers or ranges of numbers. This is something to keep in mind when using this +approach. Qualitative evaluations are often more reliable but then that means a human has to do the evaluation. {% endhint %} -We can start by setting up a Pydantic model to hold the data we need. We set -constraints to ensure that the data we're getting back are only integers between +We can start by setting up a Pydantic model to hold the data we need. We set constraints to ensure that the data we're getting back are only integers between 1 and 5, inclusive: ```python @@ -198,14 +174,11 @@ We can use this in a test function that: - takes a question and a context as inputs - generates an answer using the LLM we're evaluating -- makes a call to an (optionally different) LLM we're using to judge the quality - of the answer getting back a score for each of the four categories in JSON +- makes a call to an (optionally different) LLM we're using to judge the quality of the answer getting back a score for each of the four categories in JSON format -- parses the JSON and returns the result of the evaluation as our Pydantic - model instance +- parses the JSON and returns the result of the evaluation as our Pydantic model instance -Pydantic handles the validation of the JSON input for us, so we can be sure that -we're getting the data we expect and in a form that we can use. +Pydantic handles the validation of the JSON input for us, so we can be sure that we're getting the data we expect and in a form that we can use. ```python def llm_judged_test_e2e( @@ -264,8 +237,7 @@ def llm_judged_test_e2e( raise e ``` -Currently we're not handling retries of the output from the LLM in the case -where the JSON isn't output correctly, but potentially that's something we might +Currently we're not handling retries of the output from the LLM in the case where the JSON isn't output correctly, but potentially that's something we might want to do. We can then run this test across a set of questions and contexts: @@ -319,10 +291,8 @@ def run_llm_judged_tests( ) ``` -You'll want to use your most capable and reliable LLM to do the judging. In our -case, we used the new GPT-4 Turbo. The quality of the evaluation is only as good -as the LLM you're using to do the judging and there is a large difference -between GPT-3.5 and GPT-4 Turbo in terms of the quality of the output, not least +You'll want to use your most capable and reliable LLM to do the judging. In our case, we used the new GPT-4 Turbo. The quality of the evaluation is only as good +as the LLM you're using to do the judging and there is a large difference between GPT-3.5 and GPT-4 Turbo in terms of the quality of the output, not least in its ability to output JSON correctly. Here was the output following an evaluation for 50 randomly sampled datapoints: @@ -337,52 +307,40 @@ Step e2e_evaluation_llm_judged has finished in 8m51s. Pipeline run has finished in 8m52s. ``` -This took around 9 minutes to run using GPT-4 Turbo as the evaluator and the -default GPT-3.5 as the LLM being evaluated. +This took around 9 minutes to run using GPT-4 Turbo as the evaluator and the default GPT-3.5 as the LLM being evaluated. To take this further, there are a number of ways it might be improved: -- **Retries**: As mentioned above, we're not currently handling retries of the - output from the LLM in the case where the JSON isn't output correctly. This +- **Retries**: As mentioned above, we're not currently handling retries of the output from the LLM in the case where the JSON isn't output correctly. This could be improved by adding a retry mechanism that waits for a certain amount of time before trying again. (We could potentially use the [`instructor`](https://github.com/jxnl/instructor) library to handle this specifically.) -- **Use OpenAI's 'JSON mode'**: OpenAI has a [JSON - mode](https://platform.openai.com/docs/guides/text-generation/json-mode) that +- **Use OpenAI's 'JSON mode'**: OpenAI has a [JSON mode](https://platform.openai.com/docs/guides/text-generation/json-mode) that can be used to ensure that the output is always in JSON format. This could be used to ensure that the output is always in the correct format. -- **More sophisticated evaluation**: The evaluation we're doing here is quite - simple. We're just asking for a score in four categories. There are more +- **More sophisticated evaluation**: The evaluation we're doing here is quite simple. We're just asking for a score in four categories. There are more sophisticated ways to evaluate the quality of the output, such as using multiple evaluators and taking the average score, or using a more complex scoring system that takes into account the context of the question and the context of the answer. -- **Batch processing**: We're running the evaluation one question at a time - here. It would be more efficient to run the evaluation in batches to speed up +- **Batch processing**: We're running the evaluation one question at a time here. It would be more efficient to run the evaluation in batches to speed up the process. -- **More data**: We're only using 50 samples here. This could be increased to - get a more accurate picture of the quality of the output. -- **More LLMs**: We're only using GPT-4 Turbo here. It would be interesting to - see how other LLMs perform as evaluators. -- **Handcrafted questions based on context**: We're using the generated - questions here. It would be interesting to see how the LLM performs when given +- **More data**: We're only using 50 samples here. This could be increased to get a more accurate picture of the quality of the output. +- **More LLMs**: We're only using GPT-4 Turbo here. It would be interesting to see how other LLMs perform as evaluators. +- **Handcrafted questions based on context**: We're using the generated questions here. It would be interesting to see how the LLM performs when given handcrafted questions that are based on the context of the question. -- **Human in the loop**: The LLM actually provides qualitative feedback on the - output as well as the JSON scores. This data could be passed into an +- **Human in the loop**: The LLM actually provides qualitative feedback on the output as well as the JSON scores. This data could be passed into an annotation tool to get human feedback on the quality of the output. This would be a more reliable way to evaluate the quality of the output and would offer some insight into the kinds of mistakes the LLM is making. -Most notably, the scores we're currently getting are pretty high, so it would -make sense to pass in harder questions and be more specific in the judging -criteria. This will give us more room to improve as it is sure that the system -is not perfect. +Most notably, the scores we're currently getting are pretty high, so it would make sense to pass in harder questions and be more specific in the judging +criteria. This will give us more room to improve as it is sure that the system is not perfect. While this evaluation approach serves as a solid foundation, it's worth noting that there are other frameworks available that can further enhance the evaluation process. Frameworks such as [`ragas`](https://github.com/explodinggradients/ragas), [`trulens`](https://www.trulens.org/), [DeepEval](https://docs.confident-ai.com/), and [UpTrain](https://github.com/uptrain-ai/uptrain) can be integrated with ZenML depending on your specific use-case and understanding of the underlying concepts. These frameworks, although potentially complex to set up and use, can provide more sophisticated evaluation capabilities as your project evolves and grows in complexity. -We now have a working evaluation of both the retrieval and generation evaluation -components of our RAG pipeline. We can use this to track how our pipeline +We now have a working evaluation of both the retrieval and generation evaluation components of our RAG pipeline. We can use this to track how our pipeline improves as we make changes to the retrieval and generation components. ## Code Example diff --git a/docs/book/user-guide/llmops-guide/evaluation/retrieval.md b/docs/book/user-guide/llmops-guide/evaluation/retrieval.md index 7b39155089b..7049262da51 100644 --- a/docs/book/user-guide/llmops-guide/evaluation/retrieval.md +++ b/docs/book/user-guide/llmops-guide/evaluation/retrieval.md @@ -4,37 +4,26 @@ description: See how the retrieval component responds to changes in the pipeline # Retrieval evaluation -The retrieval component of our RAG pipeline is responsible for finding relevant -documents or document chunks to feed into the generation component. In this -section we'll explore how to evaluate the performance of the retrieval component -of your RAG pipeline. We're checking how accurate the semantic search is, or in +The retrieval component of our RAG pipeline is responsible for finding relevant documents or document chunks to feed into the generation component. In this +section we'll explore how to evaluate the performance of the retrieval component of your RAG pipeline. We're checking how accurate the semantic search is, or in other words how relevant the retrieved documents are to the query. -Our retrieval component takes the incoming query and converts it into a -vector or embedded representation that can be used to search for relevant -documents. We then use this representation to search through a corpus of -documents and retrieve the most relevant ones. +Our retrieval component takes the incoming query and converts it into a vector or embedded representation that can be used to search for relevant +documents. We then use this representation to search through a corpus of documents and retrieve the most relevant ones. ## Manual evaluation using handcrafted queries -The most naive and simple way to check this would be to handcraft some queries -where we know the specific documents needed to answer it. We can then check if -the retrieval component is able to retrieve these documents. This is a manual -evaluation process and can be time-consuming, but it's a good way to get a sense -of how well the retrieval component is working. It can also be useful to target -known edge cases or difficult queries to see how the retrieval component handles +The most naive and simple way to check this would be to handcraft some queries where we know the specific documents needed to answer it. We can then check if +the retrieval component is able to retrieve these documents. This is a manual evaluation process and can be time-consuming, but it's a good way to get a sense +of how well the retrieval component is working. It can also be useful to target known edge cases or difficult queries to see how the retrieval component handles those known scenarios. ![](../../../.gitbook/assets/retrieval-eval-manual.png) -Implementing this is pretty simple - you just need to create some queries and -check the retrieved documents. Having tested the basic inference of our RAG -setup quite a bit, there were some clear areas where the retrieval component -could be improved. I looked in our documentation to find some examples where the -information could only be found in a single page and then wrote some queries -that would require the retrieval component to find that page. For example, the -query "How do I get going with the Label Studio integration? What are the first -steps?" would require the retrieval component to find [the Label Studio integration page](https://docs.zenml.io/stacks/annotators/label-studio). +Implementing this is pretty simple - you just need to create some queries and check the retrieved documents. Having tested the basic inference of our RAG +setup quite a bit, there were some clear areas where the retrieval component could be improved. I looked in our documentation to find some examples where the +information could only be found in a single page and then wrote some queries that would require the retrieval component to find that page. For example, the +query "How do I get going with the Label Studio integration? What are the first steps?" would require the retrieval component to find [the Label Studio integration page](https://docs.zenml.io/stacks/annotators/label-studio). Some of the other examples used are: | Question | URL Ending | @@ -45,10 +34,8 @@ Some of the other examples used are: | How do I use failure hooks in my ZenML pipeline? | user-guide/advanced-guide/pipelining-features/use-failure-success-hooks | | Can I deploy ZenML self-hosted with Helm? How do I do it? | deploying-zenml/zenml-self-hosted/deploy-with-helm | -For the retrieval pipeline, all we have to do is encode the query as a vector -and then query the PostgreSQL database for the most similar vectors. We then -check whether the URL for the document we thought must show up is actually -present in the top `n` results. +For the retrieval pipeline, all we have to do is encode the query as a vector and then query the PostgreSQL database for the most similar vectors. We then +check whether the URL for the document we thought must show up is actually present in the top `n` results. ```python def query_similar_docs(question: str, url_ending: str) -> tuple: @@ -79,11 +66,9 @@ def test_retrieved_docs_retrieve_best_url(question_doc_pairs: list) -> float: return round(failure_rate, 2) ``` -We include some logging so that when running the pipeline locally we can get -some immediate feedback logged to the console. +We include some logging so that when running the pipeline locally we can get some immediate feedback logged to the console. -This functionality can then be packaged up into a ZenML step once we're happy it -does what we need: +This functionality can then be packaged up into a ZenML step once we're happy it does what we need: ```python @step @@ -93,15 +78,11 @@ def retrieval_evaluation_small() -> Annotated[float, "small_failure_rate_retriev return failure_rate ``` -We got a 20% failure rate on the first run of this test, which was a good sign -that the retrieval component could be improved. We only had 5 test cases, so -this was just a starting point. In reality, you'd want to keep adding more test -cases to cover a wider range of scenarios. You'll discover these failure cases -as you use the system more and more, so it's a good idea to keep a record of -them and add them to your test suite. +We got a 20% failure rate on the first run of this test, which was a good sign that the retrieval component could be improved. We only had 5 test cases, so +this was just a starting point. In reality, you'd want to keep adding more test cases to cover a wider range of scenarios. You'll discover these failure cases +as you use the system more and more, so it's a good idea to keep a record of them and add them to your test suite. -You'd also want to examine the logs to see exactly which query failed. In our -case, checking the logs in the ZenML dashboard, we find the following: +You'd also want to examine the logs to see exactly which query failed. In our case, checking the logs in the ZenML dashboard, we find the following: ``` Failed for question: How do I generate embeddings as part of a RAG @@ -114,16 +95,13 @@ zenml.io/user-guide/llmops-guide/rag-with-zenml', 'https://docs.zenml.io/v/docs/ user-guide/llmops-guide/rag-with-zenml'] ``` -We can maybe take a look at those documents to see why they were retrieved and -not the one we expected. This is a good way to iteratively improve the retrieval +We can maybe take a look at those documents to see why they were retrieved and not the one we expected. This is a good way to iteratively improve the retrieval component. ## Automated evaluation using synthetic generated queries -For a broader evaluation we can examine a larger number of queries to check the -retrieval component's performance. We do this by using an LLM to generate -synthetic data. In our case we take the text of each document chunk and pass it -to an LLM, telling it to generate a question. +For a broader evaluation we can examine a larger number of queries to check the retrieval component's performance. We do this by using an LLM to generate +synthetic data. In our case we take the text of each document chunk and pass it to an LLM, telling it to generate a question. ![](../../../.gitbook/assets/retrieval-eval-automated.png) @@ -158,10 +136,8 @@ orchestrator, container registry, and stack components, and how do I delete the resources when they are no longer needed? ``` -If we generate questions for all of our chunks, we can then use these -question-chunk pairs to evaluate the retrieval component. We pass the generated -query to the retrieval component and then we check if the URL for the original -document is in the top `n` results. +If we generate questions for all of our chunks, we can then use these question-chunk pairs to evaluate the retrieval component. We pass the generated +query to the retrieval component and then we check if the URL for the original document is in the top `n` results. To generate the synthetic queries we can use the following code: @@ -203,40 +179,30 @@ def generate_questions_from_chunks( return docs_with_embeddings ``` -As you can see, we're using [`litellm`](https://docs.litellm.ai/) again as the -wrapper for the API calls. This allows us to switch between using a cloud LLM +As you can see, we're using [`litellm`](https://docs.litellm.ai/) again as the wrapper for the API calls. This allows us to switch between using a cloud LLM API (like OpenAI's GPT3.5 or 4) and a local LLM (like a quantized version of -Mistral AI's Mixtral made available with [Ollama](https://ollama.com/). This has -a number of advantages: +Mistral AI's Mixtral made available with [Ollama](https://ollama.com/). This has a number of advantages: - you keep your costs down by using a local model - you can iterate faster by not having to wait for API calls - you can use the same code for both local and cloud models -For some tasks you'll want to use the best model your budget can afford, but for -this task of question generation we're fine using a local and slightly less -capable model. Even better is that it'll be much faster to generate the -questions, especially using the basic setup we have here. +For some tasks you'll want to use the best model your budget can afford, but for this task of question generation we're fine using a local and slightly less +capable model. Even better is that it'll be much faster to generate the questions, especially using the basic setup we have here. -To give you an indication of how long this process takes, generating 1800+ -questions from an equivalent number of documentation chunks took a little over +To give you an indication of how long this process takes, generating 1800+ questions from an equivalent number of documentation chunks took a little over 45 minutes using the local model on a GPU-enabled machine with Ollama. ![](../../../.gitbook/assets/hf-qa-embedding-questions.png) -You can [view the generated -dataset](https://huggingface.co/datasets/zenml/rag_qa_embedding_questions) on -the Hugging Face Hub -[here](https://huggingface.co/datasets/zenml/rag_qa_embedding_questions). This +You can [view the generated dataset](https://huggingface.co/datasets/zenml/rag_qa_embedding_questions) on +the Hugging Face Hub [here](https://huggingface.co/datasets/zenml/rag_qa_embedding_questions). This dataset contains the original document chunks, the generated questions, and the URL reference for the original document. -Once we have the generated questions, we can then pass them to the retrieval -component and check the results. For convenience we load the data from the -Hugging Face Hub and then pass it to the retrieval component for evaluation. We -shuffle the data and select a subset of it to speed up the evaluation process, -but for a more thorough evaluation you could use the entire dataset. (The best -practice of keeping a separate set of data for evaluation purposes is also +Once we have the generated questions, we can then pass them to the retrieval component and check the results. For convenience we load the data from the +Hugging Face Hub and then pass it to the retrieval component for evaluation. We shuffle the data and select a subset of it to speed up the evaluation process, +but for a more thorough evaluation you could use the entire dataset. (The best practice of keeping a separate set of data for evaluation purposes is also recommended here, though we're not doing that in this example.) ```python @@ -273,21 +239,17 @@ def retrieval_evaluation_full( return round(failure_rate, 2) ``` -When we run this as part of the evaluation pipeline, we get a 16% failure rate -which again tells us that we're doing pretty well but that there is room for -improvement. As a baseline, this is a good starting point. We can then iterate -on the retrieval component to improve its performance. +When we run this as part of the evaluation pipeline, we get a 16% failure rate which again tells us that we're doing pretty well but that there is room for +improvement. As a baseline, this is a good starting point. We can then iterate on the retrieval component to improve its performance. To take this further, there are a number of ways it might be improved: -- **More diverse question generation**: The current question generation approach - uses a single prompt to generate questions based on the document chunks. You +- **More diverse question generation**: The current question generation approach uses a single prompt to generate questions based on the document chunks. You could experiment with different prompts or techniques to generate a wider variety of questions that test the retrieval component more thoroughly. For example, you could prompt the LLM to generate questions of different types (factual, inferential, hypothetical, etc.) or difficulty levels. -- **Semantic similarity metrics**: In addition to checking if the expected URL - is retrieved, you could calculate semantic similarity scores between the query +- **Semantic similarity metrics**: In addition to checking if the expected URL is retrieved, you could calculate semantic similarity scores between the query and the retrieved documents using metrics like cosine similarity. This would give you a more nuanced view of retrieval performance beyond just binary success/failure. You could track average similarity scores and use them as a @@ -296,50 +258,35 @@ To take this further, there are a number of ways it might be improved: different embedding models, similarity search algorithms, etc.) and compare their performance on the same set of queries. This would help identify the strengths and weaknesses of each approach. -- **Error analysis**: Do a deeper dive into the failure cases to understand - patterns and potential areas for improvement. Are certain types of questions +- **Error analysis**: Do a deeper dive into the failure cases to understand patterns and potential areas for improvement. Are certain types of questions consistently failing? Are there common characteristics among the documents that aren't being retrieved properly? Insights from error analysis can guide targeted improvements to the retrieval component. -To wrap up, the retrieval evaluation process we've walked through - from manual -spot-checking with carefully crafted queries to automated testing with synthetic -question-document pairs - has provided a solid baseline understanding of our -retrieval component's performance. The failure rates of 20% on our handpicked -test cases and 16% on a larger sample of generated queries highlight clear room -for improvement, but also validate that our semantic search is generally +To wrap up, the retrieval evaluation process we've walked through - from manual spot-checking with carefully crafted queries to automated testing with synthetic +question-document pairs - has provided a solid baseline understanding of our retrieval component's performance. The failure rates of 20% on our handpicked +test cases and 16% on a larger sample of generated queries highlight clear room for improvement, but also validate that our semantic search is generally pointing in the right direction. -Going forward, we have a rich set of options to refine and upgrade our -evaluation approach. Generating a more diverse array of test questions, -leveraging semantic similarity metrics for a nuanced view beyond binary -success/failure, performing comparative evaluations of different retrieval -techniques, and conducting deep error analysis on failure cases - all of these -avenues promise to yield valuable insights. As our RAG pipeline grows to handle -more complex and wide-ranging queries, continued investment in comprehensive -retrieval evaluation will be essential to ensure we're always surfacing the most +Going forward, we have a rich set of options to refine and upgrade our evaluation approach. Generating a more diverse array of test questions, +leveraging semantic similarity metrics for a nuanced view beyond binary success/failure, performing comparative evaluations of different retrieval +techniques, and conducting deep error analysis on failure cases - all of these avenues promise to yield valuable insights. As our RAG pipeline grows to handle +more complex and wide-ranging queries, continued investment in comprehensive retrieval evaluation will be essential to ensure we're always surfacing the most relevant information. -Before we start working to improve or tweak our retrieval based on these -evaluation results, let's shift gears and look at how we can evaluate the -generation component of our RAG pipeline. Assessing the quality of the final -answers produced by the system is equally crucial to gauging the effectiveness +Before we start working to improve or tweak our retrieval based on these evaluation results, let's shift gears and look at how we can evaluate the +generation component of our RAG pipeline. Assessing the quality of the final answers produced by the system is equally crucial to gauging the effectiveness of our retrieval. -Retrieval is only half the story. The true test of our system is the quality -of the final answers it generates by combining retrieved content with LLM -intelligence. In the next section, we'll dive into a parallel evaluation process -for the generation component, exploring both automated metrics and human -assessment to get a well-rounded picture of our RAG pipeline's end-to-end -performance. By shining a light on both halves of the RAG architecture, we'll be -well-equipped to iterate and optimize our way to an ever more capable and -reliable question-answering system. +Retrieval is only half the story. The true test of our system is the quality of the final answers it generates by combining retrieved content with LLM +intelligence. In the next section, we'll dive into a parallel evaluation process for the generation component, exploring both automated metrics and human +assessment to get a well-rounded picture of our RAG pipeline's end-to-end performance. By shining a light on both halves of the RAG architecture, we'll be +well-equipped to iterate and optimize our way to an ever more capable and reliable question-answering system. ## Code Example To explore the full code, visit the [Complete Guide](https://github.com/zenml-io/zenml-projects/blob/main/llm-complete-guide/) -repository and for this section, particularly [the `eval_retrieval.py` -file](https://github.com/zenml-io/zenml-projects/blob/main/llm-complete-guide/steps/eval_retrieval.py). +repository and for this section, particularly [the `eval_retrieval.py` file](https://github.com/zenml-io/zenml-projects/blob/main/llm-complete-guide/steps/eval_retrieval.py).
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/finetuning-embeddings/synthetic-data-generation.md b/docs/book/user-guide/llmops-guide/finetuning-embeddings/synthetic-data-generation.md index 2627f1d13e0..923107230b7 100644 --- a/docs/book/user-guide/llmops-guide/finetuning-embeddings/synthetic-data-generation.md +++ b/docs/book/user-guide/llmops-guide/finetuning-embeddings/synthetic-data-generation.md @@ -2,19 +2,14 @@ description: Generate synthetic data with distilabel to finetune embeddings. --- -We already have [a dataset of technical documentation](https://huggingface.co/datasets/zenml/rag_qa_embedding_questions_0_60_0) that was generated -previously while we were working on the RAG pipeline. We'll use this dataset -to generate synthetic data with `distilabel`. You can inspect the data directly -[on the Hugging Face dataset page](https://huggingface.co/datasets/zenml/rag_qa_embedding_questions_0_60_0). +We already have [a dataset of technical documentation](https://huggingface.co/datasets/zenml/rag_qa_embedding_questions_0_60_0) that was generated previously while we were working on the RAG pipeline. We'll use this dataset +to generate synthetic data with `distilabel`. You can inspect the data directly [on the Hugging Face dataset page](https://huggingface.co/datasets/zenml/rag_qa_embedding_questions_0_60_0). ![](../../../.gitbook/assets/rag-dataset-hf.png) -As you can see, it is made up of some `page_content` (our chunks) as well as the -source URL from where the chunk was taken from. With embeddings, what we're -going to want to do is pair the `page_content` with a question that we want to -answer. In a pre-LLM world we might have actually created a new column and -worked to manually craft questions for each chunk. However, with LLMs, we can -use the `page_content` to generate questions. +As you can see, it is made up of some `page_content` (our chunks) as well as the source URL from where the chunk was taken from. With embeddings, what we're +going to want to do is pair the `page_content` with a question that we want to answer. In a pre-LLM world we might have actually created a new column and +worked to manually craft questions for each chunk. However, with LLMs, we can use the `page_content` to generate questions. ## Pipeline overview @@ -22,23 +17,17 @@ Our pipeline to generate synthetic data will look like this: ![](../../../.gitbook/assets/rag-synthetic-data-pipeline.png) -We'll load the Hugging Face dataset, then we'll use `distilabel` to generate the -synthetic data. To finish off, we'll push the newly-generated data to a new -Hugging Face dataset and also push the same data to our Argilla instance for -annotation and inspection. +We'll load the Hugging Face dataset, then we'll use `distilabel` to generate the synthetic data. To finish off, we'll push the newly-generated data to a new +Hugging Face dataset and also push the same data to our Argilla instance for annotation and inspection. ## Synthetic data generation -[`distilabel`](https://github.com/argilla-io/distilabel) provides a scalable and -reliable approach to distilling knowledge from LLMs by generating synthetic data -or providing AI feedback with LLMs as judges. We'll be using it a relatively -simple use case to generate some queries appropriate to our documentation +[`distilabel`](https://github.com/argilla-io/distilabel) provides a scalable and reliable approach to distilling knowledge from LLMs by generating synthetic data +or providing AI feedback with LLMs as judges. We'll be using it a relatively simple use case to generate some queries appropriate to our documentation chunks, but it can be used for a variety of other tasks. -We can set up a `distilabel` pipeline easily in our ZenML step to handle the -dataset creation. We'll be using `gpt-4o` as the LLM to generate the synthetic -data so you can follow along, but `distilabel` supports a variety of other LLM -providers (including Ollama) so you can use whatever you have available. +We can set up a `distilabel` pipeline easily in our ZenML step to handle the dataset creation. We'll be using `gpt-4o` as the LLM to generate the synthetic +data so you can follow along, but `distilabel` supports a variety of other LLM providers (including Ollama) so you can use whatever you have available. ```python import os @@ -123,49 +112,33 @@ def generate_synthetic_queries( return train_dataset, test_dataset ``` -As you can see, we set up the LLM, create a `distilabel` pipeline, load the -dataset, mapping the `page_content` column so that it becomes `anchor`. (This -column renaming will make things easier a bit later when we come to finetuning -the embeddings.) Then we generate the synthetic data by using the `GenerateSentencePair` -step. This will create queries for each of the chunks in the dataset, so if the -chunk was about registering a ZenML stack, the query might be "How do I register -a ZenML stack?". It will also create negative queries, which are queries that -would be inappropriate for the chunk. We do this so that the embeddings model +As you can see, we set up the LLM, create a `distilabel` pipeline, load the dataset, mapping the `page_content` column so that it becomes `anchor`. (This +column renaming will make things easier a bit later when we come to finetuning the embeddings.) Then we generate the synthetic data by using the `GenerateSentencePair` +step. This will create queries for each of the chunks in the dataset, so if the chunk was about registering a ZenML stack, the query might be "How do I register +a ZenML stack?". It will also create negative queries, which are queries that would be inappropriate for the chunk. We do this so that the embeddings model can learn to distinguish between appropriate and inappropriate queries. -We add some context to the generation process to help the LLM -understand the task and the data we're working with. In particular, we explain -that some parts of the text are code snippets and logs. We found performance to -be better when we added this context. +We add some context to the generation process to help the LLM understand the task and the data we're working with. In particular, we explain +that some parts of the text are code snippets and logs. We found performance to be better when we added this context. -When this step runs within ZenML it will handle spinning up the necessary -processes to make batched LLM calls to the OpenAI API. This is really useful -when working with large datasets. `distilabel` has also implemented a caching -mechanism to avoid recomputing results for the same inputs. So in this case you +When this step runs within ZenML it will handle spinning up the necessary processes to make batched LLM calls to the OpenAI API. This is really useful +when working with large datasets. `distilabel` has also implemented a caching mechanism to avoid recomputing results for the same inputs. So in this case you have two layers of caching: one in the `distilabel` pipeline and one in the ZenML orchestrator. This helps [speed up the pace of iteration](https://www.zenml.io/blog/iterate-fast) and saves you money. ## Data annotation with Argilla -Once we've let the LLM generate the synthetic data, we'll want to inspect it -and make sure it looks good. We'll do this by pushing the data to an Argilla -instance. We add a few extra pieces of metadata to the data to make it easier to -navigate and inspect within our data annotation tool. These include: +Once we've let the LLM generate the synthetic data, we'll want to inspect it and make sure it looks good. We'll do this by pushing the data to an Argilla +instance. We add a few extra pieces of metadata to the data to make it easier to navigate and inspect within our data annotation tool. These include: -- `parent_section`: This will be the section of the documentation that the chunk - is from. +- `parent_section`: This will be the section of the documentation that the chunk is from. - `token_count`: This will be the number of tokens in the chunk. -- `similarity-positive-negative`: This will be the cosine similarity between the - positive and negative queries. -- `similarity-anchor-positive`: This will be the cosine similarity between the - anchor and positive queries. -- `similarity-anchor-negative`: This will be the cosine similarity between the - anchor and negative queries. - -We'll also add the embeddings for the anchor column so that we can use these -for retrieval. We'll use the base model (in our case, -`Snowflake/snowflake-arctic-embed-large`) to generate the embeddings. We use -this function to map the dataset and process all the metadata: +- `similarity-positive-negative`: This will be the cosine similarity between the positive and negative queries. +- `similarity-anchor-positive`: This will be the cosine similarity between the anchor and positive queries. +- `similarity-anchor-negative`: This will be the cosine similarity between the anchor and negative queries. + +We'll also add the embeddings for the anchor column so that we can use these for retrieval. We'll use the base model (in our case, +`Snowflake/snowflake-arctic-embed-large`) to generate the embeddings. We use this function to map the dataset and process all the metadata: ```python def format_data(batch): @@ -203,21 +176,16 @@ def format_data(batch): return batch ``` -The [rest of the `push_to_argilla` step](https://github.com/zenml-io/zenml-projects/blob/main/llm-complete-guide/steps/push_to_argilla.py) is just setting up the Argilla -dataset and pushing the data to it. +The [rest of the `push_to_argilla` step](https://github.com/zenml-io/zenml-projects/blob/main/llm-complete-guide/steps/push_to_argilla.py) is just setting up the Argilla dataset and pushing the data to it. -At this point you'd move to Argilla to view the data, see which examples seem to -make sense and which don't. You can update the questions (positive and negative) -which were generated by the LLM. If you want, you can do some data cleaning and -exploration to improve the data quality, perhaps using the similarity metrics +At this point you'd move to Argilla to view the data, see which examples seem to make sense and which don't. You can update the questions (positive and negative) +which were generated by the LLM. If you want, you can do some data cleaning and exploration to improve the data quality, perhaps using the similarity metrics that we calculated earlier. ![Argilla interface for data annotation](../../../.gitbook/assets/argilla-interface-embeddings-finetuning.png) -We'll next move to actually finetuning the embeddings, assuming you've done some -data exploration and annotation. The code will work even without the annotation, -however, since we'll just use the full generated dataset and assume that the -quality is good enough. +We'll next move to actually finetuning the embeddings, assuming you've done some data exploration and annotation. The code will work even without the annotation, +however, since we'll just use the full generated dataset and assume that the quality is good enough.
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/finetuning-llms/deploying-finetuned-models.md b/docs/book/user-guide/llmops-guide/finetuning-llms/deploying-finetuned-models.md index eb3f2171701..4f91e4df30a 100644 --- a/docs/book/user-guide/llmops-guide/finetuning-llms/deploying-finetuned-models.md +++ b/docs/book/user-guide/llmops-guide/finetuning-llms/deploying-finetuned-models.md @@ -4,37 +4,30 @@ Deploying your finetuned LLM is a critical step in bringing your custom finetune ## Deployment Considerations -Before diving into specific deployment options, you should understand the various factors that influence the deployment process. One of the primary considerations is the memory and machine requirements for your finetuned model.LLMs -are typically resource-intensive, requiring substantial RAM, processing power and specialized hardware. This choice of hardware can significantly impact both +Before diving into specific deployment options, you should understand the various factors that influence the deployment process. One of the primary considerations is the memory and machine requirements for your finetuned model.LLMs are typically resource-intensive, requiring substantial RAM, processing power and specialized hardware. This choice of hardware can significantly impact both performance and cost, so it's crucial to strike the right balance based on your specific use case. -Real-time considerations play a vital role in deployment planning, especially for applications that require immediate responses. This includes preparing for potential failover scenarios if your finetuned model encounters issues, -conducting thorough benchmarks and load testing, and modeling expected user load and usage patterns. Additionally, you'll need to decide between streaming +Real-time considerations play a vital role in deployment planning, especially for applications that require immediate responses. This includes preparing for potential failover scenarios if your finetuned model encounters issues, conducting thorough benchmarks and load testing, and modeling expected user load and usage patterns. Additionally, you'll need to decide between streaming and non-streaming approaches, each with its own set of trade-offs in terms of latency and resource utilization. -Optimization techniques, such as quantization, can help reduce the resource footprint of your model. However, these Optimizations often come with additional -steps in your workflow and require careful evaluation to ensure they don't negatively impact model performance. [Rigorous evaluation](./evaluation-for-finetuning.md) +Optimization techniques, such as quantization, can help reduce the resource footprint of your model. However, these Optimizations often come with additional steps in your workflow and require careful evaluation to ensure they don't negatively impact model performance. [Rigorous evaluation](./evaluation-for-finetuning.md) becomes crucial in quantifying the extent to which you can optimize without compromising accuracy or functionality. ## Deployment Options and Trade-offs When it comes to deploying your finetuned LLM, several options are available, each with its own set of advantages and challenges: -1. **Roll Your Own**: This approach involves setting up and managing your own - infrastructure. While it offers the most control and customization, it also +1. **Roll Your Own**: This approach involves setting up and managing your own infrastructure. While it offers the most control and customization, it also requires expertise and resources to maintain. For this, you'd usually create some kind of Docker-based service (a FastAPI endpoint, for example) and deploy this on your infrastructure, with you taking care of all of the steps along the way. -2. **Serverless Options**: Serverless deployments can provide scalability and - cost-efficiency, as you only pay for the compute resources you use. However, +2. **Serverless Options**: Serverless deployments can provide scalability and cost-efficiency, as you only pay for the compute resources you use. However, be aware of the "cold start" phenomenon, which can introduce latency for infrequently accessed models. -3. **Always-On Options**: These deployments keep your model constantly running - and ready to serve requests. While this approach minimizes latency, it can be +3. **Always-On Options**: These deployments keep your model constantly running and ready to serve requests. While this approach minimizes latency, it can be more costly as you're paying for resources even during idle periods. -4. **Fully Managed Solutions**: Many cloud providers and AI platforms offer - managed services for deploying LLMs. These solutions can simplify the deployment process but may come with less flexibility and potentially higher costs. +4. **Fully Managed Solutions**: Many cloud providers and AI platforms offer managed services for deploying LLMs. These solutions can simplify the deployment process but may come with less flexibility and potentially higher costs. When choosing a deployment option, consider factors such as your team's expertise, budget constraints, expected load patterns, and specific use case requirements like speed, throughput, and accuracy needs. @@ -44,11 +37,9 @@ When choosing a deployment option, consider factors such as your team's expertis **Note**: The example below uses the Model Deployer component, which is maintained for backward compatibility. For new projects, consider using [Pipeline Deployments](https://docs.zenml.io/concepts/deployment) which offer greater flexibility for deploying LLM inference workflows with custom preprocessing and business logic. {% endhint %} -[vLLM](https://github.com/vllm-project/vllm) is a fast and easy-to-use library -for running large language models (LLMs) at high throughputs and low latency. +[vLLM](https://github.com/vllm-project/vllm) is a fast and easy-to-use library for running large language models (LLMs) at high throughputs and low latency. ZenML comes with a [vLLM integration](https://docs.zenml.io/stacks/model-deployers/vllm) -that makes it easy to deploy your finetuned model using vLLM. You can use a pre-built step that exposes a `VLLMDeploymentService` that can be used as part of -your deployment pipeline. +that makes it easy to deploy your finetuned model using vLLM. You can use a pre-built step that exposes a `VLLMDeploymentService` that can be used as part of your deployment pipeline. ```python @@ -72,102 +63,70 @@ def deploy_vllm_pipeline( return service ``` -In this code snippet, the `model` argument can be a path to a local model or it -can be a model ID on the Hugging Face Hub. This will then deploy the model -locally using vLLM and you can then use the `VLLMDeploymentService` for batch -inference requests using the OpenAI-compatible API. +In this code snippet, the `model` argument can be a path to a local model or it can be a model ID on the Hugging Face Hub. This will then deploy the model +locally using vLLM and you can then use the `VLLMDeploymentService` for batch inference requests using the OpenAI-compatible API. For more details on how to use this deployer, see the [vLLM integration documentation](https://docs.zenml.io/stacks/model-deployers/vllm). ## Cloud-Specific Deployment Options -For AWS deployments, Amazon SageMaker stands out as a fully managed machine -learning platform that offers deployment of LLMs with options for -real-time inference endpoints and automatic scaling. If you prefer a serverless -approach, combining AWS Lambda with API Gateway can host your model and trigger +For AWS deployments, Amazon SageMaker stands out as a fully managed machine learning platform that offers deployment of LLMs with options for +real-time inference endpoints and automatic scaling. If you prefer a serverless approach, combining AWS Lambda with API Gateway can host your model and trigger it for real-time responses, though be mindful of potential cold start issues. -For teams seeking more control over the runtime environment while still -leveraging AWS's managed infrastructure, Amazon ECS or EKS with Fargate provides -an excellent container orchestration solution, though do note that with all of -these options you're taking on a level of complexity that might become costly to +For teams seeking more control over the runtime environment while still leveraging AWS's managed infrastructure, Amazon ECS or EKS with Fargate provides +an excellent container orchestration solution, though do note that with all of these options you're taking on a level of complexity that might become costly to manage in-house. On the GCP side, Google Cloud AI Platform offers similar capabilities to -SageMaker, providing managed ML services including model deployment and -prediction. For a serverless option, Cloud Run can host your containerized LLM -and automatically scale based on incoming requests. Teams requiring more -fine-grained control over compute resources might prefer Google Kubernetes +SageMaker, providing managed ML services including model deployment and prediction. For a serverless option, Cloud Run can host your containerized LLM +and automatically scale based on incoming requests. Teams requiring more fine-grained control over compute resources might prefer Google Kubernetes Engine (GKE) for deploying containerized models. ## Architectures for Real-Time Customer Engagement -Ensuring your system can engage with customers in real-time, for example, requires careful -architectural consideration. One effective approach is to deploy your model -across multiple instances behind a load balancer, using auto-scaling to -dynamically adjust the number of instances based on incoming traffic. This setup +Ensuring your system can engage with customers in real-time, for example, requires careful architectural consideration. One effective approach is to deploy your model +across multiple instances behind a load balancer, using auto-scaling to dynamically adjust the number of instances based on incoming traffic. This setup provides both responsiveness and scalability. -To further enhance performance, consider implementing a caching layer using -solutions like Redis. This can store frequent responses, reducing the load on -your model and improving response times for common queries. For complex queries -that may take longer to process, an asynchronous architecture using message -queues (such as Amazon SQS or Google Cloud Pub/Sub) can manage request backlogs -and prevent timeouts, ensuring a smooth user experience even under heavy load. +To further enhance performance, consider implementing a caching layer using solutions like Redis. This can store frequent responses, reducing the load on +your model and improving response times for common queries. For complex queries that may take longer to process, an asynchronous architecture using message +queues (such as Amazon SQS or Google Cloud Pub/Sub) can manage request backlogs and prevent timeouts, ensuring a smooth user experience even under heavy load. -For global deployments, edge computing services like [AWS Lambda@Edge](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html?tag=soumet-20) or -[CloudFront Functions](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-functions.html?tag=soumet-20) can be invaluable. These allow you to deploy lighter -versions of your model closer to end-users, significantly reducing latency for -initial responses and improving the overall user experience. +For global deployments, edge computing services like [AWS Lambda@Edge](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html?tag=soumet-20) or [CloudFront Functions](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-functions.html?tag=soumet-20) can be invaluable. These allow you to deploy lighter +versions of your model closer to end-users, significantly reducing latency for initial responses and improving the overall user experience. ## Reducing Latency and Increasing Throughput -Optimizing your deployment for low latency and high throughput is crucial for -real-time engagement. Start by focusing on model optimization techniques such as -quantization to reduce model size and inference time. You might also explore -distillation techniques to create smaller, faster models that approximate the +Optimizing your deployment for low latency and high throughput is crucial for real-time engagement. Start by focusing on model optimization techniques such as +quantization to reduce model size and inference time. You might also explore distillation techniques to create smaller, faster models that approximate the performance of larger ones without sacrificing too much accuracy. Hardware acceleration can provide a significant performance boost. Leveraging -GPU instances for inference, particularly for larger models, can dramatically -reduce processing time. - -Implementing request batching allows you to process multiple inputs in a single -forward pass, increasing overall throughput. This can be particularly effective -when combined with parallel processing techniques, utilizing multi-threading or -multi-processing to handle multiple requests concurrently. This would make sense -if you were operating at serious scale, but this is probably unlikely in the -short-term when you are just getting started. - -Finally, implement detailed monitoring and use profiling tools to identify -bottlenecks in your inference pipeline. This ongoing process of measurement and -optimization will help you continually refine your deployment, ensuring it meets -the evolving demands of your users. - -By thoughtfully implementing these strategies and maintaining a focus on -continuous improvement, you can create a robust, scalable system that provides -real-time engagement with low latency and high throughput, regardless of whether -you're deploying on AWS, GCP, or a multi-cloud environment. +GPU instances for inference, particularly for larger models, can dramatically reduce processing time. + +Implementing request batching allows you to process multiple inputs in a single forward pass, increasing overall throughput. This can be particularly effective +when combined with parallel processing techniques, utilizing multi-threading or multi-processing to handle multiple requests concurrently. This would make sense +if you were operating at serious scale, but this is probably unlikely in the short-term when you are just getting started. + +Finally, implement detailed monitoring and use profiling tools to identify bottlenecks in your inference pipeline. This ongoing process of measurement and +optimization will help you continually refine your deployment, ensuring it meets the evolving demands of your users. + +By thoughtfully implementing these strategies and maintaining a focus on continuous improvement, you can create a robust, scalable system that provides +real-time engagement with low latency and high throughput, regardless of whether you're deploying on AWS, GCP, or a multi-cloud environment. ## Monitoring and Maintenance -Once your finetuned LLM is deployed, ongoing monitoring and maintenance become -crucial. Key areas to watch include: +Once your finetuned LLM is deployed, ongoing monitoring and maintenance become crucial. Key areas to watch include: -1. **Evaluation Failures**: Regularly run your model through evaluation sets to - catch any degradation in performance. -2. **Latency Metrics**: Monitor response times to ensure they meet your - application's requirements. -3. **Load and Usage Patterns**: Keep an eye on how users interact with your model - to inform scaling decisions and potential Optimizations. -4. **Data Analysis**: Regularly analyze the inputs and outputs of your model to - identify trends, potential biases, or areas for improvement. +1. **Evaluation Failures**: Regularly run your model through evaluation sets to catch any degradation in performance. +2. **Latency Metrics**: Monitor response times to ensure they meet your application's requirements. +3. **Load and Usage Patterns**: Keep an eye on how users interact with your model to inform scaling decisions and potential Optimizations. +4. **Data Analysis**: Regularly analyze the inputs and outputs of your model to identify trends, potential biases, or areas for improvement. -It's also important to consider privacy and security when capturing and logging -responses. Ensure that your logging practices comply with relevant data +It's also important to consider privacy and security when capturing and logging responses. Ensure that your logging practices comply with relevant data protection regulations and your organization's privacy policies. -By carefully considering these deployment options and maintaining vigilant -monitoring practices, you can ensure that your finetuned LLM performs optimally +By carefully considering these deployment options and maintaining vigilant monitoring practices, you can ensure that your finetuned LLM performs optimally and continues to meet the needs of your users and organization.
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/finetuning-llms/evaluation-for-finetuning.md b/docs/book/user-guide/llmops-guide/finetuning-llms/evaluation-for-finetuning.md index 69095f92fca..374324c5d5d 100644 --- a/docs/book/user-guide/llmops-guide/finetuning-llms/evaluation-for-finetuning.md +++ b/docs/book/user-guide/llmops-guide/finetuning-llms/evaluation-for-finetuning.md @@ -20,22 +20,16 @@ By implementing a robust evaluation strategy, you can develop more reliable, per ## Types of Evaluations -It's common for finetuning projects to use generic out-of-the-box evaluation -frameworks, but it's also useful to understand how to implement custom evals -for your specific use case. In the end, building out a robust set of evaluations -is a crucial part of knowing whether what you finetune is actually working. It -also will allow you to benchmark your progress over time as well as check -- -when a new model gets released -- whether it even makes sense to continue with -the finetuning work you've done. New open-source and open-weights models are -released all the time, and you might find that your use case is better solved by +It's common for finetuning projects to use generic out-of-the-box evaluation frameworks, but it's also useful to understand how to implement custom evals +for your specific use case. In the end, building out a robust set of evaluations is a crucial part of knowing whether what you finetune is actually working. It +also will allow you to benchmark your progress over time as well as check -- when a new model gets released -- whether it even makes sense to continue with +the finetuning work you've done. New open-source and open-weights models are released all the time, and you might find that your use case is better solved by a new model. Evaluations will allow you to make this decision. ### Custom Evals -The approach taken for custom evaluations is similar to that used and [showcased -in the RAG guide](../evaluation/README.md), but it is adapted here for the -finetuning use case. The main distinction here is that we are not looking to -evaluate retrieval, but rather the performance of the finetuned model (i.e. +The approach taken for custom evaluations is similar to that used and [showcased in the RAG guide](../evaluation/README.md), but it is adapted here for the +finetuning use case. The main distinction here is that we are not looking to evaluate retrieval, but rather the performance of the finetuned model (i.e. [the generation part](../evaluation/generation.md)). Custom evals are tailored to your specific use case and can be categorized into two main types: @@ -93,10 +87,8 @@ When using Generalized evals, it's important to consider their limitations and c - [langcheck](https://github.com/citadel-ai/langcheck) - [nervaluate](https://github.com/MantisAI/nervaluate) (for NER) -It's easy to build in one of these frameworks into your ZenML pipeline. The -implementation of evaluation in [the `llm-lora-finetuning` project](https://github.com/zenml-io/zenml-projects/tree/main/gamesense) is a good -example of how to do this. We used the `evaluate` library for ROUGE evaluation, -but you could easily swap this out for another framework if you prefer. See [the previous section](finetuning-with-accelerate.md#implementation-details) for more details. +It's easy to build in one of these frameworks into your ZenML pipeline. The implementation of evaluation in [the `llm-lora-finetuning` project](https://github.com/zenml-io/zenml-projects/tree/main/gamesense) is a good +example of how to do this. We used the `evaluate` library for ROUGE evaluation, but you could easily swap this out for another framework if you prefer. See [the previous section](finetuning-with-accelerate.md#implementation-details) for more details. ## Data and Tracking @@ -110,13 +102,9 @@ As part of this, implementing comprehensive logging from the early stages of dev - [langfuse](https://langfuse.com/) - [braintrust](https://www.braintrust.dev/) -Alongside collecting the raw data and viewing it periodically, creating simple -dashboards that display core metrics reflecting your model's performance is an -effective way to visualize and monitor progress. These metrics should align with -your iteration goals and capture improvements over time, allowing you to quickly -assess the impact of changes and identify areas that require attention. Again, -as with everything else, don't let perfect be the enemy of the good; a simple -dashboard using simple technology with a few key metrics is better than no -dashboard at all. +Alongside collecting the raw data and viewing it periodically, creating simple dashboards that display core metrics reflecting your model's performance is an +effective way to visualize and monitor progress. These metrics should align with your iteration goals and capture improvements over time, allowing you to quickly +assess the impact of changes and identify areas that require attention. Again, as with everything else, don't let perfect be the enemy of the good; a simple +dashboard using simple technology with a few key metrics is better than no dashboard at all.
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/finetuning-llms/finetuning-100-loc.md b/docs/book/user-guide/llmops-guide/finetuning-llms/finetuning-100-loc.md index 1e1b9dfe083..4122f6fcbc7 100644 --- a/docs/book/user-guide/llmops-guide/finetuning-llms/finetuning-100-loc.md +++ b/docs/book/user-guide/llmops-guide/finetuning-llms/finetuning-100-loc.md @@ -11,8 +11,7 @@ There's a lot to understand about LLM fine-tuning - from choosing the right base - Fine-tuning the model on custom data - Using the fine-tuned model to generate responses -This example uses the same [fictional "ZenML World" setting as our RAG -example](../rag-with-zenml/rag-85-loc.md), but now we're teaching the model to +This example uses the same [fictional "ZenML World" setting as our RAG example](../rag-with-zenml/rag-85-loc.md), but now we're teaching the model to generate content about this world rather than just retrieving information. You'll need to `pip install` the following packages: @@ -196,8 +195,7 @@ This implementation is intentionally simplified and has several limitations: 3. **Training Time**: We use minimal epochs and a simple learning rate to keep the example runnable. 4. **Evaluation**: A production system would need proper evaluation metrics and validation data. -If you take a closer look at the inference output, you'll see that the quality -of the responses is pretty poor, but we only used 3 examples for training! +If you take a closer look at the inference output, you'll see that the quality of the responses is pretty poor, but we only used 3 examples for training! ## Next Steps diff --git a/docs/book/user-guide/llmops-guide/finetuning-llms/finetuning-with-accelerate.md b/docs/book/user-guide/llmops-guide/finetuning-llms/finetuning-with-accelerate.md index ddf4200c9b2..f4e181b2472 100644 --- a/docs/book/user-guide/llmops-guide/finetuning-llms/finetuning-with-accelerate.md +++ b/docs/book/user-guide/llmops-guide/finetuning-llms/finetuning-with-accelerate.md @@ -4,34 +4,25 @@ description: "Finetuning an LLM with Accelerate and PEFT" # Finetuning an LLM with Accelerate and PEFT -We're finally ready to get our hands on the code and see how it works. In this -example we'll be finetuning models on [the Viggo -dataset](https://huggingface.co/datasets/GEM/viggo). This is a dataset that -contains pairs of meaning representations and their corresponding natural language -descriptions for video game dialogues. The dataset was created to help train -models that can generate natural language responses from structured meaning -representations in the video game domain. It contains over 5,000 examples with -both the structured input and the target natural language output. We'll be -finetuning a model to learn this mapping and generate fluent responses from the -structured meaning representations. +We're finally ready to get our hands on the code and see how it works. In this example we'll be finetuning models on [the Viggo +dataset](https://huggingface.co/datasets/GEM/viggo). This is a dataset that contains pairs of meaning representations and their corresponding natural language +descriptions for video game dialogues. The dataset was created to help train models that can generate natural language responses from structured meaning +representations in the video game domain. It contains over 5,000 examples with both the structured input and the target natural language output. We'll be +finetuning a model to learn this mapping and generate fluent responses from the structured meaning representations. {% hint style="info" %} -For a -full walkthrough of how to run the LLM finetuning yourself, visit [the LLM Lora -Finetuning -project](https://github.com/zenml-io/zenml-projects/tree/main/gamesense) +For a full walkthrough of how to run the LLM finetuning yourself, visit [the LLM Lora +Finetuning project](https://github.com/zenml-io/zenml-projects/tree/main/gamesense) where you'll find instructions and the code. {% endhint %} ## The Finetuning Pipeline -Our finetuning pipeline combines the actual model finetuning with some -evaluation steps to check the performance of the finetuned model. +Our finetuning pipeline combines the actual model finetuning with some evaluation steps to check the performance of the finetuned model. ![](../../../.gitbook/assets/finetuning-pipeline.png) -As you can see in the DAG visualization, the pipeline consists of the following -steps: +As you can see in the DAG visualization, the pipeline consists of the following steps: - **prepare_data**: We load and preprocess the Viggo dataset. - **finetune**: We finetune the model on the Viggo dataset. @@ -39,28 +30,21 @@ steps: - **evaluate_finetuned**: We evaluate the finetuned model on the Viggo dataset. - **promote**: We promote the best performing model to "staging" in the [Model Control Plane](https://docs.zenml.io/how-to/model-management-metrics/model-control-plane). -If you adapt the code to your own use case, the specific logic in each step -might differ but the overall structure should remain the same. When you're -starting out with this pipeline, you'll probably want to start with model with -smaller size (e.g. one of the Llama 3.1 family at the ~8B parameter mark) and -then iterate on that. This will allow you to quickly run through a number of -experiments and see how the model performs on your use case. +If you adapt the code to your own use case, the specific logic in each step might differ but the overall structure should remain the same. When you're +starting out with this pipeline, you'll probably want to start with model with smaller size (e.g. one of the Llama 3.1 family at the ~8B parameter mark) and +then iterate on that. This will allow you to quickly run through a number of experiments and see how the model performs on your use case. In this early stage, experimentation is important. Accordingly, any way you can maximize the number of experiments you can run will help increase the amount you can learn. So we want to minimize the amount of time it takes to iterate to a new experiment. Depending on the precise details of what you do, you might iterate on your data, on some hyperparameters of the finetuning process, or you might even try out different use case options. ## Implementation details Our `prepare_data` step is very minimalistic. It loads the data from the Hugging -Face hub and tokenizes it with the model tokenizer. Potentially for your use -case you might want to do some more sophisticated filtering or formatting of the -data. Make sure to be especially careful about the format of your input data, -particularly when using instruction tuned models, since a mismatch here can -easily lead to unexpected results. It's a good rule of thumb to log inputs and -outputs for the finetuning step and to inspect these to make sure they look +Face hub and tokenizes it with the model tokenizer. Potentially for your use case you might want to do some more sophisticated filtering or formatting of the +data. Make sure to be especially careful about the format of your input data, particularly when using instruction tuned models, since a mismatch here can +easily lead to unexpected results. It's a good rule of thumb to log inputs and outputs for the finetuning step and to inspect these to make sure they look correct. -For finetuning we use the `accelerate` library. This allows us to easily run the -finetuning on multiple GPUs should you choose to do so. After setting up the +For finetuning we use the `accelerate` library. This allows us to easily run the finetuning on multiple GPUs should you choose to do so. After setting up the parameters, the actual finetuning step is set up quite concisely: ```python @@ -108,28 +92,22 @@ model = load_base_model( Here are some things to note: -- The `ZenMLCallback` is used to log the training and evaluation metrics to - ZenML. -- The `gradient_checkpointing_kwargs` are used to enable gradient checkpointing - when using Accelerate. -- All the other significant parameters are parameterised in the configuration file that is - used to run the pipeline. This means that you can easily swap out different +- The `ZenMLCallback` is used to log the training and evaluation metrics to ZenML. +- The `gradient_checkpointing_kwargs` are used to enable gradient checkpointing when using Accelerate. +- All the other significant parameters are parameterised in the configuration file that is used to run the pipeline. This means that you can easily swap out different values to try out different configurations without having to edit the code. -For the evaluation steps, we use [the `evaluate` library](https://github.com/huggingface/evaluate) to compute the ROUGE -scores. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics for evaluating automatic summarization and machine translation. It works by comparing generated text against reference texts by measuring: +For the evaluation steps, we use [the `evaluate` library](https://github.com/huggingface/evaluate) to compute the ROUGE scores. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics for evaluating automatic summarization and machine translation. It works by comparing generated text against reference texts by measuring: - **ROUGE-N**: Overlap of n-grams (sequences of n consecutive words) between generated and reference texts - **ROUGE-L**: Longest Common Subsequence between generated and reference texts - **ROUGE-W**: Weighted Longest Common Subsequence that favors consecutive matches - **ROUGE-S**: Skip-bigram co-occurrence statistics between generated and reference texts -These metrics help quantify how well the generated text captures the key -information and phrasing from the reference text, making them useful for +These metrics help quantify how well the generated text captures the key information and phrasing from the reference text, making them useful for evaluating model outputs. -It is a generic evaluation that can be used for a wide range of tasks beyond -just finetuning LLMs. We use it here as a placeholder for a more sophisticated +It is a generic evaluation that can be used for a wide range of tasks beyond just finetuning LLMs. We use it here as a placeholder for a more sophisticated evaluation step. See the next [evaluation section](./evaluation-for-finetuning.md) for more. ### Using the ZenML Accelerate Decorator @@ -185,23 +163,17 @@ def finetuning_pipeline(...): # Your pipeline steps here ``` -This configuration ensures that your training environment has all the necessary -components for distributed training. For more details, see the [Accelerate documentation](https://docs.zenml.io/user-guides/tutorial/distributed-training). +This configuration ensures that your training environment has all the necessary components for distributed training. For more details, see the [Accelerate documentation](https://docs.zenml.io/user-guides/tutorial/distributed-training). ## Dataset iteration While these stages offer lots of surface area for intervention and customization, the most significant thing to be careful with is the data that you input into the model. If you find that your finetuned model offers worse performance than the base, or if you get garbled output post-fine tuning, this would be a strong indicator that you have not correctly formatted your input data, or something is mismatched with the tokeniser and so on. To combat this, be sure to inspect your data at all stages of the process! -The main behavior and activity while using this notebook should be around being -more serious about your data. If you are finding that you're on the low end of -the spectrum, consider ways to either supplement that data or to synthetically -generate data that could be substituted in. You should also start to think about -evaluations at this stage (see [the next guide](./evaluation-for-finetuning.md) for more) since -the changes you will likely want to measure how well your model is doing, -especially when you make changes and customizations. Once you have some basic -evaluations up and running, you can then start thinking through all the optimal -parameters and measuring whether these updates are actually doing what you think -they will. +The main behavior and activity while using this notebook should be around being more serious about your data. If you are finding that you're on the low end of +the spectrum, consider ways to either supplement that data or to synthetically generate data that could be substituted in. You should also start to think about +evaluations at this stage (see [the next guide](./evaluation-for-finetuning.md) for more) since the changes you will likely want to measure how well your model is doing, +especially when you make changes and customizations. Once you have some basic evaluations up and running, you can then start thinking through all the optimal +parameters and measuring whether these updates are actually doing what you think they will. At a certain point, your mind will start to think beyond the details of what data you use as inputs and what hyperparameters or base models to experiment with. At that point you'll start to turn to the following: @@ -211,7 +183,6 @@ At a certain point, your mind will start to think beyond the details of what dat A goal that might be also worth considering: 'how small can we get our model that would be acceptable for our needs and use case?' This is where evaluations become important. In general, smaller models mean less complexity and better outcomes, especially if you can solve a specific scoped-down use case. -Check out the sections that follow as suggestions for ways to think about these -larger questions. +Check out the sections that follow as suggestions for ways to think about these larger questions.
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/finetuning-llms/next-steps.md b/docs/book/user-guide/llmops-guide/finetuning-llms/next-steps.md index dd38b5b9b0f..d57269da8b2 100644 --- a/docs/book/user-guide/llmops-guide/finetuning-llms/next-steps.md +++ b/docs/book/user-guide/llmops-guide/finetuning-llms/next-steps.md @@ -20,13 +20,10 @@ You might be tempted to just continue escalating the ladder of larger and larger Some other resources for reading or learning about LLM finetuning that we'd recommend are: -- [Mastering LLMs Course](https://parlance-labs.com/education/) - videos from - the LLM finetuning course run by Hamel Husain and Dan Becker. A great place to +- [Mastering LLMs Course](https://parlance-labs.com/education/) - videos from the LLM finetuning course run by Hamel Husain and Dan Becker. A great place to start if you enjoy watching videos -- [Phil Schmid's blog](https://www.philschmid.de/) - contains many worked - examples of LLM finetuning using the latest models and techniques -- [Sam Witteveen's YouTube channel](https://www.youtube.com/@samwitteveenai) - - videos on a wide range of topics from finetuning to prompt engineering, +- [Phil Schmid's blog](https://www.philschmid.de/) - contains many worked examples of LLM finetuning using the latest models and techniques +- [Sam Witteveen's YouTube channel](https://www.youtube.com/@samwitteveenai) - videos on a wide range of topics from finetuning to prompt engineering, including many examples of LLM finetuning and explorations of the latest base models
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/finetuning-llms/starter-choices-for-finetuning-llms.md b/docs/book/user-guide/llmops-guide/finetuning-llms/starter-choices-for-finetuning-llms.md index 2666599d4a8..88a18647d15 100644 --- a/docs/book/user-guide/llmops-guide/finetuning-llms/starter-choices-for-finetuning-llms.md +++ b/docs/book/user-guide/llmops-guide/finetuning-llms/starter-choices-for-finetuning-llms.md @@ -4,17 +4,12 @@ description: Get started with finetuning LLMs by picking a use case and data. # Starter choices for finetuning LLMs -Finetuning large language models can be a powerful way to tailor their -capabilities to specific tasks and datasets. This guide will walk you through -the initial steps of finetuning LLMs, including selecting a use case, gathering -the appropriate data, choosing a base model, and evaluating the success of your -finetuning efforts. By following these steps, you can ensure that your -finetuning project is well-scoped, manageable, and aligned with your goals. - -This is a high-level overview before we dive into the code examples, but it's -important to get these decisions right before you start coding. Your use case is -only as good as your data, and you'll need to choose a base model that is -appropriate for your use case. +Finetuning large language models can be a powerful way to tailor their capabilities to specific tasks and datasets. This guide will walk you through +the initial steps of finetuning LLMs, including selecting a use case, gathering the appropriate data, choosing a base model, and evaluating the success of your +finetuning efforts. By following these steps, you can ensure that your finetuning project is well-scoped, manageable, and aligned with your goals. + +This is a high-level overview before we dive into the code examples, but it's important to get these decisions right before you start coding. Your use case is +only as good as your data, and you'll need to choose a base model that is appropriate for your use case. ## 🔍 Quick Assessment Questions @@ -73,10 +68,8 @@ In terms of data volume, a good rule of thumb is that for a result that will be | **Complex Decision Making** | Multiple dependencies, hard to verify | Break down into smaller, measurable subtasks | | **Real-time Content Generation** | Consistency issues, timing constraints | Use templating or hybrid approaches | -As you can see, the challenging use cases are often the ones that are more -open-ended or creative, and so on. With LLMs and finetuning, the real skill is -finding a way to scope down your use case to something that is both small and -manageable, but also where you can still make meaningful progress. +As you can see, the challenging use cases are often the ones that are more open-ended or creative, and so on. With LLMs and finetuning, the real skill is +finding a way to scope down your use case to something that is both small and manageable, but also where you can still make meaningful progress. ### 📊 Success Indicators @@ -90,21 +83,16 @@ You can get a sense of how well-scoped your use case is by considering the follo | **Evaluation Method** | Field-by-field accuracy metrics | "Users will tell us if it's good" | | **Business Impact** | "Save 10 hours of manual data entry" | "Make our AI more human-like" | -You'll want to pick a use case that has a good mix of these indicators and where -you can reasonably expect to be able to measure success in a timely manner. +You'll want to pick a use case that has a good mix of these indicators and where you can reasonably expect to be able to measure success in a timely manner. ## Picking a base model In these early stages, picking the right model probably won't be the most significant choice you make. If you stick to some tried-and-tested base models you will usually be able to get a sense of how well the LLM is able to align itself to your particular task. That said, choosing from the Llama3.1-8B or Mistral-7B families would probably be the best option. -As to whether to go with a base model or one that has been instruction-tuned, -this depends a little on your use case. If your use case is in the area of -structured data extraction (highly recommended to start with something -well-scoped like this) then you're advised to use the base model as it is more -likely to align to this kind of text generation. If you're looking for something -that more resembles a chat-style interface, then an instruction-tuned model is -probably more likely to give you results that suit your purposes. In the end -you'll probably want to try both out to confirm this, but this rule of thumb +As to whether to go with a base model or one that has been instruction-tuned, this depends a little on your use case. If your use case is in the area of +structured data extraction (highly recommended to start with something well-scoped like this) then you're advised to use the base model as it is more +likely to align to this kind of text generation. If you're looking for something that more resembles a chat-style interface, then an instruction-tuned model is +probably more likely to give you results that suit your purposes. In the end you'll probably want to try both out to confirm this, but this rule of thumb should give you a sense of what to start with. ### 📊 Quick Model Selection Matrix @@ -159,10 +147,8 @@ These are all covered in more detail in the [evaluation section](./evaluation-fo ## Next steps -Now that you have a clear understanding of how to scope your finetuning project, -select appropriate data, and evaluate results, you're ready to dive into the -technical implementation. In the next section, we'll walk through [a practical example of finetuning using the Accelerate library](./finetuning-with-accelerate.md), -showing you how to implement +Now that you have a clear understanding of how to scope your finetuning project, select appropriate data, and evaluate results, you're ready to dive into the +technical implementation. In the next section, we'll walk through [a practical example of finetuning using the Accelerate library](./finetuning-with-accelerate.md), showing you how to implement these concepts in code.
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/finetuning-llms/why-and-when-to-finetune-llms.md b/docs/book/user-guide/llmops-guide/finetuning-llms/why-and-when-to-finetune-llms.md index da315530160..ba6590f8cfc 100644 --- a/docs/book/user-guide/llmops-guide/finetuning-llms/why-and-when-to-finetune-llms.md +++ b/docs/book/user-guide/llmops-guide/finetuning-llms/why-and-when-to-finetune-llms.md @@ -4,8 +4,7 @@ description: Deciding when is the right time to finetune LLMs. # Why and when to finetune LLMs -This guide is intended to be a practical overview that gets you started with -finetuning models on your custom data and use cases. Before we dive into the details of this, it's worth taking a moment to bear in mind the following: +This guide is intended to be a practical overview that gets you started with finetuning models on your custom data and use cases. Before we dive into the details of this, it's worth taking a moment to bear in mind the following: - LLM finetuning is not a universal solution or approach: it won't and cannot solve every problem, it might not reach the required levels of accuracy or performance for your use case and you should know that by going the route of finetuning you are taking on a not-inconsiderable amount of technical debt. - Chatbot-style interfaces are not the only way LLMs can be used: there are lots of uses for LLMs and this finetuning approach which don't include any kind of chatbot. What's more, these non-chatbot interfaces should often to be considered preferable since the surface area of failure is much lower. @@ -15,22 +14,19 @@ finetuning models on your custom data and use cases. Before we dive into the det Finetuning an LLM can be a powerful approach in certain scenarios. Here are some situations where it might make sense: -1. **Domain-specific knowledge**: When you need the model to have deep understanding - of a particular domain (e.g., medical, legal, or technical fields) that isn't +1. **Domain-specific knowledge**: When you need the model to have deep understanding of a particular domain (e.g., medical, legal, or technical fields) that isn't well-represented in the base model's training data. Usually, RAG will be a better choice for novel domains, but if you have a lot of data and a very specific use case, finetuning might be the way to go. -2. **Consistent style or format**: If you require outputs in a very specific style - or format that the base model doesn't naturally produce. This is especially +2. **Consistent style or format**: If you require outputs in a very specific style or format that the base model doesn't naturally produce. This is especially true for things like code generation or structured data generation/extraction. 3. **Improved accuracy on specific tasks**: When you need higher accuracy on particular tasks that are crucial for your application. 4. **Handling proprietary information**: If your use case involves working with confidential or proprietary information that can't be sent to external API endpoints. -5. **Custom instructions or prompts**: If you find yourself repeatedly using the - same set of instructions or prompts, finetuning can bake these into the model +5. **Custom instructions or prompts**: If you find yourself repeatedly using the same set of instructions or prompts, finetuning can bake these into the model itself. This might save you latency and costs compared to repeatedly sending the same prompt to an API. 6. **Improved efficiency**: Finetuning can sometimes lead to better performance with shorter prompts, potentially reducing costs and latency. @@ -80,7 +76,6 @@ Before deciding to finetune an LLM, consider these alternatives: Finetuning LLMs can be a powerful tool when used appropriately, but it's important to carefully consider whether it's the best approach for your specific use case. Always start with simpler solutions and move towards finetuning only when you've exhausted other options and have a clear need for the benefits it provides. -In the next section we'll look at some of the practical considerations you have -to take into account when finetuning LLMs. +In the next section we'll look at some of the practical considerations you have to take into account when finetuning LLMs.
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/rag-with-zenml/README.md b/docs/book/user-guide/llmops-guide/rag-with-zenml/README.md index de495d3ff43..ac2ffdbf733 100644 --- a/docs/book/user-guide/llmops-guide/rag-with-zenml/README.md +++ b/docs/book/user-guide/llmops-guide/rag-with-zenml/README.md @@ -4,33 +4,25 @@ description: RAG is a sensible way to get started with LLMs. # RAG Pipelines with ZenML -Retrieval-Augmented Generation (RAG) is a powerful technique that combines the -strengths of retrieval-based and generation-based models. In this guide, we'll -explore how to set up RAG pipelines with ZenML, including data ingestion, index -store management, and tracking RAG-associated artifacts. +Retrieval-Augmented Generation (RAG) is a powerful technique that combines the strengths of retrieval-based and generation-based models. In this guide, we'll +explore how to set up RAG pipelines with ZenML, including data ingestion, index store management, and tracking RAG-associated artifacts. -LLMs are a powerful tool, as they can generate human-like responses to a wide -variety of prompts. However, they can also be prone to generating incorrect or -inappropriate responses, especially when the input prompt is ambiguous or -misleading. They are also (currently) limited in the amount of text they can +LLMs are a powerful tool, as they can generate human-like responses to a wide variety of prompts. However, they can also be prone to generating incorrect or +inappropriate responses, especially when the input prompt is ambiguous or misleading. They are also (currently) limited in the amount of text they can understand and/or generate. While there are some LLMs [like Google's Gemini 1.5 Pro](https://developers.googleblog.com/2024/02/gemini-15-available-for-private-preview-in-google-ai-studio.html) -that can consistently handle 1 million tokens (small units of text), the vast majority (particularly -the open-source ones currently available) handle far less. +that can consistently handle 1 million tokens (small units of text), the vast majority (particularly the open-source ones currently available) handle far less. -The first part of this guide to RAG pipelines with ZenML is about understanding -the basic components and how they work together. We'll cover the following +The first part of this guide to RAG pipelines with ZenML is about understanding the basic components and how they work together. We'll cover the following topics: - why RAG exists and what problem it solves - how to ingest and preprocess data that we'll use in our RAG pipeline -- how to leverage embeddings to represent our data; this will be the basis for - our retrieval mechanism +- how to leverage embeddings to represent our data; this will be the basis for our retrieval mechanism - how to store these embeddings in a vector database - how to track RAG-associated artifacts with ZenML -At the end, we'll bring it all together and show all the components working -together to perform basic RAG inference. +At the end, we'll bring it all together and show all the components working together to perform basic RAG inference.
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/rag-with-zenml/basic-rag-inference-pipeline.md b/docs/book/user-guide/llmops-guide/rag-with-zenml/basic-rag-inference-pipeline.md index 7539efb0d63..86ee759f1b3 100644 --- a/docs/book/user-guide/llmops-guide/rag-with-zenml/basic-rag-inference-pipeline.md +++ b/docs/book/user-guide/llmops-guide/rag-with-zenml/basic-rag-inference-pipeline.md @@ -4,15 +4,12 @@ description: Use your RAG components to generate responses to prompts. # Simple RAG Inference -Now that we have our index store, we can use it to make queries based on the -documents in the index store. We use some utility functions to make this happen -but no external libraries are needed beyond an interface to the index store as -well as the LLM itself. +Now that we have our index store, we can use it to make queries based on the documents in the index store. We use some utility functions to make this happen +but no external libraries are needed beyond an interface to the index store as well as the LLM itself. ![](../../../.gitbook/assets/rag-stage-4.png) -If you've been following along with the guide, you should have some documents -ingested already and you can pass a query in as a flag to the Python command +If you've been following along with the guide, you should have some documents ingested already and you can pass a query in as a flag to the Python command used to run the pipeline: ```bash @@ -22,10 +19,8 @@ steps? i.e. how do I set it? inside the @step decorator?" --model=gpt4 ![](../../../.gitbook/assets/rag-inference.png) -This inference query itself is not a ZenML pipeline, but rather a function call -which uses the outputs and components of our pipeline to generate the response. -For a more complex inference setup, there might be even more going on here, but -for the purposes of this initial guide we will keep it simple. +This inference query itself is not a ZenML pipeline, but rather a function call which uses the outputs and components of our pipeline to generate the response. +For a more complex inference setup, there might be even more going on here, but for the purposes of this initial guide we will keep it simple. Bringing everything together, the code for the inference pipeline is as follows: @@ -70,8 +65,7 @@ def process_input_with_retrieval( return get_completion_from_messages(messages, model=model) ``` -For the `get_topn_similar_docs` function, we use the embeddings generated from -the documents in the index store to find the most similar documents to the +For the `get_topn_similar_docs` function, we use the embeddings generated from the documents in the index store to find the most similar documents to the query: ```python @@ -105,17 +99,12 @@ def get_topn_similar_docs( return cur.fetchall() ``` -Luckily we are able to get these similar documents using a function in -[`pgvector`](https://github.com/pgvector/pgvector), a plugin package for -PostgreSQL: `ORDER BY embedding <=> %s` orders the documents by their similarity -to the query embedding. This is a very efficient way to get the most relevant -documents to the query and is a great example of how we can leverage the power -of the database to do the heavy lifting for us. - -For the `get_completion_from_messages` function, we use -[`litellm`](https://github.com/BerriAI/litellm) as a universal interface that -allows us to use lots of different LLMs. As you can see above, the model is able -to synthesize the documents it has been given and provide a response to the +Luckily we are able to get these similar documents using a function in [`pgvector`](https://github.com/pgvector/pgvector), a plugin package for +PostgreSQL: `ORDER BY embedding <=> %s` orders the documents by their similarity to the query embedding. This is a very efficient way to get the most relevant +documents to the query and is a great example of how we can leverage the power of the database to do the heavy lifting for us. + +For the `get_completion_from_messages` function, we use [`litellm`](https://github.com/BerriAI/litellm) as a universal interface that +allows us to use lots of different LLMs. As you can see above, the model is able to synthesize the documents it has been given and provide a response to the query. ```python @@ -133,22 +122,15 @@ def get_completion_from_messages( return completion_response.choices[0].message.content ``` -We're using `litellm` because it makes sense not to have to implement separate -functions for each LLM we might want to use. The pace of development in the -field is such that you will want to experiment with new LLMs as they come out, -and `litellm` gives you the flexibility to do that without having to rewrite +We're using `litellm` because it makes sense not to have to implement separate functions for each LLM we might want to use. The pace of development in the +field is such that you will want to experiment with new LLMs as they come out, and `litellm` gives you the flexibility to do that without having to rewrite your code. -We've now completed a basic RAG inference pipeline that uses the embeddings -generated by the pipeline to retrieve the most relevant chunks of text based on -a given query. We can inspect the various components of the pipeline to see how -they work together to provide a response to the query. This gives us a solid -foundation to move onto more complex RAG pipelines and to look into how we might -improve this. The next section will cover how to improve retrieval by finetuning -the embeddings generated by the pipeline. This will boost our performance in -situations where we have a large volume of documents and also when the documents -are potentially very different from the training data that was used for the -embeddings. +We've now completed a basic RAG inference pipeline that uses the embeddings generated by the pipeline to retrieve the most relevant chunks of text based on +a given query. We can inspect the various components of the pipeline to see how they work together to provide a response to the query. This gives us a solid +foundation to move onto more complex RAG pipelines and to look into how we might improve this. The next section will cover how to improve retrieval by finetuning +the embeddings generated by the pipeline. This will boost our performance in situations where we have a large volume of documents and also when the documents +are potentially very different from the training data that was used for the embeddings. ## Code Example diff --git a/docs/book/user-guide/llmops-guide/rag-with-zenml/data-ingestion.md b/docs/book/user-guide/llmops-guide/rag-with-zenml/data-ingestion.md index e3bc175544c..18778bb68ae 100644 --- a/docs/book/user-guide/llmops-guide/rag-with-zenml/data-ingestion.md +++ b/docs/book/user-guide/llmops-guide/rag-with-zenml/data-ingestion.md @@ -2,30 +2,22 @@ description: Understand how to ingest and preprocess data for RAG pipelines with ZenML. --- -The first step in setting up a RAG pipeline is to ingest the data that will be -used to train and evaluate the retriever and generator models. This data can -include a large corpus of documents, as well as any relevant metadata or -annotations that can be used to train the retriever and generator. +The first step in setting up a RAG pipeline is to ingest the data that will be used to train and evaluate the retriever and generator models. This data can +include a large corpus of documents, as well as any relevant metadata or annotations that can be used to train the retriever and generator. ![](../../../.gitbook/assets/rag-stage-1.png) -In the interests of keeping things simple, we'll implement the bulk of what we -need ourselves. However, it's worth noting that there are a number of tools and -frameworks that can help you manage the data ingestion process, including -downloading, preprocessing, and indexing large corpora of documents. ZenML -integrates with a number of these tools and frameworks, making it easy to set up -and manage RAG pipelines. +In the interests of keeping things simple, we'll implement the bulk of what we need ourselves. However, it's worth noting that there are a number of tools and +frameworks that can help you manage the data ingestion process, including downloading, preprocessing, and indexing large corpora of documents. ZenML +integrates with a number of these tools and frameworks, making it easy to set up and manage RAG pipelines. {% hint style="info" %} -You can view all the code referenced in this guide in the associated project -repository. Please visit the -`llm-complete-guide` project inside the ZenML projects repository if you +You can view all the code referenced in this guide in the associated project repository. Please visit the `llm-complete-guide` project inside the ZenML projects repository if you want to dive deeper. {% endhint %} -You can add a ZenML step that scrapes a series of URLs and outputs the URLs quite -easily. Here we assemble a step that scrapes URLs related to ZenML from its documentation. +You can add a ZenML step that scrapes a series of URLs and outputs the URLs quite easily. Here we assemble a step that scrapes URLs related to ZenML from its documentation. We leverage some simple helper utilities that we have created for this purpose: ```python @@ -50,25 +42,18 @@ def url_scraper( return docs_urls ``` -The `get_all_pages` function simply crawls our documentation website and -retrieves a unique set of URLs. We've limited it to only scrape the -documentation relating to the most recent releases so that we're not mixing old -syntax and information with the new. This is a simple way to ensure that we're +The `get_all_pages` function simply crawls our documentation website and retrieves a unique set of URLs. We've limited it to only scrape the +documentation relating to the most recent releases so that we're not mixing old syntax and information with the new. This is a simple way to ensure that we're only ingesting the most relevant and up-to-date information into our pipeline. -We also log the count of those URLs as metadata for the step output. This will -be visible in the dashboard for extra visibility around the data that's being -ingested. Of course, you can also add more complex logic to this step, such as -filtering out certain URLs or adding more metadata. +We also log the count of those URLs as metadata for the step output. This will be visible in the dashboard for extra visibility around the data that's being +ingested. Of course, you can also add more complex logic to this step, such as filtering out certain URLs or adding more metadata. ![Partial screenshot from the dashboard showing the metadata from the step](../../../.gitbook/assets/llm-data-ingestion-metadata.png) -Once we have our list of URLs, we use [the `unstructured` -library](https://github.com/Unstructured-IO/unstructured) to load and parse the -pages. This will allow us to use the text without having to worry about the -details of the HTML structure and/or markup. This specifically helps us keep the -text -content as small as possible since we are operating in a constrained environment +Once we have our list of URLs, we use [the `unstructured` library](https://github.com/Unstructured-IO/unstructured) to load and parse the +pages. This will allow us to use the text without having to worry about the details of the HTML structure and/or markup. This specifically helps us keep the +text content as small as possible since we are operating in a constrained environment with LLMs. ```python @@ -87,26 +72,19 @@ def web_url_loader(urls: List[str]) -> List[str]: return document_texts ``` -The previously-mentioned frameworks offer many more options when it comes to -data ingestion, including the ability to load documents from a variety of -sources, preprocess the text, and extract relevant features. For our purposes, -though, we don't need anything too fancy. It also makes our pipeline easier to +The previously-mentioned frameworks offer many more options when it comes to data ingestion, including the ability to load documents from a variety of +sources, preprocess the text, and extract relevant features. For our purposes, though, we don't need anything too fancy. It also makes our pipeline easier to debug since we can see exactly what's being loaded and how it's being processed. You don't get that same level of visibility with more complex frameworks. # Preprocessing the data -Once we have loaded the documents, we can preprocess them into a form that's -useful for a RAG pipeline. There are a lot of options here, depending on how -complex you want to get, but to start with you can think of the 'chunk size' as -one of the key parameters to think about. - -Our text is currently in the form of various long strings, with each one -representing a single web page. These are going to be too long to pass into our -LLM, especially if we care about the speed at which we get our answers back. So -the strategy here is to split our text into smaller chunks that can be processed -more efficiently. There's a sweet spot between having tiny chunks, which will -make it harder for our search / retrieval step to find relevant information to +Once we have loaded the documents, we can preprocess them into a form that's useful for a RAG pipeline. There are a lot of options here, depending on how +complex you want to get, but to start with you can think of the 'chunk size' as one of the key parameters to think about. + +Our text is currently in the form of various long strings, with each one representing a single web page. These are going to be too long to pass into our +LLM, especially if we care about the speed at which we get our answers back. So the strategy here is to split our text into smaller chunks that can be processed +more efficiently. There's a sweet spot between having tiny chunks, which will make it harder for our search / retrieval step to find relevant information to pass into the LLM, and having large chunks, which will make it harder for the LLM to process the text. @@ -140,33 +118,25 @@ def preprocess_documents( raise ``` -It's really important to know your data to have a good intuition about what kind -of chunk size might make sense. If your data is structured in such a way where -you need large paragraphs to capture a particular concept, then you might want a -larger chunk size. If your data is more conversational or question-and-answer +It's really important to know your data to have a good intuition about what kind of chunk size might make sense. If your data is structured in such a way where +you need large paragraphs to capture a particular concept, then you might want a larger chunk size. If your data is more conversational or question-and-answer based, then you might want a smaller chunk size. -For our purposes, given that we're working with web pages that are written as -documentation for a software library, we're going to use a chunk size of 500 and -we'll make sure that the chunks overlap by 50 characters. This means that we'll -have a lot of overlap between our chunks, which can be useful for ensuring that +For our purposes, given that we're working with web pages that are written as documentation for a software library, we're going to use a chunk size of 500 and +we'll make sure that the chunks overlap by 50 characters. This means that we'll have a lot of overlap between our chunks, which can be useful for ensuring that we don't miss any important information when we're splitting up our text. -Again, depending on your data and use case, there is more you might want to do -with your data. You might want to clean the text, remove code snippets or make -sure that code snippets were not split across chunks, or even extract metadata -from the text. This is a good starting point, but you can always add more +Again, depending on your data and use case, there is more you might want to do with your data. You might want to clean the text, remove code snippets or make +sure that code snippets were not split across chunks, or even extract metadata from the text. This is a good starting point, but you can always add more complexity as needed. -Next up, generating embeddings so that we can use them to retrieve relevant -documents... +Next up, generating embeddings so that we can use them to retrieve relevant documents... ## Code Example To explore the full code, visit the [Complete Guide](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide) -repository and particularly [the code for the steps](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide/steps/) in this section. Note, too, -that a lot of the logic is encapsulated in utility functions inside [`url_scraping_utils.py`](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide/steps/url_scraping_utils.py). +repository and particularly [the code for the steps](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide/steps/) in this section. Note, too, that a lot of the logic is encapsulated in utility functions inside [`url_scraping_utils.py`](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide/steps/url_scraping_utils.py).
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/rag-with-zenml/embeddings-generation.md b/docs/book/user-guide/llmops-guide/rag-with-zenml/embeddings-generation.md index a5d21b0f00a..c9c3c6093e0 100644 --- a/docs/book/user-guide/llmops-guide/rag-with-zenml/embeddings-generation.md +++ b/docs/book/user-guide/llmops-guide/rag-with-zenml/embeddings-generation.md @@ -4,37 +4,24 @@ description: Generate embeddings to improve retrieval performance. # Generating Embeddings for Retrieval -In this section, we'll explore how to generate embeddings for your data to -improve retrieval performance in your RAG pipeline. Embeddings are a crucial -part of the retrieval mechanism in RAG, as they represent the data in a -high-dimensional space where similar items are closer together. By generating -embeddings for your data, you can enhance the retrieval capabilities of your RAG -pipeline and provide more accurate and relevant responses to user queries. +In this section, we'll explore how to generate embeddings for your data to improve retrieval performance in your RAG pipeline. Embeddings are a crucial +part of the retrieval mechanism in RAG, as they represent the data in a high-dimensional space where similar items are closer together. By generating +embeddings for your data, you can enhance the retrieval capabilities of your RAG pipeline and provide more accurate and relevant responses to user queries. ![](../../../.gitbook/assets/rag-stage-2.png) -{% hint style="info" %} Embeddings are vector representations of data that capture the semantic -meaning and context of the data in a high-dimensional space. They are generated -using machine learning models, such as word embeddings or sentence embeddings, -that learn to encode the data in a way that preserves its underlying structure -and relationships. Embeddings are commonly used in natural language processing -(NLP) tasks, such as text classification, sentiment analysis, and information -retrieval, to represent textual data in a format that is suitable for -computational processing. {% endhint %} - -The whole purpose of the embeddings is to allow us to quickly find the small -chunks that are most relevant to our input query at inference time. An even -simpler way of doing this would be to just to search for some keywords in the -query and hope that they're also represented in the chunks. However, this -approach is not very robust and may not work well for more complex queries or -longer documents. By using embeddings, we can capture the semantic meaning and -context of the data and retrieve the most relevant chunks based on their -similarity to the query. - -We're using the [`sentence-transformers`](https://www.sbert.net/) library to generate embeddings for our -data. This library provides pre-trained models for generating sentence -embeddings that capture the semantic meaning of the text. It's an open-source -library that is easy to use and provides high-quality embeddings for a wide +{% hint style="info" %} Embeddings are vector representations of data that capture the semantic meaning and context of the data in a high-dimensional space. They are generated +using machine learning models, such as word embeddings or sentence embeddings, that learn to encode the data in a way that preserves its underlying structure +and relationships. Embeddings are commonly used in natural language processing (NLP) tasks, such as text classification, sentiment analysis, and information +retrieval, to represent textual data in a format that is suitable for computational processing. {% endhint %} + +The whole purpose of the embeddings is to allow us to quickly find the small chunks that are most relevant to our input query at inference time. An even +simpler way of doing this would be to just to search for some keywords in the query and hope that they're also represented in the chunks. However, this +approach is not very robust and may not work well for more complex queries or longer documents. By using embeddings, we can capture the semantic meaning and +context of the data and retrieve the most relevant chunks based on their similarity to the query. + +We're using the [`sentence-transformers`](https://www.sbert.net/) library to generate embeddings for our data. This library provides pre-trained models for generating sentence +embeddings that capture the semantic meaning of the text. It's an open-source library that is easy to use and provides high-quality embeddings for a wide range of NLP tasks. ```python @@ -73,29 +60,20 @@ def generate_embeddings( raise ``` -We update the `Document` Pydantic model to include an `embedding` attribute that -stores the embedding generated for each document. This allows us to associate -the embeddings with the corresponding documents and use them for retrieval -purposes in the RAG pipeline. - -There are smaller embeddings models if we cared a lot about speed, and larger -ones (with more dimensions) if we wanted to boost our ability to retrieve more -relevant chunks. [The model we're using -here](https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2) is on the -smaller side, but it should work well for our use case. The embeddings generated -by this model have a dimensionality of 384, which means that each embedding is +We update the `Document` Pydantic model to include an `embedding` attribute that stores the embedding generated for each document. This allows us to associate +the embeddings with the corresponding documents and use them for retrieval purposes in the RAG pipeline. + +There are smaller embeddings models if we cared a lot about speed, and larger ones (with more dimensions) if we wanted to boost our ability to retrieve more +relevant chunks. [The model we're using here](https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2) is on the +smaller side, but it should work well for our use case. The embeddings generated by this model have a dimensionality of 384, which means that each embedding is represented as a 384-dimensional vector in the high-dimensional space. -We can use dimensionality reduction functionality in -[`umap`](https://umap-learn.readthedocs.io/) and +We can use dimensionality reduction functionality in [`umap`](https://umap-learn.readthedocs.io/) and [`scikit-learn`](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html#sklearn-manifold-tsne) -to represent the 384 dimensions of our embeddings in two-dimensional space. This -allows us to visualize the embeddings and see how similar chunks are clustered -together based on their semantic meaning and context. We can also use this -visualization to identify patterns and relationships in the data that can help +to represent the 384 dimensions of our embeddings in two-dimensional space. This allows us to visualize the embeddings and see how similar chunks are clustered +together based on their semantic meaning and context. We can also use this visualization to identify patterns and relationships in the data that can help us improve the retrieval performance of our RAG pipeline. It's worth trying both -UMAP and t-SNE to see which one works best for our use case since they both have -somewhat different representations of the data and reduction algorithms, as +UMAP and t-SNE to see which one works best for our use case since they both have somewhat different representations of the data and reduction algorithms, as you'll see. ```python @@ -181,31 +159,23 @@ def umap_visualization(embeddings, parent_sections): ![UMAP visualization of the ZenML documentation chunks as embeddings](../../../.gitbook/assets/umap.png) ![t-SNE visualization of the ZenML documentation chunks as embeddings](../../../.gitbook/assets/tsne.png) -In this stage, we have utilized the 'parent directory', which we had previously -stored in the vector store as an additional attribute, as a means to color the -values. This approach allows us to gain some insight into the semantic space -inherent in our data. It demonstrates that you can visualize the embeddings and -observe how similar chunks are grouped together based on their semantic meaning -and context. - -So this step iterates through all the chunks and generates embeddings -representing each piece of text. These embeddings are then stored as an artifact -in the ZenML artifact store as a NumPy array. We separate this generation from -the point where we upload those embeddings to the vector database to keep the -pipeline modular and flexible; in the future we might want to use a different -vector database so we can just swap out the upload step without having to +In this stage, we have utilized the 'parent directory', which we had previously stored in the vector store as an additional attribute, as a means to color the +values. This approach allows us to gain some insight into the semantic space inherent in our data. It demonstrates that you can visualize the embeddings and +observe how similar chunks are grouped together based on their semantic meaning and context. + +So this step iterates through all the chunks and generates embeddings representing each piece of text. These embeddings are then stored as an artifact +in the ZenML artifact store as a NumPy array. We separate this generation from the point where we upload those embeddings to the vector database to keep the +pipeline modular and flexible; in the future we might want to use a different vector database so we can just swap out the upload step without having to re-generate the embeddings. -In the next section, we'll explore how to store these embeddings in a vector -database to enable fast and efficient retrieval of relevant chunks at inference +In the next section, we'll explore how to store these embeddings in a vector database to enable fast and efficient retrieval of relevant chunks at inference time. ## Code Example To explore the full code, visit the [Complete Guide](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide) -repository. The embeddings generation step can be found -[here](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide/steps/populate_index.py). +repository. The embeddings generation step can be found [here](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide/steps/populate_index.py).
ZenML Scarf
diff --git a/docs/book/user-guide/llmops-guide/rag-with-zenml/rag-85-loc.md b/docs/book/user-guide/llmops-guide/rag-with-zenml/rag-85-loc.md index c5ca98f5964..87d12b1d59f 100644 --- a/docs/book/user-guide/llmops-guide/rag-with-zenml/rag-85-loc.md +++ b/docs/book/user-guide/llmops-guide/rag-with-zenml/rag-85-loc.md @@ -2,17 +2,13 @@ description: Learn how to implement a RAG pipeline in just 85 lines of code. --- -There's a lot of theory and context to think about when it comes to RAG, but -let's start with a quick implementation in code to motivate what follows. The +There's a lot of theory and context to think about when it comes to RAG, but let's start with a quick implementation in code to motivate what follows. The following 85 lines do the following: - load some data (a fictional dataset about 'ZenML World') as our corpus -- process that text (split it into chunks and 'tokenize' it (i.e. split into - words)) -- take a query as input and find the most relevant chunks of text from our - corpus data -- use OpenAI's GPT-3.5 model to answer the question based on the relevant - chunks +- process that text (split it into chunks and 'tokenize' it (i.e. split into words)) +- take a query as input and find the most relevant chunks of text from our corpus data +- use OpenAI's GPT-3.5 model to answer the question based on the relevant chunks ```python import os @@ -113,24 +109,17 @@ Question: What is the capital of Panglossia? Answer: The capital of Panglossia is not mentioned in the provided context. ``` -The implementation above is by no means sophisticated or performant, but it's -simple enough that you can see all the moving parts. Our tokenization process +The implementation above is by no means sophisticated or performant, but it's simple enough that you can see all the moving parts. Our tokenization process consists of splitting the text into individual words. -The way we check for similarity between the question / query and the chunks of -text is extremely naive and inefficient. The similarity between the query and -the current chunk is calculated using the [Jaccard similarity -coefficient](https://www.statology.org/jaccard-similarity/). This coefficient -measures the similarity between two sets and is defined as the size of the -intersection divided by the size of the union of the two sets. So we count the -number of words that are common between the query and the chunk and divide it by -the total number of unique words in both the query and the chunk. There are much -better ways of measuring the similarity between two pieces of text, such as -using embeddings or other more sophisticated techniques, but this example is +The way we check for similarity between the question / query and the chunks of text is extremely naive and inefficient. The similarity between the query and +the current chunk is calculated using the [Jaccard similarity coefficient](https://www.statology.org/jaccard-similarity/). This coefficient +measures the similarity between two sets and is defined as the size of the intersection divided by the size of the union of the two sets. So we count the +number of words that are common between the query and the chunk and divide it by the total number of unique words in both the query and the chunk. There are much +better ways of measuring the similarity between two pieces of text, such as using embeddings or other more sophisticated techniques, but this example is kept simple for illustrative purposes. -The rest of this guide will showcase a more performant and scalable way of -performing the same task using ZenML. If you ever are unsure why we're doing +The rest of this guide will showcase a more performant and scalable way of performing the same task using ZenML. If you ever are unsure why we're doing something, feel free to return to this example for the high-level overview. diff --git a/docs/book/user-guide/llmops-guide/reranking/README.md b/docs/book/user-guide/llmops-guide/reranking/README.md index 729e22e821c..f3617e612cd 100644 --- a/docs/book/user-guide/llmops-guide/reranking/README.md +++ b/docs/book/user-guide/llmops-guide/reranking/README.md @@ -2,23 +2,17 @@ description: Add reranking to your RAG inference for better retrieval performance. --- -Rerankers are a crucial component of retrieval systems that use LLMs. They help -improve the quality of the retrieved documents by reordering them based on -additional features or scores. In this section, we'll explore how to add a -reranker to your RAG inference pipeline in ZenML. +Rerankers are a crucial component of retrieval systems that use LLMs. They help improve the quality of the retrieved documents by reordering them based on +additional features or scores. In this section, we'll explore how to add a reranker to your RAG inference pipeline in ZenML. -In previous sections, we set up the overall workflow, from data ingestion and -preprocessing to embeddings generation and retrieval. We then set up some basic -evaluation metrics to assess the performance of our retrieval system. A reranker -is a way to squeeze a bit of extra performance out of the system by reordering +In previous sections, we set up the overall workflow, from data ingestion and preprocessing to embeddings generation and retrieval. We then set up some basic +evaluation metrics to assess the performance of our retrieval system. A reranker is a way to squeeze a bit of extra performance out of the system by reordering the retrieved documents based on additional features or scores. ![](../../../.gitbook/assets/reranking-workflow.png) -As you can see, reranking is an optional addition we make to what we've already -set up. It's not strictly necessary, but it can help improve the relevance and -quality of the retrieved documents, which in turn can lead to better responses -from the LLM. Let's dive in! +As you can see, reranking is an optional addition we make to what we've already set up. It's not strictly necessary, but it can help improve the relevance and +quality of the retrieved documents, which in turn can lead to better responses from the LLM. Let's dive in! diff --git a/docs/book/user-guide/llmops-guide/reranking/implementing-reranking.md b/docs/book/user-guide/llmops-guide/reranking/implementing-reranking.md index dc87fc5d4b2..98e7a25ec5c 100644 --- a/docs/book/user-guide/llmops-guide/reranking/implementing-reranking.md +++ b/docs/book/user-guide/llmops-guide/reranking/implementing-reranking.md @@ -4,26 +4,19 @@ description: Learn how to implement reranking in ZenML. # Implementing Reranking in ZenML -We already have a working RAG pipeline, so inserting a reranker into the -pipeline is relatively straightforward. The reranker will take the retrieved -documents from the initial retrieval step and reorder them in terms of the query -that was used to retrieve them. +We already have a working RAG pipeline, so inserting a reranker into the pipeline is relatively straightforward. The reranker will take the retrieved +documents from the initial retrieval step and reorder them in terms of the query that was used to retrieve them. ![](../../../.gitbook/assets/reranking-workflow.png) ## How and where to add reranking -We'll use the [`rerankers`](https://github.com/AnswerDotAI/rerankers/) package -to handle the reranking process in our RAG inference pipeline. It's a relatively -low-cost (in terms of technical debt and complexity) and lightweight dependency -to add into our pipeline. It offers an interface to most of the model types that -are commonly used for reranking and means we don't have to worry about the -specifics of each model. - -This package provides a `Reranker` abstract class that you can use to define -your own reranker. You can also use the provided implementations to add -reranking to your pipeline. The reranker takes the query and a list of retrieved -documents as input and outputs a reordered list of documents based on the +We'll use the [`rerankers`](https://github.com/AnswerDotAI/rerankers/) package to handle the reranking process in our RAG inference pipeline. It's a relatively +low-cost (in terms of technical debt and complexity) and lightweight dependency to add into our pipeline. It offers an interface to most of the model types that +are commonly used for reranking and means we don't have to worry about the specifics of each model. + +This package provides a `Reranker` abstract class that you can use to define your own reranker. You can also use the provided implementations to add +reranking to your pipeline. The reranker takes the query and a list of retrieved documents as input and outputs a reordered list of documents based on the reranking scores. Here's a toy example: ```python @@ -60,19 +53,14 @@ RankedResults( ) ``` -We can see that the reranker has reordered the documents based on the reranking -scores, with the most relevant document appearing at the top of the list. The -texts about sport are at the top and the less relevant ones about animals are -down at the bottom. +We can see that the reranker has reordered the documents based on the reranking scores, with the most relevant document appearing at the top of the list. The +texts about sport are at the top and the less relevant ones about animals are down at the bottom. -We specified that we want a `cross-encoder` reranker, but you can also use other -reranker models from the Hugging Face Hub, use API-driven reranker models (from -Jina or Cohere, for example), or even define your own reranker model. Read -[their documentation](https://github.com/AnswerDotAI/rerankers/) to see how to +We specified that we want a `cross-encoder` reranker, but you can also use other reranker models from the Hugging Face Hub, use API-driven reranker models (from +Jina or Cohere, for example), or even define your own reranker model. Read [their documentation](https://github.com/AnswerDotAI/rerankers/) to see how to use these different configurations. -In our case, we can simply add a helper function that can optionally be invoked -when we want to use the reranker: +In our case, we can simply add a helper function that can optionally be invoked when we want to use the reranker: ```python @@ -96,12 +84,9 @@ def rerank_documents( return reranked_documents_and_urls ``` -This function takes a query and a list of documents (each document is a tuple of -content and URL) and reranks the documents based on the query. It returns a list -of tuples, where each tuple contains the reranked document text and the URL of -the original document. We use the `flashrank` model from the `rerankers` package -by default as it appeared to be a good choice for our use case during -development. +This function takes a query and a list of documents (each document is a tuple of content and URL) and reranks the documents based on the query. It returns a list +of tuples, where each tuple contains the reranked document text and the URL of the original document. We use the `flashrank` model from the `rerankers` package +by default as it appeared to be a good choice for our use case during development. This function then gets used in tests in the following way: @@ -132,17 +117,12 @@ def query_similar_docs( return (question, url_ending, urls) ``` -We get the embeddings for the question being passed into the function and -connect to our PostgreSQL database. If we're using reranking, we get the top 20 -documents similar to our query and rerank them using the `rerank_documents` -helper function. We then extract the URLs from the reranked documents and return -them. Note that we only return 5 URLs, but in the case of reranking we get a -larger number of documents and URLs back from the database to pass to our -reranker, but in the end we always choose the top five reranked documents to -return. - -Now that we've added reranking to our pipeline, we can evaluate the performance -of our reranker and see how it affects the quality of the retrieved documents. +We get the embeddings for the question being passed into the function and connect to our PostgreSQL database. If we're using reranking, we get the top 20 +documents similar to our query and rerank them using the `rerank_documents` helper function. We then extract the URLs from the reranked documents and return +them. Note that we only return 5 URLs, but in the case of reranking we get a larger number of documents and URLs back from the database to pass to our +reranker, but in the end we always choose the top five reranked documents to return. + +Now that we've added reranking to our pipeline, we can evaluate the performance of our reranker and see how it affects the quality of the retrieved documents. ## Code Example diff --git a/docs/book/user-guide/llmops-guide/reranking/understanding-reranking.md b/docs/book/user-guide/llmops-guide/reranking/understanding-reranking.md index 226c2e6a5c6..9dcac8cc411 100644 --- a/docs/book/user-guide/llmops-guide/reranking/understanding-reranking.md +++ b/docs/book/user-guide/llmops-guide/reranking/understanding-reranking.md @@ -4,26 +4,19 @@ description: Understand how reranking works. ## What is reranking? -Reranking is the process of refining the initial ranking of documents retrieved -by a retrieval system. In the context of Retrieval-Augmented Generation (RAG), -reranking plays a crucial role in improving the relevance and quality of the -retrieved documents that are used to generate the final output. - -The initial retrieval step in RAG typically uses a sparse retrieval method, such -as BM25 or TF-IDF, to quickly find a set of potentially relevant documents based -on the input query. However, these methods rely on lexical matching and may not -capture the semantic meaning or context of the query effectively. - -Rerankers, on the other hand, are designed to reorder the retrieved documents by -considering additional features, such as semantic similarity, relevance scores, -or domain-specific knowledge. They aim to push the most relevant and informative -documents to the top of the list, ensuring that the LLM has access to the best +Reranking is the process of refining the initial ranking of documents retrieved by a retrieval system. In the context of Retrieval-Augmented Generation (RAG), +reranking plays a crucial role in improving the relevance and quality of the retrieved documents that are used to generate the final output. + +The initial retrieval step in RAG typically uses a sparse retrieval method, such as BM25 or TF-IDF, to quickly find a set of potentially relevant documents based +on the input query. However, these methods rely on lexical matching and may not capture the semantic meaning or context of the query effectively. + +Rerankers, on the other hand, are designed to reorder the retrieved documents by considering additional features, such as semantic similarity, relevance scores, +or domain-specific knowledge. They aim to push the most relevant and informative documents to the top of the list, ensuring that the LLM has access to the best possible context for generating accurate and coherent responses. ## Types of Rerankers -There are different types of rerankers that can be used in RAG, each with its -own strengths and trade-offs: +There are different types of rerankers that can be used in RAG, each with its own strengths and trade-offs: 1. **Cross-Encoders**: Cross-encoders are a popular choice for reranking in RAG. They take the concatenated query and document as input and output a relevance @@ -31,14 +24,12 @@ own strengths and trade-offs: tasks. Cross-encoders can capture the interaction between the query and document effectively but are computationally expensive. -2. **Bi-Encoders**: Bi-encoders, also known as dual encoders, use separate - encoders for the query and document. They generate embeddings for the query +2. **Bi-Encoders**: Bi-encoders, also known as dual encoders, use separate encoders for the query and document. They generate embeddings for the query and document independently and then compute the similarity between them. Bi-encoders are more efficient than cross-encoders but may not capture the query-document interaction as effectively. -3. **Lightweight Models**: Lightweight rerankers, such as distilled models or - small transformer variants, aim to strike a balance between effectiveness and +3. **Lightweight Models**: Lightweight rerankers, such as distilled models or small transformer variants, aim to strike a balance between effectiveness and efficiency. They are faster and have a smaller footprint compared to large cross-encoders, making them suitable for real-time applications. @@ -46,26 +37,21 @@ own strengths and trade-offs: Reranking offers several benefits in the context of RAG: -1. **Improved Relevance**: By considering additional features and scores, - rerankers can identify the most relevant documents for a given query, +1. **Improved Relevance**: By considering additional features and scores, rerankers can identify the most relevant documents for a given query, ensuring that the LLM has access to the most informative context for generating accurate responses. -2. **Semantic Understanding**: Rerankers can capture the semantic meaning and - context of the query and documents, going beyond simple keyword matching. +2. **Semantic Understanding**: Rerankers can capture the semantic meaning and context of the query and documents, going beyond simple keyword matching. This enables the retrieval of documents that are semantically similar to the query, even if they don't contain exact keyword matches. -3. **Domain Adaptation**: Rerankers can be fine-tuned on domain-specific data to - incorporate domain knowledge and improve performance in specific verticals or +3. **Domain Adaptation**: Rerankers can be fine-tuned on domain-specific data to incorporate domain knowledge and improve performance in specific verticals or industries. -4. **Personalization**: Rerankers can be personalized based on user preferences, - historical interactions, or user profiles, enabling the retrieval of +4. **Personalization**: Rerankers can be personalized based on user preferences, historical interactions, or user profiles, enabling the retrieval of documents that are more tailored to individual users' needs. -In the next section, we'll dive into how to implement reranking in ZenML and -integrate it into your RAG inference pipeline. +In the next section, we'll dive into how to implement reranking in ZenML and integrate it into your RAG inference pipeline.
ZenML Scarf
diff --git a/docs/book/user-guide/production-guide/ci-cd.md b/docs/book/user-guide/production-guide/ci-cd.md index ae1c8663b69..5e9d4e3a410 100644 --- a/docs/book/user-guide/production-guide/ci-cd.md +++ b/docs/book/user-guide/production-guide/ci-cd.md @@ -6,38 +6,31 @@ description: >- # Set up CI/CD -Until now, we have been executing ZenML pipelines locally. While this is a good mode of operating pipelines, in -production it is often desirable to mediate runs through a central workflow engine baked into your CI. +Until now, we have been executing ZenML pipelines locally. While this is a good mode of operating pipelines, in production it is often desirable to mediate runs through a central workflow engine baked into your CI. -This allows data scientists to experiment with data processing and model training locally and then have code changes -automatically tested and validated through the standard pull request/merge request peer review process. Changes that +This allows data scientists to experiment with data processing and model training locally and then have code changes automatically tested and validated through the standard pull request/merge request peer review process. Changes that pass the CI and code review are then deployed automatically to production. Here is how this could look like: ![Pipeline being run on staging/production stack through ci/cd](../../.gitbook/assets/ci-cd-overall.png) ## Breaking it down -To illustrate this, let's walk through how this process could be set up with -a GitHub Repository. Basically we'll be using Github Actions in order to set up +To illustrate this, let's walk through how this process could be set up with a GitHub Repository. Basically we'll be using Github Actions in order to set up a proper CI/CD workflow. {% hint style="info" %} -To see this in action, check out the [ZenML Gitflow Repository](https://github.com/zenml-io/zenml-gitflow/). This -repository showcases how ZenML can be used for machine learning with a GitHub workflow that automates CI/CD with -continuous model training and continuous model deployment to production. The repository is also meant to be used as a -template: you can fork it and easily adapt it to your own MLOps stack, infrastructure, code and data.{% endhint %} +To see this in action, check out the [ZenML Gitflow Repository](https://github.com/zenml-io/zenml-gitflow/). This repository showcases how ZenML can be used for machine learning with a GitHub workflow that automates CI/CD with +continuous model training and continuous model deployment to production. The repository is also meant to be used as a template: you can fork it and easily adapt it to your own MLOps stack, infrastructure, code and data.{% endhint %} ### Configure an API Key in ZenML -In order to facilitate machine-to-machine connection you need to create an API key within ZenML. Learn more about those -[here](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account). +In order to facilitate machine-to-machine connection you need to create an API key within ZenML. Learn more about those [here](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account). ```bash zenml service-account create github_action_api_key ``` -This will return the API Key to you like this. This will not be shown to you again, so make sure to copy it here for -use in the next section. +This will return the API Key to you like this. This will not be shown to you again, so make sure to copy it here for use in the next section. ```bash Created service account 'github_action_api_key'. @@ -51,34 +44,28 @@ To configure a ZenML client to use this API key, run: ### Set up your secrets in Github -For our Github Actions we will need to set up some -secrets [for our repository](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#creating-secrets-for-a-repository). +For our Github Actions we will need to set up some secrets [for our repository](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#creating-secrets-for-a-repository). Specifically, you should use github secrets to store the `ZENML_API_KEY` that you created above. ![create_gh_secret.png](../../.gitbook/assets/create_gh_secret.png) -The other values that are loaded from secrets into the -environment [here](https://github.com/zenml-io/zenml-gitflow/blob/main/.github/workflows/pipeline_run.yaml#L14-L23) +The other values that are loaded from secrets into the environment [here](https://github.com/zenml-io/zenml-gitflow/blob/main/.github/workflows/pipeline_run.yaml#L14-L23) can also be set explicitly or as variables. ### (Optional) Set up different stacks for Staging and Production You might not necessarily want to use the same stack with the same resources for your staging and production use. -This step is optional, all you'll need for certain is a stack that runs remotely (remote orchestration and artifact -storage). The rest is up to you. You might for example want to parametrize your pipeline to use different data sources +This step is optional, all you'll need for certain is a stack that runs remotely (remote orchestration and artifact storage). The rest is up to you. You might for example want to parametrize your pipeline to use different data sources for the respective environments. You can also use different [configuration files](https://docs.zenml.io/concepts/steps_and_pipelines/yaml_configuration) -for the different environments to configure the [Model](https://docs.zenml.io/how-to/model-management-metrics/model-control-plane), the -[DockerSettings](https://docs.zenml.io/how-to/customize-docker-builds/docker-settings-on-a-pipeline), the +for the different environments to configure the [Model](https://docs.zenml.io/how-to/model-management-metrics/model-control-plane), the [DockerSettings](https://docs.zenml.io/how-to/customize-docker-builds/docker-settings-on-a-pipeline), the [ResourceSettings like accelerators](../tutorial/distributed-training.md) differently for the different environments. ### Trigger a pipeline on a Pull Request (Merge Request) -One way to ensure only fully working code makes it into production, you should use a staging environment to test all -the changes made to your code base and verify they work as intended. To do so automatically you should set up a +One way to ensure only fully working code makes it into production, you should use a staging environment to test all the changes made to your code base and verify they work as intended. To do so automatically you should set up a github action workflow that runs your pipeline for you when you make changes to it. -[Here](https://github.com/zenml-io/zenml-gitflow/blob/main/.github/workflows/pipeline_run.yaml) is an example that you -can use. +[Here](https://github.com/zenml-io/zenml-gitflow/blob/main/.github/workflows/pipeline_run.yaml) is an example that you can use. To only run the Github Action on a PR, you can configure the yaml like this @@ -102,8 +89,7 @@ jobs: ZENML_GITHUB_URL_PR: ${{ github.event.pull_request._links.html.href }} ``` -After configuring these values so they apply to your specific situation the rest of the template should work as is for -you. Specifically you will need to install all requirements, connect to your ZenML Server, set an active stack +After configuring these values so they apply to your specific situation the rest of the template should work as is for you. Specifically you will need to install all requirements, connect to your ZenML Server, set an active stack and run a pipeline within your github action. ```yaml diff --git a/docs/book/user-guide/production-guide/cloud-orchestration.md b/docs/book/user-guide/production-guide/cloud-orchestration.md index 395f1c5fe9c..42b7ad09594 100644 --- a/docs/book/user-guide/production-guide/cloud-orchestration.md +++ b/docs/book/user-guide/production-guide/cloud-orchestration.md @@ -16,10 +16,8 @@ These, along with [remote storage](remote-storage.md), complete a basic cloud st {% hint style="info" %} Would you like to skip ahead and deploy a full ZenML cloud stack already? -Check out the -[in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), -the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), -or [the ZenML Terraform modules](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) +Check out the [in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), +the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), or [the ZenML Terraform modules](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register a cloud stack. {% endhint %} diff --git a/docs/book/user-guide/production-guide/configure-pipeline.md b/docs/book/user-guide/production-guide/configure-pipeline.md index 727db535946..de72d108ea8 100644 --- a/docs/book/user-guide/production-guide/configure-pipeline.md +++ b/docs/book/user-guide/production-guide/configure-pipeline.md @@ -77,11 +77,9 @@ zenml model version list breast_cancer_classifier ``` {% endtab %} -{% tab title="Dashboard" %} -[ZenML Pro](https://www.zenml.io/pro) ships with a Model Control Plane dashboard where you can visualize all the versions: +{% tab title="Dashboard" %} [ZenML Pro](https://www.zenml.io/pro) ships with a Model Control Plane dashboard where you can visualize all the versions: -

All model versions listed

-{% endtab %} +

All model versions listed

{% endtab %} {% endtabs %} ### Passing parameters @@ -157,8 +155,7 @@ steps: {% hint style="info" %} -Read more about settings in ZenML [here](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) and -[here](https://docs.zenml.io/user-guides/tutorial/distributed-training) +Read more about settings in ZenML [here](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) and [here](https://docs.zenml.io/user-guides/tutorial/distributed-training) {% endhint %} Now let's run the pipeline again: diff --git a/docs/book/user-guide/production-guide/deploying-zenml.md b/docs/book/user-guide/production-guide/deploying-zenml.md index 380c828ac52..5da79922876 100644 --- a/docs/book/user-guide/production-guide/deploying-zenml.md +++ b/docs/book/user-guide/production-guide/deploying-zenml.md @@ -53,18 +53,15 @@ Having trouble connecting with a browser? There are other ways to connect. Read This command will start a series of steps to validate the device from where you are connecting that will happen in your browser. After that, you're now locally connected to a remote ZenML. Nothing of your experience changes, except that all metadata that you produce will be tracked centrally in one place from now on. {% hint style="info" %} -You can always go back to the local zenml experience by using `zenml logout` -{% endhint %} +You can always go back to the local zenml experience by using `zenml logout` {% endhint %} ## Further resources To learn more about deploying ZenML, check out the following resources: -- [Deploying ZenML](https://docs.zenml.io/getting-started/deploying-zenml): an overview of - the different options for deploying ZenML and the system architecture of a +- [Deploying ZenML](https://docs.zenml.io/getting-started/deploying-zenml): an overview of the different options for deploying ZenML and the system architecture of a deployed ZenML instance. -- [Full how-to guides](https://docs.zenml.io/getting-started/deploying-zenml): guides on how to - deploy ZenML on Docker or Hugging Face Spaces or Kubernetes or some other cloud +- [Full how-to guides](https://docs.zenml.io/getting-started/deploying-zenml): guides on how to deploy ZenML on Docker or Hugging Face Spaces or Kubernetes or some other cloud provider.
ZenML Scarf
diff --git a/docs/book/user-guide/production-guide/remote-storage.md b/docs/book/user-guide/production-guide/remote-storage.md index 4268ca02818..12be472d6fb 100644 --- a/docs/book/user-guide/production-guide/remote-storage.md +++ b/docs/book/user-guide/production-guide/remote-storage.md @@ -15,10 +15,8 @@ When using a stack with remote storage, nothing changes except the fact that the {% hint style="info" %} Would you like to skip ahead and deploy a full ZenML cloud stack already? -Check out the -[in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), -the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), -or [the ZenML Terraform modules](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) +Check out the [in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), +the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), or [the ZenML Terraform modules](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register a cloud stack. {% endhint %} @@ -117,14 +115,10 @@ Having trouble with setting up infrastructure? Join the [ZenML community](https: While you can go ahead and [run your pipeline on your stack](remote-storage.md#running-a-pipeline-on-a-cloud-stack) if your local client is configured to access it, it is best practice to use a [service connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management) for this purpose. Service connectors are quite a complicated concept (We have a whole [docs section](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management) on them) - but we're going to be starting with a very basic approach. -First, let's understand what a service connector does. In simple words, a -service connector contains credentials that grant stack components access to -cloud infrastructure. These credentials are stored in the form of a -[secret](https://docs.zenml.io/how-to/project-setup-and-management/interact-with-secrets), -and are available to the ZenML server to use. Using these credentials, the -service connector brokers a short-lived token and grants temporary permissions -to the stack component to access that infrastructure. This diagram represents -this process: +First, let's understand what a service connector does. In simple words, a service connector contains credentials that grant stack components access to +cloud infrastructure. These credentials are stored in the form of a [secret](https://docs.zenml.io/how-to/project-setup-and-management/interact-with-secrets), +and are available to the ZenML server to use. Using these credentials, the service connector brokers a short-lived token and grants temporary permissions +to the stack component to access that infrastructure. This diagram represents this process:

Service Connectors abstract away complexity and implement security best practices

@@ -204,11 +198,9 @@ zenml artifact version list --created="gte:$(date -v-15M '+%Y-%m-%d %H:%M:%S')" ``` {% endtab %} -{% tab title="Cloud Dashboard" %} -[ZenML Pro](https://zenml.io/pro) features an [Artifact Control Plane](../starter-guide/manage-artifacts.md) to visualize artifact versions: +{% tab title="Cloud Dashboard" %} [ZenML Pro](https://zenml.io/pro) features an [Artifact Control Plane](../starter-guide/manage-artifacts.md) to visualize artifact versions: -

See artifact versions in the cloud.

-{% endtab %} +

See artifact versions in the cloud.

{% endtab %} {% endtabs %} You will notice above that some artifacts are stored locally, while others are stored in a remote storage location. diff --git a/docs/book/user-guide/production-guide/understand-stacks.md b/docs/book/user-guide/production-guide/understand-stacks.md index caf766f14ad..c0fa277c457 100644 --- a/docs/book/user-guide/production-guide/understand-stacks.md +++ b/docs/book/user-guide/production-guide/understand-stacks.md @@ -172,11 +172,9 @@ Stack 'a_new_local_stack' with id '...' is owned by user default and is 'private ![GIF of our VS code extension, showing some of the uses of the sidebar](../../.gitbook/assets/zenml-extension-shortened.gif) -If you are using [our VS Code -extension](https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode), +If you are using [our VS Code extension](https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode), you can easily view and switch your stacks by opening the sidebar (click on the -ZenML icon). You can then click on the stack you want to switch to as well as -view the stack components it's made up of. +ZenML icon). You can then click on the stack you want to switch to as well as view the stack components it's made up of. ### Run a pipeline on the new local stack diff --git a/docs/book/user-guide/starter-guide/cache-previous-executions.md b/docs/book/user-guide/starter-guide/cache-previous-executions.md index cea3443c5df..0f941178398 100644 --- a/docs/book/user-guide/starter-guide/cache-previous-executions.md +++ b/docs/book/user-guide/starter-guide/cache-previous-executions.md @@ -120,8 +120,7 @@ When any of these factors change, the cache key changes, and the step will be re ### Configuring cache policies You can configure cache policies at both the step and pipeline level using the `CachePolicy` class. -Similar to enabling and disabling the cache above, you can define this cache policy -on both pipeline and step either via the decorator or the `with_options(...)` method. Configuring +Similar to enabling and disabling the cache above, you can define this cache policy on both pipeline and step either via the decorator or the `with_options(...)` method. Configuring a cache policy for a pipeline will configure it for all its steps. @@ -189,8 +188,7 @@ cache_policy = CachePolicy(source_dependencies=["run.my_helper_function"]) #### Cache expiration -By default, any step that executes successfully is a caching candidate for future step runs. Any step with the same [cache key](#understanding-cache-keys) running afterwards -can reuse the output artifacts produced by the caching candidate instead of actually executing the step code. In some cases however you might want to limit for how long a step run is a valid cache candidate for future steps. You can do that by configuring an expiration time for your step runs: +By default, any step that executes successfully is a caching candidate for future step runs. Any step with the same [cache key](#understanding-cache-keys) running afterwards can reuse the output artifacts produced by the caching candidate instead of actually executing the step code. In some cases however you might want to limit for how long a step run is a valid cache candidate for future steps. You can do that by configuring an expiration time for your step runs: ```python from zenml.config import CachePolicy diff --git a/docs/book/user-guide/starter-guide/create-an-ml-pipeline.md b/docs/book/user-guide/starter-guide/create-an-ml-pipeline.md index c2a159d0487..abf49aecd10 100644 --- a/docs/book/user-guide/starter-guide/create-an-ml-pipeline.md +++ b/docs/book/user-guide/starter-guide/create-an-ml-pipeline.md @@ -18,8 +18,7 @@ pip install "zenml[server]" zenml login --local # Will launch the dashboard locally ``` -It is also highly recommended that you run [`zenml init`](https://docs.zenml.io/how-to/project-setup-and-management/setting-up-a-project-repository/set-up-repository#zen) at your project root directory when starting a new project. This will tell ZenML which files to include when -running your pipelines remotely. +It is also highly recommended that you run [`zenml init`](https://docs.zenml.io/how-to/project-setup-and-management/setting-up-a-project-repository/set-up-repository#zen) at your project root directory when starting a new project. This will tell ZenML which files to include when running your pipelines remotely. {% endhint %} ## Start with a simple ML pipeline diff --git a/docs/book/user-guide/starter-guide/manage-artifacts.md b/docs/book/user-guide/starter-guide/manage-artifacts.md index 1d273c593e8..c28f76c907b 100644 --- a/docs/book/user-guide/starter-guide/manage-artifacts.md +++ b/docs/book/user-guide/starter-guide/manage-artifacts.md @@ -50,8 +50,7 @@ Artifacts named `iris_dataset` can then be found swiftly using various ZenML int {% tabs %} {% tab title="OSS (CLI)" %} -To list artifacts: `zenml artifact list` -{% endtab %} +To list artifacts: `zenml artifact list` {% endtab %} {% tab title="Cloud (Dashboard)" %} The [ZenML Pro](https://zenml.io/pro) dashboard offers advanced visualization features for artifact exploration. @@ -99,14 +98,12 @@ After execution, `iris_dataset` and its version `raw_2023` can be seen using: {% tabs %} {% tab title="OSS (CLI)" %} -To list versions: `zenml artifact version list` -{% endtab %} +To list versions: `zenml artifact version list` {% endtab %} {% tab title="Cloud (Dashboard)" %} The Cloud dashboard visualizes version history for your review. -

ZenML Data Versions List.

-{% endtab %} +

ZenML Data Versions List.

{% endtab %} {% endtabs %} ### Add metadata and tags @@ -198,8 +195,7 @@ This feature is currently in Alpha Preview. We encourage you to share feedback a Assigning a type to an artifact allows ZenML to highlight them differently in the dashboard and also lets you filter your artifacts better. {% hint style="info" %} -If you don't specify a type for your artifact, ZenML will use the default artifact type provided by the materializer that is used to -save the artifact. +If you don't specify a type for your artifact, ZenML will use the default artifact type provided by the materializer that is used to save the artifact. {% endhint %} @@ -338,8 +334,7 @@ from zenml import load_artifact load_artifact("iris_predictions") ``` -{% hint style="info" %} -`load_artifact` is simply short-hand for the following Client call: +{% hint style="info" %} `load_artifact` is simply short-hand for the following Client call: ```python from zenml.client import Client @@ -425,14 +420,12 @@ artifact.run_metadata["shape"].value # Shape e.g. (500,20) {% tab title="OSS (Dashboard)" %} The information regarding the metadata of an artifact can be found within the DAG visualizer interface on the OSS dashboard: -

ZenML Artifact Control Plane.

-{% endtab %} +

ZenML Artifact Control Plane.

{% endtab %} {% tab title="Cloud (Dashboard)" %} The [ZenML Pro](https://zenml.io/pro) dashboard offers advanced visualization features for artifact exploration, including a dedicated artifacts tab with metadata visualization: -

ZenML Artifact Control Plane.

-{% endtab %} +

ZenML Artifact Control Plane.

{% endtab %} {% endtabs %} A user can also add metadata to an artifact directly within a step using the `log_metadata` method: diff --git a/docs/book/user-guide/starter-guide/track-ml-models.md b/docs/book/user-guide/starter-guide/track-ml-models.md index 45813bf1125..5306f1d5ad4 100644 --- a/docs/book/user-guide/starter-guide/track-ml-models.md +++ b/docs/book/user-guide/starter-guide/track-ml-models.md @@ -28,8 +28,7 @@ These models can be viewed within ZenML: {% tab title="Cloud (Dashboard)" %} The [ZenML Pro](https://zenml.io/pro) dashboard has additional capabilities, that include visualizing these models in the dashboard. -

ZenML Model Control Plane.

-{% endtab %} +

ZenML Model Control Plane.

{% endtab %} {% endtabs %} ## Configuring a model in a pipeline @@ -105,8 +104,7 @@ The following commands can be used to list the various artifacts associated with {% tab title="Cloud (Dashboard)" %} The [ZenML Pro](https://zenml.io/pro) dashboard has additional capabilities, that include visualizing all associated runs and artifacts for a model version: -
ZenML Model Versions List.

ZenML Model versions List.

-{% endtab %} +
ZenML Model Versions List.

ZenML Model versions List.

{% endtab %} {% endtabs %} ## Fetching the model in a pipeline @@ -201,8 +199,7 @@ model_version.run_metadata["accuracy"].value {% tab title="Cloud (Dashboard)" %} The [ZenML Pro](https://zenml.io/pro) dashboard offers advanced visualization features for artifact exploration, including a dedicated artifacts tab with metadata visualization: -

ZenML Artifact Control Plane.

-{% endtab %} +

ZenML Artifact Control Plane.

{% endtab %} {% endtabs %} Choosing [log metadata with artifacts](manage-artifacts.md#logging-metadata-for-an-artifact) or model versions depends on the scope and purpose of the information you wish to capture. Artifact metadata is best for details specific to individual outputs, while model version metadata is suitable for broader information relevant to the overall model. By utilizing ZenML's metadata logging capabilities and special types, you can enhance the traceability, reproducibility, and analysis of your ML workflows. diff --git a/docs/book/user-guide/tutorial/datasets.md b/docs/book/user-guide/tutorial/datasets.md index 7c37d01c3da..2152e2d2127 100644 --- a/docs/book/user-guide/tutorial/datasets.md +++ b/docs/book/user-guide/tutorial/datasets.md @@ -180,8 +180,7 @@ def etl_pipeline(mode: str = "develop"): ## Best Practices for Designing Flexible and Maintainable Pipelines -When working with custom Dataset classes in ZenML pipelines, it's crucial to design your pipelines\ -to accommodate various data sources and processing requirements. +When working with custom Dataset classes in ZenML pipelines, it's crucial to design your pipelines\ to accommodate various data sources and processing requirements. Here are some best practices to ensure your pipelines remain flexible and maintainable: diff --git a/docs/book/user-guide/tutorial/hyper-parameter-tuning.md b/docs/book/user-guide/tutorial/hyper-parameter-tuning.md index 6ae4961aaa1..55d85028b2e 100644 --- a/docs/book/user-guide/tutorial/hyper-parameter-tuning.md +++ b/docs/book/user-guide/tutorial/hyper-parameter-tuning.md @@ -119,8 +119,7 @@ For a deeper exploration of how to query past pipeline runs, see the [Inspecting * Replace the simple grid‑search with a more sophisticated tuner (e.g. `sklearn.model_selection.GridSearchCV` or [Optuna](https://optuna.org/)). For a complete Optuna example that uses ZenML dynamic pipelines for parallel trial execution, see [`examples/optuna_hyperparameter_tuning`](https://github.com/zenml-io/zenml/tree/main/examples/optuna_hyperparameter_tuning). * Deploy the winning model as an HTTP service using [Pipeline Deployments](https://docs.zenml.io/concepts/deployment) (recommended) or via the legacy [Model Deployer](https://docs.zenml.io/stacks/stack-components/model-deployers). -* Move the pipeline to a [remote - orchestrator](https://docs.zenml.io/stacks/orchestrators) to scale out the +* Move the pipeline to a [remote orchestrator](https://docs.zenml.io/stacks/orchestrators) to scale out the search.
ZenML Scarf
diff --git a/docs/book/user-guide/tutorial/managing-scheduled-pipelines.md b/docs/book/user-guide/tutorial/managing-scheduled-pipelines.md index 66345720de9..517c31683ac 100644 --- a/docs/book/user-guide/tutorial/managing-scheduled-pipelines.md +++ b/docs/book/user-guide/tutorial/managing-scheduled-pipelines.md @@ -19,8 +19,7 @@ ZenML doesn't implement its own scheduler but acts as a wrapper around the sched 2. Registers the schedule with the orchestrator's scheduling system 3. Records the schedule in the ZenML metadata store -The orchestrator then takes over responsibility for executing the pipeline\ -according to the schedule. +The orchestrator then takes over responsibility for executing the pipeline\ according to the schedule. {% hint style="info" %} For our full reference documentation on schedules, see the [Schedule a Pipeline](https://docs.zenml.io/concepts/steps_and_pipelines/scheduling) page. @@ -80,8 +79,7 @@ scheduled_pipeline = daily_data_pipeline.with_options(schedule=schedule) scheduled_pipeline() ``` -Running the pipeline will create the schedule in the ZenML metadata store. as\ -well as the scheduled run in the orchestrator. +Running the pipeline will create the schedule in the ZenML metadata store. as\ well as the scheduled run in the orchestrator. {% hint style="info" %} **Best Practice: Use Descriptive Schedule Names** diff --git a/docs/book/user-guide/tutorial/organizing-pipelines-and-models.md b/docs/book/user-guide/tutorial/organizing-pipelines-and-models.md index 75ef7bb32b4..dd29b3a958d 100644 --- a/docs/book/user-guide/tutorial/organizing-pipelines-and-models.md +++ b/docs/book/user-guide/tutorial/organizing-pipelines-and-models.md @@ -14,8 +14,7 @@ This cookbook demonstrates how to effectively organize your machine learning ass As ML projects grow, effective organization becomes critical. ZenML provides two powerful organization mechanisms: 1. **Tags**: Flexible labels that can be applied to various entities (pipelines, runs, artifacts, models) -2. **Projects** (ZenML Pro): Namespace-based isolation for logical separation\ - between initiatives or teams +2. **Projects** (ZenML Pro): Namespace-based isolation for logical separation\ between initiatives or teams {% hint style="info" %} For our full reference documentation on things covered in this tutorial, see the [Tagging](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/tagging) page, the [Projects](https://docs.zenml.io/pro/core-concepts/projects) page, and the [Model Control Plane](https://docs.zenml.io/how-to/model-management-metrics/model-control-plane) page. diff --git a/docs/book/user-guide/tutorial/replaying-runs-steps.md b/docs/book/user-guide/tutorial/replaying-runs-steps.md index c145d7aa4b0..6dd2edb5330 100644 --- a/docs/book/user-guide/tutorial/replaying-runs-steps.md +++ b/docs/book/user-guide/tutorial/replaying-runs-steps.md @@ -15,8 +15,7 @@ A replay re-executes a pipeline or step using the same input artifacts (and para ## Replaying a pipeline -Replay a pipeline either from local pipeline code (`.replay()`) or directly via -the server (`Client().replay_pipeline_run(...)`). +Replay a pipeline either from local pipeline code (`.replay()`) or directly via the server (`Client().replay_pipeline_run(...)`). {% tabs %} {% tab title="From local code" %} @@ -62,8 +61,7 @@ Client().replay_pipeline_run(name_id_or_prefix="run_name_or_id") ### Skipping steps -You can skip steps that don't need to be re-executed. Skipped steps reuse -their output artifacts from the original run. +You can skip steps that don't need to be re-executed. Skipped steps reuse their output artifacts from the original run. {% tabs %} {% tab title="From local code" %} @@ -103,8 +101,7 @@ A step can only be skipped if all of its upstream dependencies are also skipped. {% tabs %} {% tab title="From local code" %} -Pass `input_overrides` to change parameters for the replayed run. Any -parameters you don't override are carried over from the original run. +Pass `input_overrides` to change parameters for the replayed run. Any parameters you don't override are carried over from the original run. ```python training_pipeline.replay(input_overrides={"learning_rate": 0.01}) @@ -126,13 +123,11 @@ Client().replay_pipeline_run( ### Overriding step inputs -Use `step_input_overrides` to replace specific step inputs for a replayed -pipeline run. +Use `step_input_overrides` to replace specific step inputs for a replayed pipeline run. {% tabs %} {% tab title="From local code" %} -`step_input_overrides` expects a mapping of `invocation_id -> input_name -> -value`. +`step_input_overrides` expects a mapping of `invocation_id -> input_name -> value`. ```python training_pipeline.replay( @@ -149,8 +144,7 @@ training_pipeline.replay( For server-side replay, `step_input_overrides` values can be either: - **UUIDs** of existing artifact versions (no new upload), or -- **inline values** (server uploads them to the active artifact store before - replay starts). +- **inline values** (server uploads them to the active artifact store before replay starts). ```python from zenml import Client @@ -175,16 +169,12 @@ Client().replay_pipeline_run( #### Overriding an input for every invocation of a step -If you want to override inputs for **every invocation** of a step, you can -use `step_default_input_overrides`. It is keyed by the step's name instead of -the invocation ID. Per-invocation overrides in `step_input_overrides` take -precedence per input key. +If you want to override inputs for **every invocation** of a step, you can use `step_default_input_overrides`. It is keyed by the step's name instead of +the invocation ID. Per-invocation overrides in `step_input_overrides` take precedence per input key. {% hint style="warning" %} -The override applies to every invocation that shares the given name, and ZenML -does not enforce that a name maps to a single step function. If two different -steps share a name (for example two functions named `train` in different -modules both default to the name `train`), the override is applied to both. +The override applies to every invocation that shares the given name, and ZenML does not enforce that a name maps to a single step function. If two different +steps share a name (for example two functions named `train` in different modules both default to the name `train`), the override is applied to both. Give such steps distinct names if you want to target only one of them. {% endhint %} @@ -223,8 +213,7 @@ Client().replay_pipeline_run( ## Replaying a single step -Call `.replay()` on any `@step`-decorated function. ZenML loads the -original input artifacts, feeds them to your (potentially updated) step code, +Call `.replay()` on any `@step`-decorated function. ZenML loads the original input artifacts, feeds them to your (potentially updated) step code, and runs it as a single-step pipeline on the active stack. The step to replay is resolved using the first match: @@ -259,8 +248,7 @@ train.replay( ### Using `invocation_id` -If a step appears more than once in a pipeline, specify which invocation to -replay: +If a step appears more than once in a pipeline, specify which invocation to replay: ```python train.replay(pipeline="training_pipeline", invocation_id="train_2") @@ -270,10 +258,8 @@ train.replay(pipeline="training_pipeline", invocation_id="train_2") ## Debug mode -Both pipeline and step replays accept `debug=True`. This runs the replay -on a **local orchestrator** while keeping the rest of your active stack -(artifact store, etc.), so you can iterate quickly without waiting for remote -infrastructure. +Both pipeline and step replays accept `debug=True`. This runs the replay on a **local orchestrator** while keeping the rest of your active stack +(artifact store, etc.), so you can iterate quickly without waiting for remote infrastructure. ```python training_pipeline.replay(debug=True) diff --git a/docs/book/user-guide/tutorial/trigger-pipelines-from-external-systems.md b/docs/book/user-guide/tutorial/trigger-pipelines-from-external-systems.md index 7fb201b2502..fb797f974a8 100644 --- a/docs/book/user-guide/tutorial/trigger-pipelines-from-external-systems.md +++ b/docs/book/user-guide/tutorial/trigger-pipelines-from-external-systems.md @@ -227,8 +227,7 @@ if snapshots: #### Using the REST API: -For this you'll need a URL for a ZenML server. For those with a ZenML Pro -account, you can find the URL in the dashboard in the following location: +For this you'll need a URL for a ZenML server. For those with a ZenML Pro account, you can find the URL in the dashboard in the following location: ![Where to find the ZenML server URL](../.gitbook/assets/zenml-pro-server-url.png) @@ -321,8 +320,7 @@ print(f"Store this token securely: {token.token}") - **Secure**: Can be granted minimal permissions needed for the task - **Traceable**: Clear audit trail of which system performed actions -Use this token in your API calls, and store it securely in your external system -(e.g., as a GitHub Secret, AWS Secret, or environment variable). Read more +Use this token in your API calls, and store it securely in your external system (e.g., as a GitHub Secret, AWS Secret, or environment variable). Read more about [service accounts and tokens](https://docs.zenml.io/api-reference/oss-api/getting-started#using-a-service-account-and-an-api-key). ## Method 2: Pipeline Deployments (Open Source) @@ -533,8 +531,7 @@ For comprehensive documentation on deployment configuration, lifecycle managemen ## Method 3: Building a Custom Trigger API (Open Source) If you need full control over the triggering mechanism beyond what Pipeline -Deployments offer — for example, custom routing logic, integration with a -specific framework, or embedding pipeline triggers within an existing +Deployments offer — for example, custom routing logic, integration with a specific framework, or embedding pipeline triggers within an existing application — you can create your own API wrapper around pipeline execution. The custom trigger API solution consists of the following components: @@ -953,8 +950,7 @@ When deploying this API in production: ### Tag Snapshots -You should tag your snapshots to make them easier to find and manage. It is -currently only possible using the Python SDK: +You should tag your snapshots to make them easier to find and manage. It is currently only possible using the Python SDK: ```python from zenml import add_tags