diff --git a/site/docs/concepts/README.md b/site/docs/concepts/README.md index cd605d45d5d..1c3189568af 100644 --- a/site/docs/concepts/README.md +++ b/site/docs/concepts/README.md @@ -24,7 +24,7 @@ Estuary's web application is where you'll most likely create, monitor, and updat You can find it at [dashboard.estuary.dev](https://dashboard.estuary.dev). The app is backed by secure, cloud-hosted infrastructure that Estuary manages. -[Take a tour of the web app](web-app.md). +[Learn more about the dashboard](web-app.md). [Create your first Data Flow with the web app](../guides/create-dataflow.md). diff --git a/site/docs/concepts/web-app.md b/site/docs/concepts/web-app.md index 54bd63110dc..f989e43385c 100644 --- a/site/docs/concepts/web-app.md +++ b/site/docs/concepts/web-app.md @@ -4,46 +4,37 @@ sidebar_position: 12 # Web Application -Estuary's web application is at [dashboard.estuary.dev](https://dashboard.estuary.dev). +Estuary's web application, or dashboard, is located at [dashboard.estuary.dev](https://dashboard.estuary.dev). -The web app is the central, low-code environment for creating, managing, and monitoring Data Flows. +The dashboard is the central, low-code environment for creating, managing, and monitoring data flows. ## When to use the web app -The web app and [flowctl](./flowctl.md) are designed to work together as a complete platform. -You can use either, or both, to work on your Data Flows, depending on your preference. +The web app and the [`flowctl` CLI](./flowctl.md) are designed to work together as a complete platform. +You can use either, or both, to work on your data flows, depending on your preference. -With Estuary's web app, you can perform most common workflows, including: +In Estuary's dashboard, you can perform most common workflows, including: -* Creating end-to-end Data Flows: **capturing** data from source systems and **materializing** it to destinations. +* Creating end-to-end data flows: **capturing** data from source systems and **materializing** it to destinations. * Creating, viewing, and editing individual captures and materializations. -* Monitor the amount of data being processed by the system. +* Monitoring the amount of data being processed by the system. * Viewing data **collections**. * Viewing users and permissions. * Granting permissions to other users. -* Authenticating with the flowctl CLI. +* Authenticating with the `flowctl` CLI. * Managing billing details. -Some advanced workflows, like transforming data with **derivations**, aren't fully available in the web app. +Some advanced workflows, like transforming data with **derivations**, require using `flowctl`. -Even if you prefer the command line or plan to perform a task that's only available through flowctl, we recommend you begin your work in the web app; -it provides a quicker and easier path to create captures and materializations. You can then switch to flowctl to continue working. - -## Signing in - -You use a Google, Microsoft, or GitHub account to sign into Estuary. Alternatively, [contact us](https://estuary.dev/contact-us) about [Single Sign-On (SSO) options](/getting-started/sso-setup). - -![](<./webapp-images/login-screen.png>) - -If you've never used Estuary before, you'll be prompted to register before being issued a trial account. If you want to use Estuary for production workflows or collaborate with team members, you'll need an organizational account. -[Contact Estuary](mailto:support@estuary.dev) to create a new organizational account or join an existing organization. +Even if you prefer the command line or plan to perform a task that's only available through `flowctl`, we recommend you begin your work in the dashboard; +it provides a quicker and easier path to create captures and materializations. You can then switch to `flowctl` to continue working. ## Navigating the web app -When you log into the web app, you land on the **Welcome** page. +When you log into the dashboard, you land on the **Welcome** page. There are four additional pages visible as tabs in the side navigation: **Sources (captures)**, **Collections**, **Destinations (materializations)**, and **Admin**. -The order of the tabs mirrors the order of a basic Data Flow: +The order of the tabs mirrors the order of a basic data flow: import Mermaid from '@theme/Mermaid'; @@ -54,336 +45,73 @@ import Mermaid from '@theme/Mermaid'; `}/> While you may choose to [use the tabs in this sequence](../guides/create-dataflow.md), it's not necessary. -All Estuary entities exist individually, outside of the context of a complete Data Flow. -You can use the different pages in the web app to monitor and manage your items in a number of other ways, as described below. - -## Captures page - -The **Captures** page shows you a table of existing [captures](./captures.md) to which you have [access](/reference/authentication). -The **New Capture** button is also visible. -You can use the table to monitor your captures. - -![](<./webapp-images/capture-page.png>) - -**1:** Select all or deselect all. - -**2:** Enable, Disable, and Delete buttons. These actions will be applied to the selected table rows. Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the capture(s). - -**3:** Materialize button. When you click this button, you're directed to the **Create Materializations** page. -All the collections of the selected capture(s) will be added to the materialization. - -**4:** Filter captures. Type a catalog prefix, unique capture name, or connector name to return captures that match your query. - -Capture names follow the pattern `prefix/unique-identifier/connector-name`, with `prefix` supporting multiple layers of nesting. -You can search for any part of this full capture name. You can also use the `*` wildcard. -For example, if you have a capture called `acmeCo/logistics/anvil-locations/source-postgres`, -you can find it by filtering for `acmeCo*source-postgres`. - -**5:** Status indicator. Shows the status of the primary task [shard](./advanced/shards.md) that backs this capture. - -* **Primary (Green)**: Data is actively flowing through the capture. -* **Pending (Yellow)**: The capture is attempting to re-connect. Often, you'll see this after you re-enable the capture as Estuary backfills historical data. -* **Failed (Red)**: The capture has failed with an unrecoverable error. -* **Disabled (Hollow circle)**: The capture is disabled. -* **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error. - -**6:** Capture name. The full name is shown, including all [prefixes](./catalogs.md#namespace). It is also a link to the [details page](#capture-details-page) of the capture. - -**7:** Capture type. The icon shows the type of source system data is captured from. - -**8:** Capture [statistics](./advanced/logs-stats.md#statistics). The **Data Written** column shows the total amount of data, in bytes and in [documents](./collections.md#documents), -that the capture has written to its associated collections within a configurable time interval. -Click the time interval in the header to select from **Today**, **Yesterday**, **This Week**, **Last Week**, **This Month**, **Last Month**, or **All Time**. - -**9:** Associated collections. The **Writes to** column shows all the collections to which the capture writes data. For captures with a large number of collections, there is a chip stating how many collections are hidden. Clicking on this will allow you to hover over this column and scroll to view the full list. These also link to the details page of the collection. - -**10:** Publish time. Hover over this value to see the exact time the capture was last published. - -**11:** Options. Click to open the menu to **Edit Specification**. - -### Editing captures and collections - -When you click **Edit specification** for a capture, you're taken to the **Edit Capture** page. - -This page is similar to the [**Create Capture**](#creating-a-capture) page as it was filled out just before the capture was published. - -For detailed steps to edit a capture, see the [guide](/guides/edit-data-flows/#edit-a-capture). - -### Creating a capture - -When you click **Create Capture**, you're taken to the Create Capture page. -In the first view, all available capture connectors are displayed. - -Select the tile of the system from which you want to capture data to show the full capture form. -The form details are specific to the connector you chose. - -For detailed steps to create a capture, see the [guide](../guides/create-dataflow.md#create-a-capture). - -After you successfully publish a capture, you're given the option to materialize the collections you just captured. -You can proceed to the materialization, or opt to exit to a different page of the web app. - -## Collections page - -The **Collections** page shows a table of [collections](./collections.md) to which you have access. There is also a button to begin a new derivation, or transformation. - -The table has nearly all of the same features as the **Captures** table, with several important distinctions -that are called out in the image below. - -You can use the table to view each collection's specification and see a sample of its data. -This can help you verify that collection data was captured as expected and that you'll be able to materialize it how you want, and troubleshoot if necessary. - -![](<./webapp-images/collections-page.png>) - -**1:** Select all or deselect all. - -**2:** Enable, Disable, and Delete buttons. These actions will be applied to the selected table rows. Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the collection(s). - -**3:** Filter collections by name. Type a catalog prefix, unique collection name to return collections that match your query. - -**4:** Status indicator. If the collection does not contain a [derivation](./README.md#derivations), the indicator should always show green, and hover text will say "Collection." -In the event that the server cannot be reached, the indicator will show "Unknown" status (black in light mode and white in dark mode). - - If the collection contains a derivation, the status of the derivation's primary task [shard](./advanced/shards.md) will be indicated: - - * **Primary (Green)**: Data is actively flowing through the derivation. - * **Pending (Yellow)**: The derivation is attempting to re-connect. - * **Failed (Red)**: The derivation has failed with an unrecoverable error. - * **Disabled (Hollow circle)**: The derivation is disabled. - * **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error. - -**5:** Collection name. The full name is shown, including all [prefixes](./catalogs.md#namespace). It is also a link to the [details page](#collection-details-page) of the collection. - -**6:** Collection [statistics](./advanced/logs-stats.md#statistics). The **Data Written** column shows the total amount of data, in bytes and in [documents](./collections.md#documents), -that has been written to each collection from its associated capture or derivation within a configurable time interval. -Click the time interval in the header to select from **Today**, **Yesterday**, **This Week**, **Last Week**, **This Month**, **Last Month**, or **All Time**. - -**7:** Publish time. Hover over this value to see the exact time the collection was last published. - -## Materializations page - -The **Materializations** page shows you a table of existing [materializations](/concepts/materialization) to which you have [access](/reference/authentication). -The **New Materialization** button is also visible. - -You can use the table to monitor your materializations. It's nearly identical to the table on the [Captures page](#captures-page), with a few exceptions. - -![](<./webapp-images/materialization-page.png>) - -**1:** Select all or deselect all. - -**2:** Enable, Disable, and Delete buttons. These actions will be applied to the selected table rows. Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the materialization(s). - -**3:** Filter materializations by name. Type a catalog prefix, unique materialization name, or connector name to return materializations that match your query. - -Materialization names follow the pattern `prefix/unique-identifier/connector-name`, with `prefix` supporting multiple layers of nesting. -You can search for any part of this full materialization name. You can also use the `*` wildcard. -For example, if you have a materialization called `acmeCo/anvil-locations/materialize-mysql`, -you can find it by filtering for `acmeCo*mysql`. - -**4:** Status indicator. Shows the status of the primary task [shard](./advanced/shards.md) that backs this materialization. - -* **Primary (Green)**: Data is actively flowing through the materialization. -* **Pending (Yellow)**: The materialization is attempting to re-connect. Often, you'll see this after you re-enable the materialization as Estuary backfills historical data. -* **Failed (Red)**: The materialization has failed with an unrecoverable error. -* **Disabled (Hollow circle)**: The materialization is disabled. -* **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error. - -**5:** Materialization name. The full name is shown, including all [prefixes](./catalogs.md#namespace). It is also a link to the [details page](#materialization-details-page) of the materialization. - -**6:** Materialization type. The icon shows the type of destination system data is materialized to. - -**7:** Materialization [statistics](./advanced/logs-stats.md#statistics). The **Data Read** column shows the total amount of data, in bytes and in [documents](./collections.md#documents), -that the materialization has read from its associated collections within a configurable time interval. -Click the time interval in the header to select from **Today**, **Yesterday**, **This Week**, **Last Week**, **This Month**, **Last Month**, or **All Time**. - -**8:** Associated collections. The **Reads from** column shows all the collections from which the materialization reads data. For materializations with a large number of collections, there is a chip stating how many collections are hidden. Clicking on this will allow you to hover over this column and scroll to view the full list. These also link to the details page of the collection. - -**9:** Publish time. Hover over this value to see the exact time the materialization was last published. - -**10:** Options. Click to open the menu to **Edit Specification**. - -### Editing materializations - -When you click **Edit specification** for a materialization, you're taken to the **Edit Materialization** page. +All Estuary entities exist individually, outside of the context of a complete data flow. +You can use the different pages in the dashboard to monitor and manage your items in a number of other ways. -This page is similar to the [**Create Materialization**](#creating-a-materialization) page as it was filled out just before the materialization was published. - -For detailed steps to edit a materialization, see the [guide](/guides/edit-data-flows/#edit-a-materialization). - -### Creating a materialization - -There are three ways to begin creating a materialization: - -* Clicking **New Materialization** on the Materializations page. -* Selecting one or more captures from the Captures page and clicking **Materialize**. -* Clicking **Materialize Collections** immediately after publishing a capture. - -When you initiate the workflow in any of these ways, all available materialization connectors are displayed. -Select a connector to reveal the full form with configuration options specific to your desired destination. - -Fill out the **Endpoint Config** form and in the **Source Collections** section use the **Collection Selector** to map collections to resources in the destination system. - -Note that if you entered the workflow from the Captures page or after publishing a capture, collections will be pre-populated for you. - -For detailed steps to create a materialization, see the [guide](../guides/create-dataflow.md#create-a-materialization). - -## Capture Details page - -When you click on the **name** of a capture on the [captures page](#captures-page) you will be taken to the capture details page to view data stats, sharding information, and general details. - - -### Overview Tab - -![](<./webapp-images/capture-details-page--overview.png>) - -**1:** The full name of the capture. - -**2:** Capture [statistics](./advanced/logs-stats.md#statistics). The **Usage** section displays the total amount of data, in bytes and in [documents](./collections.md#documents) written by the capture, per hour. The number of hours being displayed in the chart can be changed by clicking the time interval in the header to select from **6 hours**, **12 hours**, **24 hours**, **48 hours**, or **30 days**. - -**3:** The **Details** section shows information about the capture: when it was last updated, when it was created, the connector being used, and the collections to which the capture writes data. - -**4:** Detailed tooltip. You can hover over a section in the graph to see the specific data of that hour. - -**5:** The most recent hour. This will automatically update every 15 seconds with the most recent data and docs. - -**6:** Associated collections. Shows all the collections to which the capture writes data and when clicked will take you to the collection's [detail page](#collection-details-page). - -**7:** The **Shard Information** section shows the full identifier of the shard(s) that back your capture. If there's an error, you'll see an alert identifying the failing shard(s). Use the drop-down to open an expanded view of the failed shard's logs. - -### Spec Tab -![](<./webapp-images/capture-details-page--spec.png>) - -In the **Spec** tab, you can view the specification of the capture itself. - -## Collection Details page - -When you click on the **name** of a collection on the [collections page](#collections-page) you will be taken to the collection details page to view data stats, sharding information, preview of data, and general details. - -### Overview Tab - -![](<./webapp-images/collection-details-page--overview.png>) - -**1:** The full name of the collection. - -**2:** Collection [statistics](./advanced/logs-stats.md#statistics). The **Usage** section shows the total amount of data, in bytes and in [documents](./collections.md#documents) passing through a collection, per hour. The number of hours being displayed in the chart can be changed by clicking the time interval in the header to select from **6 hours**, **12 hours**, **24 hours**, **48 hours**, or **30 days**. - -**3:** The **Details** section shows information about the collection: when it was last updated, when it was created, and the associated collections (if any). - -**4:** Detailed tooltip. You can hover over a section in the graph to see the specific data of that hour. - -**5:** The most recent hour. This will automatically update every 15 seconds with the most recent data and docs. - -**6:** Associated collections. Shows source collections that this collection reads from. Click to go to the collection's [detail page](#collection-details-page). - -**7:** The **Shard Information** section (for derivations) shows the full identifier of the shard(s) that back your derivation. If there's an error, you'll see an alert identifying the failing shard(s). Use the drop-down to open an expanded view of the failed shard's logs. - -**8:** The **Data Preview** section shows a sample of collection [documents](./collections.md#documents): the individual JSON files that comprise the collection. -Documents are organized by their collection key value. Click a key from the list to view its document. - -### Spec Tab -![](<./webapp-images/collection-details-page--spec.png>) - -**1:** The collection's [key](./collections.md#keys) of the collection. - -**2:** The collection's [schema](./schemas.md) displayed in a read only table. The table columns can be sorted to more easily find what you need. - -:::tip -If you need to modify a collection, edit the [capture](./captures.md) or [derivation](./derivations.md) that provides its data. -::: - -## Materialization Details Page - -When you click on the **name** of a materialization on the [materializations page](#materializations-page) you will be taken to the detail page to view data stats, sharding information, and general details. - - -### Overview Tab - -![](<./webapp-images/materialization-details-page--overview.png>) - -**1:** The full name of the materialization. - -**2:** Materialization [statistics](./advanced/logs-stats.md#statistics). The **Usage** section shows the total amount of data, in bytes and in [documents](./collections.md#documents) read by a materialization, per hour. The number of hours being displayed in the chart can be changed by clicking the time interval in the header to select from **6 hours**, **12 hours**, **24 hours**, **48 hours**, or **30 days**. - -**3:** The **Details** section shows information about the materialization: when it was last updated, when it was created, and the associated collections. - -**4:** Detailed tooltip. You can hover over a section in the graph to see the specific data of that hour. - -**5:** The most recent hour. This will automatically update every 15 seconds with the most recent data and docs. - -**6:** Associated collections. Shows all the collections that provide data to this materialization. Click to go to the collection's [detail page](#collection-details-page). - -**7:** The **Shard Information** section shows the full identifier of the shard(s) that back your materialization. If there's an error, you'll see an alert identifying the failing shard(s). Use the drop-down to open an expanded view of the failed shard's logs. - -### Spec Tab -![](<./webapp-images/materialization-details-page--spec.png>) - -In the **Spec** tab, you can view the specification of the materialization itself. - -## Admin page - -On the **Admin** page, you can view users' access grants, your organization's cloud storage locations, and a complete list of connectors. -You can also get an access token to authenticate with flowctl and manage billing information. - -### Account Access - -The **Account Access** tab shows you all provisioned access grants on objects to which you also have access. -Both users and catalog prefixes can receive access grants. -These are split up into two tables called **Organization Membership** and **Data Sharing**. -Each access grant has its own row, so a given user or prefix may have multiple rows. - -For example, if you had read access to `foo/` and write access to `bar/`, you'd have a separate table row in the **Organization Membership** table for each of these capabilities. -If users Alice, Bob, and Carol each had write access on `foo/`, you'd see three more table rows representing these access grants. +--- -Taking this a step further, the prefix `foo/` could have read access to `buz/`. You'd see this in the **Data Sharing** table, -and it'd signify that everyone who has access to `foo/` also inherits read access to `buz/`. +### Capture pages -Use the search boxes to filter by username, prefix, or object. +The main **Sources** page provides a list of existing captures. From here, you can: -You can manage access by generating new user invitations, granting data sharing access, or selecting users or prefixes to revoke access. +* Create a new capture +* Manage existing captures, such as editing, enabling/disabling, or deleting them +* Select a capture to view its **Details** page -![](<./webapp-images/access-grant-invitation.png>) +Each capture has a **Details** page with additional information. The Details page allows you to: -Generating a new invitation will create a URL with a grant token parameter. This token will allow access based on the prefix, capability, and type you select. Copy the URL and share it with its intended recipient to invite them to your organization. +* Monitor usage +* View alerts and logs +* Inspect the capture's underlying specification and its history -[Learn more about capabilities and access.](/reference/authentication) +[See in-depth details on navigating these pages.](/guides/dashboard/captures) -### Settings +--- -The **Settings** tab includes additional configuration, such as organization notifications and storage mappings. +### Collection pages -#### Organization Notifications +The main **Collections** page provides a list of existing data collections. From here, you can: -Here, you are able to configure which email address(es) will receive [notifications](/reference/notifications) related to your organization or prefix. +* Create a draft for a new derived, or transformed, collection +* Manage existing collections, such as enabling/disabling or deleting them +* Select a collection to view its **Details** page -#### Cloud Storage +Each collection has a **Details** page with additional information. The Details page allows you to: -This section provides a table of the cloud storage locations that back your collections. -You're able to view the table if you're an admin. +* Monitor usage +* View alerts +* Inspect the collection's underlying specification and its history -Each top-level Estuary [prefix](./catalogs.md#namespace) is backed by one or more cloud storage bucket that you own. -You typically have just one prefix: your organization name, which you provided when configuring your Estuary organizational account. -If you're a trial user, your data is stored temporarily in Estuary's cloud storage bucket for your trial period. +[See in-depth details on navigating these pages.](/guides/dashboard/collections) -[Learn more about storage mappings.](./storage-mappings.md) +--- -#### Data Planes +### Materialization pages -The **Data Planes** section provides a table of all available data plane options, broken out by **Private** and **Public** data planes. +The main **Destinations** page provides a list of existing materializations. From here, you can: -You can find information here related to connecting and allowing access to your data plane of choice. See [Allowlisting IP Addresses](/reference/allow-ip-addresses) for more. +* Create a new materialization +* Manage existing materializations, such as editing, enabling/disabling, or deleting them +* Select a materialization to view its **Details** page -### Billing +Each materialization has a **Details** page with additional information. The Details page allows you to: -The **Billing** tab allows you to view and manage information related to past usage, the current billing cycle, and payment methods. +* Monitor usage +* View alerts and logs +* Inspect the materialization's underlying specification and its history -Your usage is broken down by the amount of data processed and number of task hours. View usage trends across previous months in the **Usage by Month** chart and preview your bill based on usage for the current month. If you are on the free tier (up to 2 connectors and 10 GB per month), you will still be able to preview your bill breakdown, and will have a "Free tier credit" deduction. To help estimate your bill, also see the [Pricing Calculator](https://estuary.dev/pricing/#pricing-calculator). +[See in-depth details on navigating these pages.](/guides/dashboard/materializations) -To pay your bill, add a payment method to your account. You can choose to pay via card or bank account. You will not be charged until you exceed the free tier's limits. +--- -### Connectors +### Admin pages -The **Connectors** tab offers a complete view of all connectors that are currently available through the web application, including both capture and materialization connectors. -If a connector you need is missing, you can [request it](https://github.com/estuary/connectors/issues/new/choose). +The **Admin** section in the dashboard provides information and settings related to your tenant. +Here, you can manage: -### CLI-API +* Organization membership and data sharing +* Data planes and storage mappings +* Billing information +* Refresh tokens and access tokens for `flowctl` -The **CLI-API** tab provides the access token required to [authenticate with flowctl](/reference/authentication/#authenticating-estuary-using-the-cli). You can also revoke old tokens. +[See in-depth details on navigating these pages.](/guides/dashboard/admin) diff --git a/site/docs/concepts/webapp-images/access-grant-invitation.png b/site/docs/concepts/webapp-images/access-grant-invitation.png deleted file mode 100644 index 81ede8bfc93..00000000000 Binary files a/site/docs/concepts/webapp-images/access-grant-invitation.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/capture-details-page--overview.png b/site/docs/concepts/webapp-images/capture-details-page--overview.png deleted file mode 100644 index 3d8ae66d2dc..00000000000 Binary files a/site/docs/concepts/webapp-images/capture-details-page--overview.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/capture-details-page--spec.png b/site/docs/concepts/webapp-images/capture-details-page--spec.png deleted file mode 100644 index 51ff114eb6f..00000000000 Binary files a/site/docs/concepts/webapp-images/capture-details-page--spec.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/capture-page.png b/site/docs/concepts/webapp-images/capture-page.png deleted file mode 100644 index e3ff7d61278..00000000000 Binary files a/site/docs/concepts/webapp-images/capture-page.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/collection-details-page--overview.png b/site/docs/concepts/webapp-images/collection-details-page--overview.png deleted file mode 100644 index 6b834b18da5..00000000000 Binary files a/site/docs/concepts/webapp-images/collection-details-page--overview.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/collection-details-page--spec.png b/site/docs/concepts/webapp-images/collection-details-page--spec.png deleted file mode 100644 index b7f4889c485..00000000000 Binary files a/site/docs/concepts/webapp-images/collection-details-page--spec.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/collections-page.png b/site/docs/concepts/webapp-images/collections-page.png deleted file mode 100644 index d6dfdd23e4f..00000000000 Binary files a/site/docs/concepts/webapp-images/collections-page.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/login-screen.png b/site/docs/concepts/webapp-images/login-screen.png deleted file mode 100644 index cf722738c98..00000000000 Binary files a/site/docs/concepts/webapp-images/login-screen.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/materialization-details-page--overview.png b/site/docs/concepts/webapp-images/materialization-details-page--overview.png deleted file mode 100644 index e00a20a281c..00000000000 Binary files a/site/docs/concepts/webapp-images/materialization-details-page--overview.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/materialization-details-page--spec.png b/site/docs/concepts/webapp-images/materialization-details-page--spec.png deleted file mode 100644 index df927c5b612..00000000000 Binary files a/site/docs/concepts/webapp-images/materialization-details-page--spec.png and /dev/null differ diff --git a/site/docs/concepts/webapp-images/materialization-page.png b/site/docs/concepts/webapp-images/materialization-page.png deleted file mode 100644 index 084a840e668..00000000000 Binary files a/site/docs/concepts/webapp-images/materialization-page.png and /dev/null differ diff --git a/site/docs/getting-started/sso-setup.md b/site/docs/getting-started/sso-setup.md index 3d3113f629d..e65c5aa4bb9 100644 --- a/site/docs/getting-started/sso-setup.md +++ b/site/docs/getting-started/sso-setup.md @@ -44,7 +44,7 @@ You may invite additional members of your company to collaborate with you. Estua 1. Meanwhile, the tenant admin can sign into their Estuary dashboard and navigate to the **Admin** tab. -2. Under [Account Access](../concepts/web-app.md#account-access), in the **Organization Membership** section, click the **Add Users** button. +2. Under [Account Access](/guides/dashboard/admin/#account-access), in the **Organization Membership** section, click the **Add Users** button. 3. Choose your desired prefix (such as your entire tenant or a defined subset), capability (admin or read access), and type (multi- or single-use). Then create an invite link. diff --git a/site/docs/getting-started/support.md b/site/docs/getting-started/support.md index 9ed1690754c..6209ce5cf1e 100644 --- a/site/docs/getting-started/support.md +++ b/site/docs/getting-started/support.md @@ -41,4 +41,4 @@ You should then see `estuary_support/` in the list of entities that you're shari When your issue is resolved, you can revoke access again by selecting the `estuary_support/` row and clicking **Remove**. -Learn more about [access grants](../concepts/web-app.md#account-access). +Learn more about [access grants](/guides/dashboard/admin/#account-access). diff --git a/site/docs/guides/dashboard/_category_.json b/site/docs/guides/dashboard/_category_.json index 58d353713ab..44b69033b49 100644 --- a/site/docs/guides/dashboard/_category_.json +++ b/site/docs/guides/dashboard/_category_.json @@ -1,4 +1,4 @@ { "label": "Estuary Dashboard", - "position": 4 + "position": 1 } \ No newline at end of file diff --git a/site/docs/guides/dashboard/admin.md b/site/docs/guides/dashboard/admin.md new file mode 100644 index 00000000000..e95a52a638e --- /dev/null +++ b/site/docs/guides/dashboard/admin.md @@ -0,0 +1,87 @@ +--- +sidebar_position: 4 +--- + +# Admin Settings + +On the **Admin** page, you can view users' access grants, your organization's cloud storage locations, and a complete list of connectors. +You can also get an access token to authenticate with flowctl and manage billing information. + +## Account Access + +The **Account Access** tab shows you all provisioned access grants on objects to which you also have access. +Both users and catalog prefixes can receive access grants. +These are split up into two tables called **Organization Membership** and **Data Sharing**. +Each access grant has its own row, so a given user or prefix may have multiple rows. + +For example, if you had read access to `foo/` and write access to `bar/`, you'd have a separate table row in the **Organization Membership** table for each of these capabilities. +If users Alice, Bob, and Carol each had write access on `foo/`, you'd see three more table rows representing these access grants. + +Taking this a step further, the prefix `foo/` could have read access to `buz/`. You'd see this in the **Data Sharing** table, +and it'd signify that everyone who has access to `foo/` also inherits read access to `buz/`. + +Use the search boxes to filter by username, prefix, or object. + +You can manage access by generating new user invitations, granting data sharing access, or selecting users or prefixes to revoke access. + +!["Add Users" dialogue](<./dashboard-images/access-grant-invitation.png>) + +When adding a new user to a tenant, you may choose: + +* Whether to grant access to the entire tenant or a sub-prefix within the tenant +* The user's read/write capabilities for that prefix +* Whether the invitation should be single-use or reusable + +Generating a new invitation will create a URL with a grant token parameter. +Copy the URL and share it with its intended recipient to invite them to your organization. + +[Learn more about capabilities and access.](/reference/authentication) + +## Settings + +The **Settings** tab includes additional configuration, such as organization notifications and storage mappings. + +### Organization Notifications + +Here, you are able to configure which email address(es) will receive [notifications](/reference/notifications) related to your organization or prefix. + +You can create new alert subscriptions or edit existing ones to choose which [alert types](/reference/notifications/#alert-types) you'd like to receive. + +### Collection Storage + +This section provides a table of the cloud storage locations that back your collections. +You're able to view the table if you're an admin. + +Each top-level Estuary [prefix](/concepts/catalogs/#namespace) is backed by one or more cloud storage buckets. +If you have not set up your own storage bucket(s) yet, your data is stored temporarily in Estuary's cloud storage bucket. + +[Learn more about storage mappings.](/concepts/storage-mappings) + +### Data Planes + +The **Data Planes** section provides a table of all available data plane options, broken out by **Private** and **Public** data planes. + +You can find information here related to connecting and allowing access to your data plane of choice. +See [Allowlisting IP Addresses](/reference/allow-ip-addresses) or [IAM authentication](/guides/iam-auth/aws) for more. + +## Billing + +The **Billing** tab allows you to view and manage information related to past usage, the current billing cycle, and payment methods. + +Your usage is broken down by the amount of data processed and number of task hours. See Estuary's [pricing](/getting-started/pricing) docs for details. + +View usage trends across previous months in the **Usage by Month** chart and preview your bill based on usage for the current month. +If you are on the free tier (up to 2 connectors and 10 GB per month), you will still be able to preview your bill breakdown, and will have a "Free tier credit" deduction. +To help estimate your bill, also see the [Pricing Calculator](https://estuary.dev/pricing/#pricing-calculator). + +To pay your bill, add a payment method to your account. You can choose to pay via card or bank account. +You will not be charged until you exceed the free tier's limits and have finished your subsequent trial period. + +## Connectors + +The **Connectors** tab offers a complete view of all connectors that are currently available through the dashboard, including both capture and materialization connectors. +If a connector you need is missing, you can [request it](https://github.com/estuary/connectors/issues/new/choose). + +## CLI-API + +The **CLI-API** tab provides the access token required to [authenticate with flowctl](/reference/authentication/#authenticating-estuary-using-the-cli). You can also revoke old refresh tokens. diff --git a/site/docs/guides/dashboard/captures.md b/site/docs/guides/dashboard/captures.md new file mode 100644 index 00000000000..ddd68409032 --- /dev/null +++ b/site/docs/guides/dashboard/captures.md @@ -0,0 +1,146 @@ +--- +sidebar_position: 1 +--- + +# Sources + +The **Sources** page displays a table of existing [captures](/concepts/captures) to which you have [access](/reference/authentication). +You can also create a capture with the **New Capture** button. + +![Annotated captures table](<./dashboard-images/capture-page.png>) + +**1:** Select all or deselect all. + +**2:** Enable, Disable, and Delete buttons. These actions will be applied to the selected table rows. Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the capture(s). + +**3:** Materialize button. When you click this button, you're directed to the **Create Materializations** page. +All the collections of the selected capture(s) will be added to the materialization. + +**4:** Filter captures. Type a catalog prefix, unique capture name, or connector name to return captures that match your query. + +Capture names follow the pattern `prefix/unique-identifier/connector-name`, with `prefix` supporting multiple layers of nesting. +You can search for any part of this full capture name. You can also use the `*` wildcard. +For example, if you have a capture called `acmeCo/logistics/anvil-locations/source-postgres`, +you can find it by filtering for `acmeCo*source-postgres`. + +**5:** Status indicator. Shows the capture's [shard](/concepts/advanced/shards) status. + +* **Primary (Green)**: Data is actively flowing through the capture. +* **Pending (Yellow)**: The capture is attempting to re-connect. Often, you'll see this after you re-enable the capture as Estuary backfills historical data. +* **Failed (Red)**: The capture has failed with an unrecoverable error. +* **Disabled (Hollow circle)**: The capture is disabled. +* **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error. + +**6:** Capture name. The full name is shown, including all [prefixes](/concepts/catalogs/#namespace). It also links to the [capture details page](#capture-details-page). + +**7:** Capture type. The icon shows the type of source system data is captured from. + +**8:** Capture [statistics](/concepts/advanced/logs-stats/#statistics). The **Data Written** column shows the total amount of data, in bytes and in [documents](/concepts/collections/#documents), +that the capture has written to its associated collections within a configurable time interval. +Click the time interval in the header to select from **Today**, **Yesterday**, **This Week**, **Last Week**, **This Month**, **Last Month**, or **All Time**. + +**9:** Associated collections. The **Writes to** column shows all the collections to which the capture writes data. For captures with a large number of collections, there is a chip stating how many collections are hidden. Clicking on this will allow you to hover over this column and scroll to view the full list. These also link to the details page of the collection. + +**10:** Publish time. Hover over this value to see the exact time the capture was last published. + +**11:** Edit button. Click to edit the capture. + +## Editing captures and collections + +When you click **Edit** for a capture, you're taken to the **Edit Capture** page. + +This page is similar to the [**Create Capture**](#creating-a-capture) page. +It comes pre-filled with the capture's current configuration, with the exception of sensitive SOPS-encrypted credentials. + +For detailed steps to edit a capture, see the [guide](/guides/edit-data-flows/#edit-a-capture). + +## Creating a capture + +When you click **Create Capture**, you're taken to the Create Capture page. +In the first view, all available capture connectors are displayed. + +Select your desired source system. This will open a connector-specific form where you can configure your connection. + +For detailed steps to create a capture, see the [guide](/guides/create-dataflow/#create-a-capture). + +After you successfully publish a capture, you're directed to the [capture details page](#capture-details-page) where you can review the capture's status and start materializing its associated collections. + +## Capture Details page + +When you click on the **name** of a capture on the captures page, you will be taken to the capture details page to view data stats, sharding information, and general details. + +### Overview tab + +![Annotated capture details page](<./dashboard-images/capture-details-overview.png>) + +**1:** The full name of the capture. + +**2:** Edit and Materialize buttons. Modify your capture or create a materialization to send the capture's associated data collections to a destination. + +**3:** Page tabs. Switch between the Overview, [Alerts](#alerts-tab), [Spec](#spec-tab), [History](#history-tab), and [Logs](#logs-tab) tabs for detailed information in each category. + +**4:** Usage chart filters. Click the time interval to select from **6 hours**, **12 hours**, **24 hours**, **48 hours**, **30 days**, or **Year**. +You can also choose to view usage in **Data Written** (in bytes) or **[Docs](/concepts/collections/#documents) Written**. + +**5:** Usage chart with capture [statistics](/concepts/advanced/logs-stats/#statistics). +This chart displays the data written by the capture in the chosen timeframe. Hover over a section to see specific details for that timeframe. +The most recent segment will automatically update every 15 seconds with the latest data and docs. + +**6:** Details section. This contains information about the capture, such as the connector type, the data plane being used, and create/update timestamps. + +**7:** Connector status. The current status of the connector. This may indicate backfill progress or streaming status. + +**8:** Associated collections. Shows all the collections to which the capture writes data. +These link to the collection's [detail page](/guides/dashboard/collections/#collection-details-page). + +The Overview tab also lists additional information, such as: + +* A **Shard Information** section. This displays the full identifier of the shard(s) that back your capture. If there's an error, you'll see an alert identifying the failing shard(s). + +* Any associated **Endpoints**. Webhook captures will list the public endpoint to provide to the source system. + +### Alerts tab + +The capture's Alerts tab breaks down into three main sections. + +**Notification Settings** + +This section allows you to configure a **Data Movement** alert for the capture. + +You can choose a time interval between **1 Hour** and **7 Days**. If data has not been processed within this timeframe, a notification will be sent to anyone subscribed to the alert. + +If you are not subscribed, this section will display an info box with the option to **Click Here** to subscribe. + +**Active Alerts** + +The Active Alerts section displays any currently firing alerts. +This includes details on the [alert type](/reference/notifications/#alert-types), when the alert fired, any recipient emails, and error messages or additional details related to why the alert fired. + +**Resolved Alerts** + +The Resolved Alerts section displays historical alerts. +The table provides information similar to the Active Alerts table with the addition of a resolution time for the alert. + +### Spec tab + +In the Spec tab, you can view the specification of the capture itself. + +The spec includes the capture's endpoint configuration, auto discover settings, and associated bindings. + +### History tab + +The History tab provides a change history of the capture's specification. + +The list of changes includes: + +* The date and time the change occurred +* Who initiated the change +* Any notes related to the change; for example, noting an automatic update, such as auto discovery or a dataflow reset + +Selecting a change from the list will display a **diff** to the right that highlights changes made to the specification. + +### Logs tab + +The Logs tab displays **task logs** for the capture. + +Logs include the log level, timestamp, and log message. Select a log to expand the log message. diff --git a/site/docs/guides/dashboard/collections.md b/site/docs/guides/dashboard/collections.md new file mode 100644 index 00000000000..3e0d880c52a --- /dev/null +++ b/site/docs/guides/dashboard/collections.md @@ -0,0 +1,101 @@ +--- +sidebar_position: 2 +--- + +# Collections + +The **Collections** page shows a table of [collections](/concepts/collections) to which you have access. There is also a button to begin a new derivation, or transformation. + +The table has nearly all of the same features as the [**Captures** table](./captures.md). These features include: + +* Select or deselect the collection checkbox to use the Enable, Disable, and Delete buttons. +Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the collection(s). + +* Select the collection's name to navigate to the [collection details page](#collection-details-page). +The full name is shown, including all [prefixes](/concepts/catalogs/#namespace). You may also filter collections by name to only display collections that match your query. + +* View collection status with the status indicator. If the collection does not contain a [derivation](/concepts/#derivations), the indicator should show green. +In the event that the server cannot be reached, the indicator will show "Unknown" status (black in light mode and white in dark mode). + + If the collection is a derivation, the derivation's [shard](/concepts/advanced/shards) status will be indicated: + + * **Primary (Green)**: Data is actively flowing through the derivation. + * **Pending (Yellow)**: The derivation is attempting to re-connect. + * **Failed (Red)**: The derivation has failed with an unrecoverable error. + * **Disabled (Hollow circle)**: The derivation is disabled. + * **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error. + +* View collection [statistics](/concepts/advanced/logs-stats/#statistics). +Columns include data (in bytes) and [documents](/concepts/collections/#documents) that have been read in or written out by the collection. +The time interval in the header controls the range for these statistics. + +* View the collection's last published date. Hover over the value to see the exact timestamp. + +## Collection Details page + +When you click on the **name** of a collection on the collections page, you will be taken to the collection details page. +This page includes data stats, sharding information, a preview of the collection data, and general details, similar to the [capture details page](./captures.md#capture-details-page). + +### Overview tab + +The collection overview breaks down into several main sections. + +**Usage chart** + +The usage chart contains collection [statistics](/concepts/advanced/logs-stats/#statistics). +You can select a timeframe for the data and choose between viewing usage in data processed (bytes) or [documents](/concepts/collections/#documents). + +The chart displays both the amount of data read into the collection and the amount of data written out. +Hover over a section for details on that timeframe or hover over the legend to highlight either data in or out. + +**Details section** + +The Details section displays general information about the collection, including: + +* The collection's data plane +* Create/update timestamps +* The capture that writes to the data collection, if applicable +* Any materializations that read from the data collection + +**Shard information** + +Derivations will include a section with shard information. This displays the full identifier of the shard(s) that back your collection along with the shard status. +If there's an error, you'll see an alert identifying the failing shard(s). + +**Data preview** + +The Data Preview section shows a sample of collection [documents](/concepts/collections/#documents): the individual JSON files that comprise the collection. +Documents are organized by their collection key value. Click a key from the list to view its document. + +### Alerts tab + +The collection's Alerts tab displays information on both active and historical alerts. + +The **Active Alerts** section displays any currently firing alerts. +This includes details on the [alert type](/reference/notifications/#alert-types), when the alert fired, any recipient emails, and error messages or additional details related to why the alert fired. + +The **Resolved Alerts** section displays historical alerts. +The table provides information similar to the Active Alerts table with the addition of a resolution time for the alert. + +### Spec tab + +The Spec tab provides the collection's specification, including its schema. +You can view the [schema](/concepts/schemas) as a table or the full specification JSON as code. + +The specification includes the [collection key](/concepts/collections/#keys), field names, field data types, and additional information, such as the source data type. + +:::tip +If you need to modify a collection, edit the [capture](/concepts/captures) or [derivation](/concepts/derivations) that provides its data. +::: + +### History tab + +The History tab provides a change history of the collection's specification. + +The list of changes includes: + +* The date and time the change occurred +* Who initiated the change +* Any notes related to the change; for example, noting an automatic update, such as auto discovery or a dataflow reset + +Selecting a change from the list will display a **diff** to the right that highlights changes made to the specification. diff --git a/site/docs/guides/dashboard/dashboard-images/access-grant-invitation.png b/site/docs/guides/dashboard/dashboard-images/access-grant-invitation.png new file mode 100644 index 00000000000..5eebfd93dcb Binary files /dev/null and b/site/docs/guides/dashboard/dashboard-images/access-grant-invitation.png differ diff --git a/site/docs/guides/dashboard/dashboard-images/capture-details-overview.png b/site/docs/guides/dashboard/dashboard-images/capture-details-overview.png new file mode 100644 index 00000000000..d2bbbf23eca Binary files /dev/null and b/site/docs/guides/dashboard/dashboard-images/capture-details-overview.png differ diff --git a/site/docs/guides/dashboard/dashboard-images/capture-page.png b/site/docs/guides/dashboard/dashboard-images/capture-page.png new file mode 100644 index 00000000000..b2090d05e01 Binary files /dev/null and b/site/docs/guides/dashboard/dashboard-images/capture-page.png differ diff --git a/site/docs/guides/dashboard/dashboard-images/login-screen.png b/site/docs/guides/dashboard/dashboard-images/login-screen.png new file mode 100644 index 00000000000..63cb088de69 Binary files /dev/null and b/site/docs/guides/dashboard/dashboard-images/login-screen.png differ diff --git a/site/docs/guides/dashboard/dashboard-images/materialization-details-overview.png b/site/docs/guides/dashboard/dashboard-images/materialization-details-overview.png new file mode 100644 index 00000000000..860b8644932 Binary files /dev/null and b/site/docs/guides/dashboard/dashboard-images/materialization-details-overview.png differ diff --git a/site/docs/guides/dashboard/dashboard-images/materialization-page.png b/site/docs/guides/dashboard/dashboard-images/materialization-page.png new file mode 100644 index 00000000000..51debf2324b Binary files /dev/null and b/site/docs/guides/dashboard/dashboard-images/materialization-page.png differ diff --git a/site/docs/guides/dashboard/dashboard.md b/site/docs/guides/dashboard/dashboard.md index 060a998bb64..07503662436 100644 --- a/site/docs/guides/dashboard/dashboard.md +++ b/site/docs/guides/dashboard/dashboard.md @@ -1,11 +1,23 @@ # Web Dashboard -Estuary provides a [dashboard](https://dashboard.estuary.dev/) to manage Estuary pipelines. +Estuary provides a [dashboard](https://dashboard.estuary.dev/) to manage data pipelines. -Much of the documentation assumes you will mainly be working within the dashboard. -This section in particular will explore some of the different features and functionality you can find within the UI. +This section offers a tour around the dashboard, covering the main features available through the UI. This includes pages on: -Also see the [web app](/concepts/web-app) concepts page to take a brief tour around the dashboard. +* [Capturing data from sources](./captures.md) +* [Working with data collections](./collections.md) +* [Materializing data to destinations](./materializations.md) +* [Managing your tenant with admin settings](./admin.md) -Or, if you'd rather take a more programmatic approach to pipeline management, check out our guides on working with [`flowctl`](/guides/flowctl). +If you'd rather take a more programmatic approach to pipeline management, check out our guides on working with [`flowctl`](/guides/flowctl). + +You can also see a general overview of the dashboard in the [web app](/concepts/web-app) concepts page. + +## Signing in + +You use a Google, Microsoft, or GitHub account to sign into Estuary. Alternatively, [contact us](https://estuary.dev/contact-us) about [Single Sign-On (SSO) options](/getting-started/sso-setup). + +![Estuary login screen](<./dashboard-images/login-screen.png>) + +If you've never used Estuary before, you'll be prompted to register. Choose a **tenant name** for your organization. diff --git a/site/docs/guides/dashboard/edit-data-flows.md b/site/docs/guides/dashboard/edit-data-flows.md index 6dbb0a17239..0d890ff424b 100644 --- a/site/docs/guides/dashboard/edit-data-flows.md +++ b/site/docs/guides/dashboard/edit-data-flows.md @@ -1,12 +1,12 @@ --- -sidebar_position: 1 +sidebar_position: 6 slug: /guides/edit-data-flows/ --- -# Edit Data Flows in the Web App +# Edit Data Flows in the Dashboard -You can edit existing captures, collection schemas, and materializations in Estuary's web app. -For a tour of the web app, [start here](/concepts/web-app). +You can edit existing captures, collection schemas, and materializations in Estuary's dashboard. +For a tour of the dashboard, [start here](/guides/dashboard). There are separate workflows for editing [captures](#edit-a-capture) and [materializations](#edit-a-materialization). You can edit a collection schema using Estuary's **Schema Inference** tool by editing either a capture or materialization associated diff --git a/site/docs/guides/dashboard/materializations.md b/site/docs/guides/dashboard/materializations.md new file mode 100644 index 00000000000..2065d924ce7 --- /dev/null +++ b/site/docs/guides/dashboard/materializations.md @@ -0,0 +1,147 @@ +--- +sidebar_position: 3 +--- + +# Destinations + +The **Destinations** page shows you a table of existing [materializations](/concepts/materialization) to which you have [access](/reference/authentication). +You can also create a new materialization with the **New Materialization** button. + +![Annotated materializations table](<./dashboard-images/materialization-page.png>) + +**1:** Select all or deselect all. + +**2:** Enable, Disable, and Delete buttons. These actions will be applied to the selected table rows. Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the materialization(s). + +**3:** Filter materializations. Type a catalog prefix, unique materialization name, or connector name to return materializations that match your query. + +Materialization names follow the pattern `prefix/unique-identifier/connector-name`, with `prefix` supporting multiple layers of nesting. +You can search for any part of this full materialization name. You can also use the `*` wildcard. +For example, if you have a materialization called `acmeCo/logistics/anvil-locations/materialize-snowflake`, +you can find it by filtering for `acmeCo*materialize-snowflake`. + +**4:** Status indicator. Shows the materialization's [shard](/concepts/advanced/shards) status. + +* **Primary (Green)**: Data is actively flowing through the materialization. +* **Pending (Yellow)**: The materialization is attempting to re-connect. Often, you'll see this after you re-enable the materialization as Estuary backfills historical data. +* **Failed (Red)**: The materialization has failed with an unrecoverable error. +* **Disabled (Hollow circle)**: The materialization is disabled. +* **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error. + +**5:** Materialization name. The full name is shown, including all [prefixes](/concepts/catalogs/#namespace). It also links to the [materialization details page](#materialization-details-page). + +**6:** Materialization type. The icon shows the destination system data is materialized to. + +**7:** Materialization [statistics](/concepts/advanced/logs-stats/#statistics). The **Data Written** column shows the total amount of data, in bytes and in [documents](/concepts/collections/#documents), +that the materialization has read from its associated collections within a configurable time interval. +Click the time interval in the header to select from **Today**, **Yesterday**, **This Week**, **Last Week**, **This Month**, **Last Month**, or **All Time**. + +**8:** Associated collections. The **Reads From** column shows all the collections from which the materialization reads data. For materializations with a large number of collections, there is a chip stating how many collections are hidden. Clicking on this will allow you to hover over this column and scroll to view the full list. These also link to the details page of the collection. + +**9:** Publish time. Hover over this value to see the exact time the materialization was last published. + +**10:** Edit button. Click to edit the materialization. + +## Editing materializations + +When you click **Edit** for a materialization, you're taken to the **Edit Materialization** page. + +This page is similar to the [**Create Materialization**](#creating-a-materialization) page. +It comes pre-filled with the materialization's's current configuration, with the exception of sensitive SOPS-encrypted credentials. + +For detailed steps to edit a materialization, see the [guide](/guides/edit-data-flows/#edit-a-materialization). + +## Creating a materialization + +There are several ways to begin creating a materialization: + +* Clicking **New Materialization** on the Materializations page. +* Selecting one or more captures from the Captures page and clicking **Materialize**. +* Selecting one or more collections from the Collections page and clicking **Materialize**. +* Clicking **Materialize** from a capture's details page. +* Clicking **Materialize** from a collection's details page. + +When you initiate the workflow in any of these ways, all available materialization connectors are displayed. +Select a connector to reveal the full form with configuration options specific to your desired destination. + +Fill out the **Endpoint Config** form. In the **Target Collections** section use the **Collection Selector** to map collections to resources in the destination system. + +Note that if you entered the workflow from a Capture or Collection page, collections will be pre-populated for you. + +For detailed steps to create a materialization, see the [guide](/guides/create-dataflow/#create-a-materialization). + +## Materialization Details Page + +When you click on the **name** of a materialization on the materializations page you will be taken to the detail page to view data stats, sharding information, and general details. + +### Overview tab + +![Annotated materialization details page](<./dashboard-images/materialization-details-overview.png>) + +**1:** The full name of the materialization. + +**2:** Edit button to modify your materialization. + +**3:** Page tabs. Switch between the Overview, [Alerts](#alerts-tab), [Spec](#spec-tab), [History](#history-tab), and [Logs](#logs-tab) tabs for detailed information in each category. + +**4:** Usage chart filters. Click the time interval to select from **6 hours**, **12 hours**, **24 hours**, **48 hours**, **30 days**, or **Year**. +You can also choose to view usage in **Data Read** (in bytes) or **[Docs](/concepts/collections/#documents) Read**. + +**5:** Usage chart with materialization [statistics](/concepts/advanced/logs-stats/#statistics). +This chart displays the data read by the materialization in the chosen timeframe. Hover over a section to see specific details for that timeframe. +The most recent segment will automatically update every 15 seconds with the latest data and docs. + +**6:** Details section. This contains information about the materialization, such as the connector type, the data plane being used, and create/update timestamps. + +**7:** Connector status. The current status of the connector. This may indicate backfill progress or streaming status. + +**8:** Associated collections. Shows all the collections from which the materialization reads data. +These link to the collection's [detail page](/guides/dashboard/collections/#collection-details-page). + +The Overview tab also includes a **Shard Information** section. This displays the full identifier of the shard(s) that back your materialization. If there's an error, you'll see an alert identifying the failing shard(s). + +### Alerts tab + +The materialization's Alerts tab breaks down into three main sections. + +**Notification Settings** + +This section allows you to configure a **Data Movement** alert for the materialization. + +You can choose a time interval between **1 Hour** and **7 Days**. If data has not been processed within this timeframe, a notification will be sent to anyone subscribed to the alert. + +If you are not subscribed, this section will display an info box with the option to **Click Here** to subscribe. + +**Active Alerts** + +The Active Alerts section displays any currently firing alerts. +This includes details on the [alert type](/reference/notifications/#alert-types), when the alert fired, any recipient emails, and error messages or additional details related to why the alert fired. + +**Resolved Alerts** + +The Resolved Alerts section displays historical alerts. +The table provides information similar to the Active Alerts table with the addition of a resolution time for the alert. + +### Spec tab + +In the Spec tab, you can view the specification of the materialization itself. + +The spec includes the materialization's endpoint configuration, settings to configure target naming and recommended fields, and associated bindings. + +### History tab + +The History tab provides a change history of the materialization's specification. + +The list of changes includes: + +* The date and time the change occurred +* Who initiated the change +* Any notes related to the change; for example, noting an automatic update, such as auto discovery or a dataflow reset + +Selecting a change from the list will display a **diff** to the right that highlights changes made to the specification. + +### Logs tab + +The Logs tab displays **task logs** for the materialization. + +Logs include the log level, timestamp, and log message. Select a log to expand the log message. diff --git a/site/docs/guides/flowctl/ci-cd.md b/site/docs/guides/flowctl/ci-cd.md index 99c3a5ab230..001504b1850 100644 --- a/site/docs/guides/flowctl/ci-cd.md +++ b/site/docs/guides/flowctl/ci-cd.md @@ -10,7 +10,7 @@ You can then set your infra-as-code repositories up with a CI/CD pipeline to aut This guide will show you how to configure Estuary resources programmatically for use in CI/CD workflows or other automation. -For instructions on using the UI instead, see information on Estuary's [web application](../../concepts/web-app.md). +For instructions on using the UI instead, see information on Estuary's [dashboard](/guides/dashboard). ## Creating Estuary Resources Locally diff --git a/site/docs/guides/dashboard/notifications.md b/site/docs/guides/notifications.md similarity index 100% rename from site/docs/guides/dashboard/notifications.md rename to site/docs/guides/notifications.md