Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion site/docs/concepts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Estuary's web application is where you'll most likely create, monitor, and updat
You can find it at [dashboard.estuary.dev](https://dashboard.estuary.dev).
The app is backed by secure, cloud-hosted infrastructure that Estuary manages.

[Take a tour of the web app](web-app.md).
[Learn more about the dashboard](web-app.md).

[Create your first Data Flow with the web app](../guides/create-dataflow.md).

Expand Down
386 changes: 57 additions & 329 deletions site/docs/concepts/web-app.md

Large diffs are not rendered by default.

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed site/docs/concepts/webapp-images/capture-page.png
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed site/docs/concepts/webapp-images/login-screen.png
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
2 changes: 1 addition & 1 deletion site/docs/getting-started/sso-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ You may invite additional members of your company to collaborate with you. Estua

1. Meanwhile, the tenant admin can sign into their Estuary dashboard and navigate to the **Admin** tab.

2. Under [Account Access](../concepts/web-app.md#account-access), in the **Organization Membership** section, click the **Add Users** button.
2. Under [Account Access](/guides/dashboard/admin/#account-access), in the **Organization Membership** section, click the **Add Users** button.

3. Choose your desired prefix (such as your entire tenant or a defined subset), capability (admin or read access), and type (multi- or single-use). Then create an invite link.

Expand Down
2 changes: 1 addition & 1 deletion site/docs/getting-started/support.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,4 @@ You should then see `estuary_support/` in the list of entities that you're shari

When your issue is resolved, you can revoke access again by selecting the `estuary_support/` row and clicking **Remove**.

Learn more about [access grants](../concepts/web-app.md#account-access).
Learn more about [access grants](/guides/dashboard/admin/#account-access).
2 changes: 1 addition & 1 deletion site/docs/guides/dashboard/_category_.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"label": "Estuary Dashboard",
"position": 4
"position": 1
}
87 changes: 87 additions & 0 deletions site/docs/guides/dashboard/admin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
sidebar_position: 4
---

# Admin Settings

On the **Admin** page, you can view users' access grants, your organization's cloud storage locations, and a complete list of connectors.
You can also get an access token to authenticate with flowctl and manage billing information.

## Account Access

The **Account Access** tab shows you all provisioned access grants on objects to which you also have access.
Both users and catalog prefixes can receive access grants.
These are split up into two tables called **Organization Membership** and **Data Sharing**.
Each access grant has its own row, so a given user or prefix may have multiple rows.

For example, if you had read access to `foo/` and write access to `bar/`, you'd have a separate table row in the **Organization Membership** table for each of these capabilities.
If users Alice, Bob, and Carol each had write access on `foo/`, you'd see three more table rows representing these access grants.

Taking this a step further, the prefix `foo/` could have read access to `buz/`. You'd see this in the **Data Sharing** table,
and it'd signify that everyone who has access to `foo/` also inherits read access to `buz/`.

Use the search boxes to filter by username, prefix, or object.

You can manage access by generating new user invitations, granting data sharing access, or selecting users or prefixes to revoke access.

!["Add Users" dialogue](<./dashboard-images/access-grant-invitation.png>)

When adding a new user to a tenant, you may choose:

* Whether to grant access to the entire tenant or a sub-prefix within the tenant
* The user's read/write capabilities for that prefix
* Whether the invitation should be single-use or reusable

Generating a new invitation will create a URL with a grant token parameter.
Copy the URL and share it with its intended recipient to invite them to your organization.

[Learn more about capabilities and access.](/reference/authentication)

## Settings

The **Settings** tab includes additional configuration, such as organization notifications and storage mappings.

### Organization Notifications

Here, you are able to configure which email address(es) will receive [notifications](/reference/notifications) related to your organization or prefix.

You can create new alert subscriptions or edit existing ones to choose which [alert types](/reference/notifications/#alert-types) you'd like to receive.

### Collection Storage

This section provides a table of the cloud storage locations that back your collections.
You're able to view the table if you're an admin.

Each top-level Estuary [prefix](/concepts/catalogs/#namespace) is backed by one or more cloud storage bucket that you own.
Comment thread
aeluce marked this conversation as resolved.
Outdated
If you have not set up your own storage bucket(s) yet, your data is stored temporarily in Estuary's cloud storage bucket.

[Learn more about storage mappings.](/concepts/storage-mappings)

### Data Planes

The **Data Planes** section provides a table of all available data plane options, broken out by **Private** and **Public** data planes.

You can find information here related to connecting and allowing access to your data plane of choice.
See [Allowlisting IP Addresses](/reference/allow-ip-addresses) or [IAM authentication](/guides/iam-auth/aws) for more.

## Billing

The **Billing** tab allows you to view and manage information related to past usage, the current billing cycle, and payment methods.

Your usage is broken down by the amount of data processed and number of task hours. See Estuary's [pricing](/getting-started/pricing) docs for details.

View usage trends across previous months in the **Usage by Month** chart and preview your bill based on usage for the current month.
If you are on the free tier (up to 2 connectors and 10 GB per month), you will still be able to preview your bill breakdown, and will have a "Free tier credit" deduction.
To help estimate your bill, also see the [Pricing Calculator](https://estuary.dev/pricing/#pricing-calculator).

To pay your bill, add a payment method to your account. You can choose to pay via card or bank account.
You will not be charged until you exceed the free tier's limits and have finished your subsequent trial period.

## Connectors

The **Connectors** tab offers a complete view of all connectors that are currently available through the dashboard, including both capture and materialization connectors.
If a connector you need is missing, you can [request it](https://github.com/estuary/connectors/issues/new/choose).

## CLI-API

The **CLI-API** tab provides the access token required to [authenticate with flowctl](/reference/authentication/#authenticating-estuary-using-the-cli). You can also revoke old refresh tokens.
147 changes: 147 additions & 0 deletions site/docs/guides/dashboard/captures.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
---
sidebar_position: 1
---

# Sources

The **Sources** page displays a table of existing [captures](/concepts/captures) to which you have [access](/reference/authentication).
You can also create a capture with the **New Capture** button.

![Annotated captures table](<./dashboard-images/capture-page.png>)

**1:** Select all or deselect all.

**2:** Enable, Disable, and Delete buttons. These actions will be applied to the selected table rows. Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the capture(s).

**3:** Materialize button. When you click this button, you're directed to the **Create Materializations** page.
All the collections of the selected capture(s) will be added to the materialization.

**4:** Filter captures. Type a catalog prefix, unique capture name, or connector name to return captures that match your query.

Capture names follow the pattern `prefix/unique-identifier/connector-name`, with `prefix` supporting multiple layers of nesting.
You can search for any part of this full capture name. You can also use the `*` wildcard.
For example, if you have a capture called `acmeCo/logistics/anvil-locations/source-postgres`,
you can find it by filtering for `acmeCo*source-postgres`.

**5:** Status indicator. Shows the status of the primary task [shard](/concepts/advanced/shards) that backs this capture.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will leave this thought here - but applies to materializations as well.

I think we are technically doing some logic to "bubble" up shard status and it is not always just the primary task's status. I doubt we need to get to this level detail here though considering the average audience just needs to know what the colors mean.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. I reworded this to be more general. If the status is an amalgamation, is there a reason that the dashboard message when hovering over active/green status is PRIMARY?


* **Primary (Green)**: Data is actively flowing through the capture.
* **Pending (Yellow)**: The capture is attempting to re-connect. Often, you'll see this after you re-enable the capture as Estuary backfills historical data.
* **Failed (Red)**: The capture has failed with an unrecoverable error.
* **Disabled (Hollow circle)**: The capture is disabled.
* **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error.

**6:** Capture name. The full name is shown, including all [prefixes](/concepts/catalogs/#namespace). It also links to the [capture details page](#capture-details-page).

**7:** Capture type. The icon shows the type of source system data is captured from.

**8:** Capture [statistics](/concepts/advanced/logs-stats/#statistics). The **Data Written** column shows the total amount of data, in bytes and in [documents](/concepts/collections/#documents),
that the capture has written to its associated collections within a configurable time interval.
Click the time interval in the header to select from **Today**, **Yesterday**, **This Week**, **Last Week**, **This Month**, **Last Month**, or **All Time**.

**9:** Associated collections. The **Writes to** column shows all the collections to which the capture writes data. For captures with a large number of collections, there is a chip stating how many collections are hidden. Clicking on this will allow you to hover over this column and scroll to view the full list. These also link to the details page of the collection.

**10:** Publish time. Hover over this value to see the exact time the capture was last published.

**11:** Edit button. Click to edit the capture.

## Editing captures and collections

When you click **Edit** for a capture, you're taken to the **Edit Capture** page.

This page is similar to the [**Create Capture**](#creating-a-capture) page.
It comes pre-filled with the capture's current configuration, with the exception of sensitive SOPS-encrypted credentials.

For detailed steps to edit a capture, see the [guide](/guides/edit-data-flows/#edit-a-capture).

## Creating a capture

When you click **Create Capture**, you're taken to the Create Capture page.
In the first view, all available capture connectors are displayed.

Select the tile of the system from which you want to capture data to show the full capture form.
The form details are specific to the connector you chose.
Comment thread
aeluce marked this conversation as resolved.
Outdated

For detailed steps to create a capture, see the [guide](/guides/create-dataflow/#create-a-capture).

After you successfully publish a capture, you're directed to the [capture details page](#capture-details-page) where you can review the capture's status and start materializing its associated collections.

## Capture Details page

When you click on the **name** of a capture on the captures page, you will be taken to the capture details page to view data stats, sharding information, and general details.

### Overview tab

![Annotated capture details page](<./dashboard-images/capture-details-overview.png>)

**1:** The full name of the capture.

**2:** Edit and Materialize buttons. Modify your capture or create a materialization to send the capture's associated data collections to a destination.

**3:** Page tabs. Switch between the Overview, [Alerts](#alerts-tab), [Spec](#spec-tab), [History](#history-tab), and [Logs](#logs-tab) tabs for detailed information in each category.

**4:** Usage chart filters. Click the time interval to select from **6 hours**, **12 hours**, **24 hours**, **48 hours**, **30 days**, or **Year**.
You can also choose to view usage in **Data Written** (in bytes) or **[Docs](/concepts/collections/#documents) Written**.

**5:** Usage chart with capture [statistics](/concepts/advanced/logs-stats/#statistics).
This chart displays the data written by the capture in the chosen timeframe. Hover over a section to see specific details for that timeframe.
The most recent segment will automatically update every 15 seconds with the latest data and docs.

**6:** Details section. This contains information about the capture, such as the connector type, the data plane being used, and create/update timestamps.

**7:** Connector status. The current status of the connector. This may indicate backfill progress or streaming status.

**8:** Associated collections. Shows all the collections to which the capture writes data.
These link to the collection's [detail page](/guides/dashboard/collections/#collection-details-page).

The Overview tab also lists additional information, such as:

* A **Shard Information** section. This displays the full identifier of the shard(s) that back your capture. If there's an error, you'll see an alert identifying the failing shard(s).

* Any associated **Endpoints**. Webhook captures will list the public endpoint to provide to the source system.

### Alerts tab

The capture's Alerts tab breaks down into three main sections.

**Notification Settings**

This section allows you to configure a **Data Movement** alert for the capture.

You can choose a time interval between **1 Hour** and **7 Days**. If data has not been processed within this timeframe, a notification will be sent to anyone subscribed to the alert.

If you are not subscribed, this section will display an info box with the option to **Click Here** to subscribe.

**Active Alerts**

The Active Alerts section displays any currently firing alerts.
This includes details on the [alert type](/reference/notifications/#alert-types), when the alert fired, any recipient emails, and error messages or additional details related to why the alert fired.

**Resolved Alerts**

The Resolved Alerts section displays historical alerts.
The table provides information similar to the Active Alerts table with the addition of a resolution time for the alert.

### Spec tab

In the Spec tab, you can view the specification of the capture itself.

The spec includes the capture's endpoint configuration, auto discover settings, and associated bindings.

### History tab

The History tab provides a change history of the capture's specification.

The list of changes includes:

* The date and time the change occurred
* Who initiated the change
* Any notes related to the change; for example, noting an automatic update, such as auto discovery or a dataflow reset

Selecting a change from the list will display a **diff** to the right that highlights changes made to the specification.

### Logs tab

The Logs tab displays **task logs** for the capture.

Logs include the log level, timestamp, and log message. Select a log to expand the log message.
101 changes: 101 additions & 0 deletions site/docs/guides/dashboard/collections.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
---
sidebar_position: 2
---

# Collections

The **Collections** page shows a table of [collections](/concepts/collections) to which you have access. There is also a button to begin a new derivation, or transformation.

The table has nearly all of the same features as the [**Captures** table](./captures.md). These features include:

* Select or deselect the collection checkbox to use the Enable, Disable, and Delete buttons.
Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the collection(s).

* Select the collection's name to navigate to the [collection details page](#collection-details-page).
The full name is shown, including all [prefixes](/concepts/catalogs/#namespace). You may also filter collections by name to only display collections that match your query.

* View collection status with the status indicator. If the collection does not contain a [derivation](/concepts/#derivations), the indicator should show green.
In the event that the server cannot be reached, the indicator will show "Unknown" status (black in light mode and white in dark mode).

If the collection contains a derivation, the status of the derivation's primary task [shard](/concepts/advanced/shards) will be indicated:

* **Primary (Green)**: Data is actively flowing through the derivation.
* **Pending (Yellow)**: The derivation is attempting to re-connect.
* **Failed (Red)**: The derivation has failed with an unrecoverable error.
* **Disabled (Hollow circle)**: The derivation is disabled.
* **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error.

* View collection [statistics](/concepts/advanced/logs-stats/#statistics).
Columns include data (in bytes) and [documents](/concepts/collections/#documents) that have been read in or written out by the collection.
The time interval in the header controls the range for these statistics.

* View the collection's last published date. Hover over the value to see the exact timestamp.

## Collection Details page

When you click on the **name** of a collection on the collections page, you will be taken to the collection details page.
This page includes data stats, sharding information, a preview of the collection data, and general details, similar to the [capture details page](./captures.md#capture-details-page).

### Overview tab

The collection overview breaks down into several main sections.

**Usage chart**

The usage chart contains collection [statistics](/concepts/advanced/logs-stats/#statistics).
You can select a timeframe for the data and choose between viewing usage in data processed (bytes) or [documents](/concepts/collections/#documents).

The chart displays both the amount of data read into the collection and the amount of data written out.
Hover over a section for details on that timeframe or hover over the legend to highlight either data in or out.

**Details section**

The Details section displays general information about the collection, including:

* The collection's data plane
* Create/update timestamps
* The capture that writes to the data collection, if applicable
* Any materializations that read from the data collection

**Shard information**

Derivations will include a section with shard information. This displays the full identifier of the shard(s) that back your collection along with the shard status.
If there's an error, you'll see an alert identifying the failing shard(s).

**Data preview**

The Data Preview section shows a sample of collection [documents](/concepts/collections/#documents): the individual JSON files that comprise the collection.
Documents are organized by their collection key value. Click a key from the list to view its document.

### Alerts tab

The collection's Alerts tab displays information on both active and historical alerts.

The **Active Alerts** section displays any currently firing alerts.
This includes details on the [alert type](/reference/notifications/#alert-types), when the alert fired, any recipient emails, and error messages or additional details related to why the alert fired.

The **Resolved Alerts** section displays historical alerts.
The table provides information similar to the Active Alerts table with the addition of a resolution time for the alert.

### Spec tab

The Spec tab provides the collection's specification, including its schema.
You can view the [schema](/concepts/schemas) as a table or the full specification JSON as code.

The specification includes the [collection key](/concepts/collections/#keys), field names, field data types, and additional information, such as the source data type.

:::tip
If you need to modify a collection, edit the [capture](/concepts/captures) or [derivation](/concepts/derivations) that provides its data.
:::

### History tab

The History tab provides a change history of the collection's specification.

The list of changes includes:

* The date and time the change occurred
* Who initiated the change
* Any notes related to the change; for example, noting an automatic update, such as auto discovery or a dataflow reset

Selecting a change from the list will display a **diff** to the right that highlights changes made to the specification.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading