-
Notifications
You must be signed in to change notification settings - Fork 87
Docs: Refresh dashboard/web app information #2893
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| { | ||
| "label": "Estuary Dashboard", | ||
| "position": 4 | ||
| "position": 1 | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| --- | ||
| sidebar_position: 4 | ||
| --- | ||
|
|
||
| # Admin Settings | ||
|
|
||
| On the **Admin** page, you can view users' access grants, your organization's cloud storage locations, and a complete list of connectors. | ||
| You can also get an access token to authenticate with flowctl and manage billing information. | ||
|
|
||
| ## Account Access | ||
|
|
||
| The **Account Access** tab shows you all provisioned access grants on objects to which you also have access. | ||
| Both users and catalog prefixes can receive access grants. | ||
| These are split up into two tables called **Organization Membership** and **Data Sharing**. | ||
| Each access grant has its own row, so a given user or prefix may have multiple rows. | ||
|
|
||
| For example, if you had read access to `foo/` and write access to `bar/`, you'd have a separate table row in the **Organization Membership** table for each of these capabilities. | ||
| If users Alice, Bob, and Carol each had write access on `foo/`, you'd see three more table rows representing these access grants. | ||
|
|
||
| Taking this a step further, the prefix `foo/` could have read access to `buz/`. You'd see this in the **Data Sharing** table, | ||
| and it'd signify that everyone who has access to `foo/` also inherits read access to `buz/`. | ||
|
|
||
| Use the search boxes to filter by username, prefix, or object. | ||
|
|
||
| You can manage access by generating new user invitations, granting data sharing access, or selecting users or prefixes to revoke access. | ||
|
|
||
|  | ||
|
|
||
| When adding a new user to a tenant, you may choose: | ||
|
|
||
| * Whether to grant access to the entire tenant or a sub-prefix within the tenant | ||
| * The user's read/write capabilities for that prefix | ||
| * Whether the invitation should be single-use or reusable | ||
|
|
||
| Generating a new invitation will create a URL with a grant token parameter. | ||
| Copy the URL and share it with its intended recipient to invite them to your organization. | ||
|
|
||
| [Learn more about capabilities and access.](/reference/authentication) | ||
|
|
||
| ## Settings | ||
|
|
||
| The **Settings** tab includes additional configuration, such as organization notifications and storage mappings. | ||
|
|
||
| ### Organization Notifications | ||
|
|
||
| Here, you are able to configure which email address(es) will receive [notifications](/reference/notifications) related to your organization or prefix. | ||
|
|
||
| You can create new alert subscriptions or edit existing ones to choose which [alert types](/reference/notifications/#alert-types) you'd like to receive. | ||
|
|
||
| ### Collection Storage | ||
|
|
||
| This section provides a table of the cloud storage locations that back your collections. | ||
| You're able to view the table if you're an admin. | ||
|
|
||
| Each top-level Estuary [prefix](/concepts/catalogs/#namespace) is backed by one or more cloud storage bucket that you own. | ||
| If you have not set up your own storage bucket(s) yet, your data is stored temporarily in Estuary's cloud storage bucket. | ||
|
|
||
| [Learn more about storage mappings.](/concepts/storage-mappings) | ||
|
|
||
| ### Data Planes | ||
|
|
||
| The **Data Planes** section provides a table of all available data plane options, broken out by **Private** and **Public** data planes. | ||
|
|
||
| You can find information here related to connecting and allowing access to your data plane of choice. | ||
| See [Allowlisting IP Addresses](/reference/allow-ip-addresses) or [IAM authentication](/guides/iam-auth/aws) for more. | ||
|
|
||
| ## Billing | ||
|
|
||
| The **Billing** tab allows you to view and manage information related to past usage, the current billing cycle, and payment methods. | ||
|
|
||
| Your usage is broken down by the amount of data processed and number of task hours. See Estuary's [pricing](/getting-started/pricing) docs for details. | ||
|
|
||
| View usage trends across previous months in the **Usage by Month** chart and preview your bill based on usage for the current month. | ||
| If you are on the free tier (up to 2 connectors and 10 GB per month), you will still be able to preview your bill breakdown, and will have a "Free tier credit" deduction. | ||
| To help estimate your bill, also see the [Pricing Calculator](https://estuary.dev/pricing/#pricing-calculator). | ||
|
|
||
| To pay your bill, add a payment method to your account. You can choose to pay via card or bank account. | ||
| You will not be charged until you exceed the free tier's limits and have finished your subsequent trial period. | ||
|
|
||
| ## Connectors | ||
|
|
||
| The **Connectors** tab offers a complete view of all connectors that are currently available through the dashboard, including both capture and materialization connectors. | ||
| If a connector you need is missing, you can [request it](https://github.com/estuary/connectors/issues/new/choose). | ||
|
|
||
| ## CLI-API | ||
|
|
||
| The **CLI-API** tab provides the access token required to [authenticate with flowctl](/reference/authentication/#authenticating-estuary-using-the-cli). You can also revoke old refresh tokens. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| --- | ||
| sidebar_position: 1 | ||
| --- | ||
|
|
||
| # Sources | ||
|
|
||
| The **Sources** page displays a table of existing [captures](/concepts/captures) to which you have [access](/reference/authentication). | ||
| You can also create a capture with the **New Capture** button. | ||
|
|
||
|  | ||
|
|
||
| **1:** Select all or deselect all. | ||
|
|
||
| **2:** Enable, Disable, and Delete buttons. These actions will be applied to the selected table rows. Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the capture(s). | ||
|
|
||
| **3:** Materialize button. When you click this button, you're directed to the **Create Materializations** page. | ||
| All the collections of the selected capture(s) will be added to the materialization. | ||
|
|
||
| **4:** Filter captures. Type a catalog prefix, unique capture name, or connector name to return captures that match your query. | ||
|
|
||
| Capture names follow the pattern `prefix/unique-identifier/connector-name`, with `prefix` supporting multiple layers of nesting. | ||
| You can search for any part of this full capture name. You can also use the `*` wildcard. | ||
| For example, if you have a capture called `acmeCo/logistics/anvil-locations/source-postgres`, | ||
| you can find it by filtering for `acmeCo*source-postgres`. | ||
|
|
||
| **5:** Status indicator. Shows the status of the primary task [shard](/concepts/advanced/shards) that backs this capture. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I will leave this thought here - but applies to materializations as well. I think we are technically doing some logic to "bubble" up shard status and it is not always just the primary task's status. I doubt we need to get to this level detail here though considering the average audience just needs to know what the colors mean.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Interesting. I reworded this to be more general. If the status is an amalgamation, is there a reason that the dashboard message when hovering over active/green status is |
||
|
|
||
| * **Primary (Green)**: Data is actively flowing through the capture. | ||
| * **Pending (Yellow)**: The capture is attempting to re-connect. Often, you'll see this after you re-enable the capture as Estuary backfills historical data. | ||
| * **Failed (Red)**: The capture has failed with an unrecoverable error. | ||
| * **Disabled (Hollow circle)**: The capture is disabled. | ||
| * **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error. | ||
|
|
||
| **6:** Capture name. The full name is shown, including all [prefixes](/concepts/catalogs/#namespace). It also links to the [capture details page](#capture-details-page). | ||
|
|
||
| **7:** Capture type. The icon shows the type of source system data is captured from. | ||
|
|
||
| **8:** Capture [statistics](/concepts/advanced/logs-stats/#statistics). The **Data Written** column shows the total amount of data, in bytes and in [documents](/concepts/collections/#documents), | ||
| that the capture has written to its associated collections within a configurable time interval. | ||
| Click the time interval in the header to select from **Today**, **Yesterday**, **This Week**, **Last Week**, **This Month**, **Last Month**, or **All Time**. | ||
|
|
||
| **9:** Associated collections. The **Writes to** column shows all the collections to which the capture writes data. For captures with a large number of collections, there is a chip stating how many collections are hidden. Clicking on this will allow you to hover over this column and scroll to view the full list. These also link to the details page of the collection. | ||
|
|
||
| **10:** Publish time. Hover over this value to see the exact time the capture was last published. | ||
|
|
||
| **11:** Edit button. Click to edit the capture. | ||
|
|
||
| ## Editing captures and collections | ||
|
|
||
| When you click **Edit** for a capture, you're taken to the **Edit Capture** page. | ||
|
|
||
| This page is similar to the [**Create Capture**](#creating-a-capture) page. | ||
| It comes pre-filled with the capture's current configuration, with the exception of sensitive SOPS-encrypted credentials. | ||
|
|
||
| For detailed steps to edit a capture, see the [guide](/guides/edit-data-flows/#edit-a-capture). | ||
|
|
||
| ## Creating a capture | ||
|
|
||
| When you click **Create Capture**, you're taken to the Create Capture page. | ||
| In the first view, all available capture connectors are displayed. | ||
|
|
||
| Select the tile of the system from which you want to capture data to show the full capture form. | ||
| The form details are specific to the connector you chose. | ||
|
aeluce marked this conversation as resolved.
Outdated
|
||
|
|
||
| For detailed steps to create a capture, see the [guide](/guides/create-dataflow/#create-a-capture). | ||
|
|
||
| After you successfully publish a capture, you're directed to the [capture details page](#capture-details-page) where you can review the capture's status and start materializing its associated collections. | ||
|
|
||
| ## Capture Details page | ||
|
|
||
| When you click on the **name** of a capture on the captures page, you will be taken to the capture details page to view data stats, sharding information, and general details. | ||
|
|
||
| ### Overview tab | ||
|
|
||
|  | ||
|
|
||
| **1:** The full name of the capture. | ||
|
|
||
| **2:** Edit and Materialize buttons. Modify your capture or create a materialization to send the capture's associated data collections to a destination. | ||
|
|
||
| **3:** Page tabs. Switch between the Overview, [Alerts](#alerts-tab), [Spec](#spec-tab), [History](#history-tab), and [Logs](#logs-tab) tabs for detailed information in each category. | ||
|
|
||
| **4:** Usage chart filters. Click the time interval to select from **6 hours**, **12 hours**, **24 hours**, **48 hours**, **30 days**, or **Year**. | ||
| You can also choose to view usage in **Data Written** (in bytes) or **[Docs](/concepts/collections/#documents) Written**. | ||
|
|
||
| **5:** Usage chart with capture [statistics](/concepts/advanced/logs-stats/#statistics). | ||
| This chart displays the data written by the capture in the chosen timeframe. Hover over a section to see specific details for that timeframe. | ||
| The most recent segment will automatically update every 15 seconds with the latest data and docs. | ||
|
|
||
| **6:** Details section. This contains information about the capture, such as the connector type, the data plane being used, and create/update timestamps. | ||
|
|
||
| **7:** Connector status. The current status of the connector. This may indicate backfill progress or streaming status. | ||
|
|
||
| **8:** Associated collections. Shows all the collections to which the capture writes data. | ||
| These link to the collection's [detail page](/guides/dashboard/collections/#collection-details-page). | ||
|
|
||
| The Overview tab also lists additional information, such as: | ||
|
|
||
| * A **Shard Information** section. This displays the full identifier of the shard(s) that back your capture. If there's an error, you'll see an alert identifying the failing shard(s). | ||
|
|
||
| * Any associated **Endpoints**. Webhook captures will list the public endpoint to provide to the source system. | ||
|
|
||
| ### Alerts tab | ||
|
|
||
| The capture's Alerts tab breaks down into three main sections. | ||
|
|
||
| **Notification Settings** | ||
|
|
||
| This section allows you to configure a **Data Movement** alert for the capture. | ||
|
|
||
| You can choose a time interval between **1 Hour** and **7 Days**. If data has not been processed within this timeframe, a notification will be sent to anyone subscribed to the alert. | ||
|
|
||
| If you are not subscribed, this section will display an info box with the option to **Click Here** to subscribe. | ||
|
|
||
| **Active Alerts** | ||
|
|
||
| The Active Alerts section displays any currently firing alerts. | ||
| This includes details on the [alert type](/reference/notifications/#alert-types), when the alert fired, any recipient emails, and error messages or additional details related to why the alert fired. | ||
|
|
||
| **Resolved Alerts** | ||
|
|
||
| The Resolved Alerts section displays historical alerts. | ||
| The table provides information similar to the Active Alerts table with the addition of a resolution time for the alert. | ||
|
|
||
| ### Spec tab | ||
|
|
||
| In the Spec tab, you can view the specification of the capture itself. | ||
|
|
||
| The spec includes the capture's endpoint configuration, auto discover settings, and associated bindings. | ||
|
|
||
| ### History tab | ||
|
|
||
| The History tab provides a change history of the capture's specification. | ||
|
|
||
| The list of changes includes: | ||
|
|
||
| * The date and time the change occurred | ||
| * Who initiated the change | ||
| * Any notes related to the change; for example, noting an automatic update, such as auto discovery or a dataflow reset | ||
|
|
||
| Selecting a change from the list will display a **diff** to the right that highlights changes made to the specification. | ||
|
|
||
| ### Logs tab | ||
|
|
||
| The Logs tab displays **task logs** for the capture. | ||
|
|
||
| Logs include the log level, timestamp, and log message. Select a log to expand the log message. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| --- | ||
| sidebar_position: 2 | ||
| --- | ||
|
|
||
| # Collections | ||
|
|
||
| The **Collections** page shows a table of [collections](/concepts/collections) to which you have access. There is also a button to begin a new derivation, or transformation. | ||
|
|
||
| The table has nearly all of the same features as the [**Captures** table](./captures.md). These features include: | ||
|
|
||
| * Select or deselect the collection checkbox to use the Enable, Disable, and Delete buttons. | ||
| Choose **Disable** to temporarily pause the flow of data, **Enable** to resume, and **Delete** to permanently remove the collection(s). | ||
|
|
||
| * Select the collection's name to navigate to the [collection details page](#collection-details-page). | ||
| The full name is shown, including all [prefixes](/concepts/catalogs/#namespace). You may also filter collections by name to only display collections that match your query. | ||
|
|
||
| * View collection status with the status indicator. If the collection does not contain a [derivation](/concepts/#derivations), the indicator should show green. | ||
| In the event that the server cannot be reached, the indicator will show "Unknown" status (black in light mode and white in dark mode). | ||
|
|
||
| If the collection contains a derivation, the status of the derivation's primary task [shard](/concepts/advanced/shards) will be indicated: | ||
|
|
||
| * **Primary (Green)**: Data is actively flowing through the derivation. | ||
| * **Pending (Yellow)**: The derivation is attempting to re-connect. | ||
| * **Failed (Red)**: The derivation has failed with an unrecoverable error. | ||
| * **Disabled (Hollow circle)**: The derivation is disabled. | ||
| * **Unknown (Black when app is in light mode; white when app is in dark mode)**: The web app is unable to determine shard status. Usually, this is due to a temporary connection error. | ||
|
|
||
| * View collection [statistics](/concepts/advanced/logs-stats/#statistics). | ||
| Columns include data (in bytes) and [documents](/concepts/collections/#documents) that have been read in or written out by the collection. | ||
| The time interval in the header controls the range for these statistics. | ||
|
|
||
| * View the collection's last published date. Hover over the value to see the exact timestamp. | ||
|
|
||
| ## Collection Details page | ||
|
|
||
| When you click on the **name** of a collection on the collections page, you will be taken to the collection details page. | ||
| This page includes data stats, sharding information, a preview of the collection data, and general details, similar to the [capture details page](./captures.md#capture-details-page). | ||
|
|
||
| ### Overview tab | ||
|
|
||
| The collection overview breaks down into several main sections. | ||
|
|
||
| **Usage chart** | ||
|
|
||
| The usage chart contains collection [statistics](/concepts/advanced/logs-stats/#statistics). | ||
| You can select a timeframe for the data and choose between viewing usage in data processed (bytes) or [documents](/concepts/collections/#documents). | ||
|
|
||
| The chart displays both the amount of data read into the collection and the amount of data written out. | ||
| Hover over a section for details on that timeframe or hover over the legend to highlight either data in or out. | ||
|
|
||
| **Details section** | ||
|
|
||
| The Details section displays general information about the collection, including: | ||
|
|
||
| * The collection's data plane | ||
| * Create/update timestamps | ||
| * The capture that writes to the data collection, if applicable | ||
| * Any materializations that read from the data collection | ||
|
|
||
| **Shard information** | ||
|
|
||
| Derivations will include a section with shard information. This displays the full identifier of the shard(s) that back your collection along with the shard status. | ||
| If there's an error, you'll see an alert identifying the failing shard(s). | ||
|
|
||
| **Data preview** | ||
|
|
||
| The Data Preview section shows a sample of collection [documents](/concepts/collections/#documents): the individual JSON files that comprise the collection. | ||
| Documents are organized by their collection key value. Click a key from the list to view its document. | ||
|
|
||
| ### Alerts tab | ||
|
|
||
| The collection's Alerts tab displays information on both active and historical alerts. | ||
|
|
||
| The **Active Alerts** section displays any currently firing alerts. | ||
| This includes details on the [alert type](/reference/notifications/#alert-types), when the alert fired, any recipient emails, and error messages or additional details related to why the alert fired. | ||
|
|
||
| The **Resolved Alerts** section displays historical alerts. | ||
| The table provides information similar to the Active Alerts table with the addition of a resolution time for the alert. | ||
|
|
||
| ### Spec tab | ||
|
|
||
| The Spec tab provides the collection's specification, including its schema. | ||
| You can view the [schema](/concepts/schemas) as a table or the full specification JSON as code. | ||
|
|
||
| The specification includes the [collection key](/concepts/collections/#keys), field names, field data types, and additional information, such as the source data type. | ||
|
|
||
| :::tip | ||
| If you need to modify a collection, edit the [capture](/concepts/captures) or [derivation](/concepts/derivations) that provides its data. | ||
| ::: | ||
|
|
||
| ### History tab | ||
|
|
||
| The History tab provides a change history of the collection's specification. | ||
|
|
||
| The list of changes includes: | ||
|
|
||
| * The date and time the change occurred | ||
| * Who initiated the change | ||
| * Any notes related to the change; for example, noting an automatic update, such as auto discovery or a dataflow reset | ||
|
|
||
| Selecting a change from the list will display a **diff** to the right that highlights changes made to the specification. |
Uh oh!
There was an error while loading. Please reload this page.