diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 80c33ccdc8e..52d8fa5888d 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -23,8 +23,7 @@ work on an issue, comment on it first and tell us the approach you want to take. * Increase our test coverage. * Fix a [bug](https://github.com/hashicorp/consul/labels/type/bug). * Implement a requested [enhancement](https://github.com/hashicorp/consul/labels/type/enhancement). -* Improve our documentation and tutorials. Consul's [Documentation](https://developer.hashicorp.com/consul/docs) and [api godoc](https://godoc.org/github.com/hashicorp/consul/api) -are deployed from this repo. +* Improve our documentation and tutorials. Consul's [Documentation](https://developer.hashicorp.com/consul/docs) content is maintained in [hashicorp/web-unified-docs](https://github.com/hashicorp/web-unified-docs/tree/main/content/consul), and the [api godoc](https://godoc.org/github.com/hashicorp/consul/api) is published from this repo. * Respond to questions about usage on the issue tracker or the Consul section of the [HashiCorp forum]: (https://discuss.hashicorp.com/c/consul) ### Reporting an Issue @@ -107,10 +106,9 @@ If a dependency is added or change, run `go mod tidy` to update `go.mod` and `go #### Developer Documentation -Developer-focused documentation about the Consul code base is under [./docs], -and godoc package document can be read at [pkg.go.dev/github.com/hashicorp/consul]. +Developer-focused documentation about content is maintained in [hashicorp/web-unified-docs](https://github.com/hashicorp/web-unified-docs/tree/main/content/consul), +and godoc package documentation can be read at [pkg.go.dev/github.com/hashicorp/consul]. -[./docs]: ../docs/README.md [pkg.go.dev/github.com/hashicorp/consul]: https://pkg.go.dev/github.com/hashicorp/consul ### Testing diff --git a/.github/workflows/go-tests.yml b/.github/workflows/go-tests.yml index 1eaa5d941db..31464814745 100644 --- a/.github/workflows/go-tests.yml +++ b/.github/workflows/go-tests.yml @@ -4,10 +4,8 @@ on: pull_request: branches-ignore: - stable-website - - 'docs/**' - 'ui/**' - 'mktg-**' # Digital Team Terraform-generated branches' prefix - - 'backport/docs/**' - 'backport/ui/**' - 'backport/mktg-**' push: diff --git a/.github/workflows/reusable-conditional-skip.yml b/.github/workflows/reusable-conditional-skip.yml index 7683c728038..e56974fc607 100644 --- a/.github/workflows/reusable-conditional-skip.yml +++ b/.github/workflows/reusable-conditional-skip.yml @@ -46,7 +46,9 @@ jobs: files: | .github/workflows/reusable-conditional-skip.yml **.md - docs/** + docs/config/checklist-adding-config-fields.md + docs/contributing/add-a-changelog-entry.md + docs/contributing/fork-the-project.md ui/** website/** grafana/** diff --git a/.github/workflows/security-scan.yml b/.github/workflows/security-scan.yml index f4bc4d72854..ab6ee3eeacb 100644 --- a/.github/workflows/security-scan.yml +++ b/.github/workflows/security-scan.yml @@ -14,7 +14,9 @@ on: # paths-ignore only works for non-required checks. # Jobs that are required for merge must use reusable-conditional-skip.yml. paths-ignore: - - 'docs/**' + - 'docs/config/checklist-adding-config-fields.md' + - 'docs/contributing/add-a-changelog-entry.md' + - 'docs/contributing/fork-the-project.md' - 'grafana/**' - '.changelog/**' diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index 99e6f7e7a2b..00000000000 --- a/docs/README.md +++ /dev/null @@ -1,88 +0,0 @@ -# Consul Developer Documentation - -See [our contributing guide](../.github/CONTRIBUTING.md) to get started. - -This directory contains documentation intended for anyone interested in -understanding, and contributing changes to, the Consul codebase. - -## Overview - -This documentation is organized into the following categories. Each category is -either a significant architectural layer, or major functional area of Consul. -These documents assume a basic understanding of Consul's feature set, which can -be found in the public [user documentation]. - -[user documentation]: https://developer.hashicorp.com/docs - -![Overview](./overview.svg) - -[source](./overview.mmd) - -## Contents - -1. [Command-Line Interface (CLI)](./cli) -1. [HTTP API](./http-api) -1. [Agent Configuration](./config) -1. [RPC](./rpc) -1. [Cluster Persistence](./persistence) -1. [V2 Architecture](v2-architecture) -1. [Client Agent](./client-agent) -1. [Service Discovery](./service-discovery) -1. [Service Mesh (Connect)](./service-mesh) -1. [Cluster Membership](./cluster-membership) -1. [Key/Value Store](./kv) -1. [ACL](./acl) -1. [Multi-Cluster Federation](./cluster-federation) - -Also see the [FAQ](./faq.md). - -## Other Docs - -1. [Integration Tests](../test/integration/connect/envoy/README.md) -1. [Upgrade Tests](../test/integration/consul-container/test/upgrade/README.md) -1. [Remote Debugging Integration Tests](../test/integration/consul-container/test/debugging.md) -1. [Peering Common Topology Tests](../test-integ/peering_commontopo/README.md) - -## Important Directories - -Most top level directories contain Go source code. The directories listed below -contain other important source related to Consul. - -* [ui] contains the source code for the Consul UI. -* [website] contains the source for [consul.io](https://developer.hashicorp.com/). A pull requests - can update the source code and Consul's documentation at the same time. -* [.github] contains the source for our CI and GitHub repository - automation. -* [.changelog] contains markdown files that are used by [hashicorp/go-changelog] to produce the - [CHANGELOG.md]. -* [build-support] contains bash functions and scripts used to automate. - development tasks. Generally these scripts are called from the [Makefile]. -* [grafana] contains the source for a [Grafana dashboard] that can be used to - monitor Consul. - -[ui]: https://github.com/hashicorp/consul/tree/main/ui -[.github]: https://github.com/hashicorp/consul/tree/main/.github -[.changelog]: https://github.com/hashicorp/consul/tree/main/.changelog -[hashicorp/go-changelog]: https://github.com/hashicorp/go-changelog -[CHANGELOG.md]: https://github.com/hashicorp/consul/blob/main/CHANGELOG.md -[build-support]: https://github.com/hashicorp/consul/tree/main/build-support -[Makefile]: https://github.com/hashicorp/consul/tree/main/Makefile -[Grafana dashboard]: https://grafana.com/grafana/dashboards -[grafana]: https://github.com/hashicorp/consul/tree/main/grafana - - -## Contributing to these docs - -This section is meta documentation about contributing to these docs. - -### Diagrams - -The diagrams in these documents are created using the [mermaid-js live editor]. -The [mermaid-js docs] provide a complete reference for how to create and edit -the diagrams. Use the [consul-mermaid-theme.json] (paste it into the Config tab -in the editor) to maintain a consistent Consul style for the diagrams. - -[mermaid-js live editor]: https://mermaid-js.github.io/mermaid-live-editor/edit/ -[mermaid-js docs]: https://mermaid-js.github.io/mermaid/ -[consul-mermaid-theme.json]: ./consul-mermaid-theme.json - diff --git a/docs/acl/README.md b/docs/acl/README.md deleted file mode 100644 index 2be733c3b6c..00000000000 --- a/docs/acl/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# ACL - -This section is a work in progress. - -The ACL subsystem is responsible for authenticating and authorizing access to Consul -operations ([HTTP API], and [RPC]). - -[HTTP API]: ../http-api -[RPC]: ../rpc - -## ACL Entities - -There are many entities in the ACL subsystem. The diagram below shows the relationship -between the entities. - -![Entity Relationship Diagram](./erd.svg) - -[source](./erd.mmd) - -ACL Tokens are at the center of the ACL system. Tokens are associated with a set of -Policies, and Roles. - -AuthMethods, which consist of BindingRules, are a mechanism for creating ACL Tokens from -policies stored in external systems (ex: kubernetes, JWT, or OIDC). - -Roles are a set of policies associated with a named role, and ServiceIdentity and -NodeIdentity are policy templates that are associated with a specific service or node and -can be rendered into a full policy. - -Each Policy contains a set of rules. Each rule relates to a specific resource, and -includes an AccessLevel (read, write, list or deny). - -An ACL Token can be resolved into an Authorizer. The Authorizer is what is used by the -[HTTP API], and [RPC] endpoints to determine if an operation is allowed or forbidden (the -enforcement decision). diff --git a/docs/acl/erd.mmd b/docs/acl/erd.mmd deleted file mode 100644 index 3a9ec446c19..00000000000 --- a/docs/acl/erd.mmd +++ /dev/null @@ -1,33 +0,0 @@ -erDiagram - - Token - Policy - Role - ServiceIdentity - NodeIdentity - AuthMethod - BindingRule - Rule { - string Resource - enum AccessLevel - } - - Policy ||--|{ Rule: grants - Role ||--|{ Policy: includes - Role }|--|{ ServiceIdentity: includes - Role }|--|{ NodeIdentity: includes - - Token }|--|{ Policy: includes - Token }|--|{ Role: includes - Token }|--|{ ServiceIdentity: includes - Token }|--|{ NodeIdentity: includes - - AuthMethod ||--|{ BindingRule: defines - AuthMethod ||--|{ Token: creates - - ServiceIdentity ||--|{ Rule: implies - NodeIdentity ||--|{ Rule: implies - - Token ||--|| Authorizer: "resolves to" - Authorizer ||--|{ EnforcementDecision: produces - diff --git a/docs/acl/erd.svg b/docs/acl/erd.svg deleted file mode 100644 index 47c4f8acaf7..00000000000 --- a/docs/acl/erd.svg +++ /dev/null @@ -1 +0,0 @@ -TokenPolicyRoleServiceIdentityNodeIdentityAuthMethodBindingRuleRulestringResourceenumAccessLevelAuthorizerEnforcementDecisiongrantsincludesincludesincludesincludesincludesincludesincludesdefinescreatesimpliesimpliesresolves toproduces \ No newline at end of file diff --git a/docs/cli/README.md b/docs/cli/README.md deleted file mode 100644 index ad99a01aa0d..00000000000 --- a/docs/cli/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Command-Line Interface (CLI) - -This section is a work in progress. - -The `consul` binary provides a CLI for interacting with the [HTTP API]. Some commands may -also exec other processes or generate data used by Consul (ex: tls certificates). The -`agent` command is responsible for starting the Consul agent. - -The [cli reference] in Consul user documentation has a full reference to all available -commands. - -[HTTP API]: ../http-api -[cli reference]: https://developer.hashicorp.com/commands - -## Code - -The CLI entrypoint is [main.go] and the majority of the source for the CLI is under the -[command] directory. Each subcommand is a separate package under [command]. The CLI uses -[github.com/mitchellh/cli] as a framework, and uses the [flag] package from the stdlib for -command line flags. - - -[command]: https://github.com/hashicorp/consul/tree/main/command -[main.go]: https://github.com/hashicorp/consul/blob/main/main.go -[flag]: https://pkg.go.dev/flag -[github.com/mitchellh/cli]: https://github.com/mitchellh/cli - -## Important notes - -The [cli.Ui] wraps an `io.Writer` for both stdout and stderr. At the time of writing both -`Info` and `Output` go to stdout. Writing `Info` to stdout has been a source of a couple -bugs. To prevent these bugs in the future it is recommended that `Info` should no longer -be used. Instead, send all information messages to stderr by using `Warn`. - - -[cli.Ui]: https://pkg.go.dev/github.com/mitchellh/cli#Ui diff --git a/docs/client-agent/README.md b/docs/client-agent/README.md deleted file mode 100644 index 2376aabd023..00000000000 --- a/docs/client-agent/README.md +++ /dev/null @@ -1,7 +0,0 @@ -# Client Agent - -- agent/cache -- [agent/local](https://github.com/hashicorp/consul/tree/main/agent/local) -- anti-entropy sync in [agent/ae](https://github.com/hashicorp/consul/tree/main/agent/ae) powering the [Anti-Entropy Sync Back](https://developer.hashicorp.com/docs/internals/anti-entropy.html) process to the Consul servers. - -Applications on client nodes use their local agent in client mode to [register services](https://developer.hashicorp.com/api/agent.html) and to discover other services or interact with the key/value store. diff --git a/docs/cluster-federation/README.md b/docs/cluster-federation/README.md deleted file mode 100644 index 964796d648b..00000000000 --- a/docs/cluster-federation/README.md +++ /dev/null @@ -1,4 +0,0 @@ -# Multi-Cluster Federation - -1. [Network Areas](./network-areas) - diff --git a/docs/cluster-federation/network-areas/README.md b/docs/cluster-federation/network-areas/README.md deleted file mode 100644 index 87c0871d09f..00000000000 --- a/docs/cluster-federation/network-areas/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# Network Areas (ENTERPRISE ONLY) - -## Overview - -### Description and Background -Network areas define pairwise gossip pools over the WAN between Consul datacenters. Gossip traffic between Consul servers in an area is done over the server RPC port, and uses TCP connections. This is unlike Consul's primary WAN and LAN pools, which primarily rely on UDP-based probes. - -TCP was used because it allows configuration with TLS certificates. The TLS encryption will be used if [configured](https://developer.hashicorp.com/docs/security/encryption#rpc-encryption-with-tls) and [enabled](https://developer.hashicorp.com/api-docs/operator/area#usetls). Because gossip connects every node in that pool, there is also a connection between servers from the same datacenter. Connections between servers from the same datacenter default to using TLS if it is configured. The overhead of the TCP protocol is limited since there is a small number of servers in an area. Note that the symmetric keys used by Consul's WAN and LAN pools are not used to encrypt traffic from network areas. - -In versions of Consul prior to v1.8, network areas would establish a new TCP connection for every network area message. This was then substituted by connection pooling, where each server will maintain a TCP connection to every server in the network area. However, note that when a server in a version > `v1.8.0` dials a server on an older version, it will fall-back to the old connection-per-message behavior. - - -### Key Components -* Consul Enterprise: - * Network areas are created via Consul Enterprise's APIs and are persisted on server agents. - * Consul servers regularly reconcile the network area gossip pools they are in against the expected network areas registered in the state store. -* Memberlist - * Memberlist manages the list of nodes in the gossip pool by implementing the failure detection mechanism from the [Lifeguard paper](https://arxiv.org/pdf/1707.00788.pdf). -* Serf - * Serf is the interface that drives node health updates in Consul. - * When memberlist updates the status of a member then Serf will send corresponding events to Consul. Based on these events Consul then updates the catalog. - - -### Telemetry - -`consul.area.connections.outgoing` - Tracks outbound network area connections. When both the dialing and dialed servers support pooling then this metric tracks open connections in the pool. When connection pooling is not supported this metric tracks connections opened over time. - - -## Implementation Overview - -### Area creation -Network areas are created with requests to the `/operator/area` HTTP endpoint. The area defined in the request is then commited through RAFT and persisted on Consul servers. - -Every Consul Enterprise server maintains a reconciliation routine where every 30s it will query the list of areas in the state store, and then join or leave gossip pools to reflect that state. - -Joining a network area pool involves: -1. Setting memberlist and Serf configuration. - * Prior to Consul `v1.8.11` and `v1.9.5`, network areas were configured with memberlist's [DefaultWANConfig](https://github.com/hashicorp/memberlist/blob/838073fef1a4e1f6cb702a57a8075304098b1c31/config.go#L315). This was then updated to instead use the server's [gossip_wan](https://developer.hashicorp.com/docs/agent/config/config-files#gossip_wan) configuration, which falls back to the DefaultWANConfig if it was not specified. - * As of Consul `v1.8.11`/`v1.9.5` it is not possible to tune gossip communication on a per-area basis. - -2. Update the server's gossip network, which keeps track of network areas that the server is a part of. This gossip network is also used to dispatch incoming **gossip** connections to handlers for the appropriate area. - -3. Update the server's router, which enables forwarding RPC requests to remote servers in the area. These are cross-datacenters requests made to Consul's public APIs, such as to list service health. - - -### Area gossip - -When a network area is added to a server's gossip network, Consul configures memberlist with a custom transport that implements memberlist's [Transport interface](https://github.com/hashicorp/memberlist/blob/619135cdd9e5dda8c12f8ceef39bdade4f5899b6/transport.go#L28). - -The primary difference from memberlists's default [NetTransport](https://github.com/hashicorp/memberlist/blob/619135cdd9e5dda8c12f8ceef39bdade4f5899b6/net_transport.go#L42), is that the area transport **exclusively** ingests from and writes to TCP connections multiplexed over the RPC port. For example, when writing to an address the area transport will acquire a connection from the pool and then write to that connection. This is unlike `NetTransport`'s `WriteTo` implementation, which sends packets to a UDP listener. - -Since TCP connections are used, establishing a connection is subject to a dial timeout. The dial timeout was initially set to 10s until `v1.8.0`, where the move to connection pooling lowered it to 100ms. This 100ms timeout was insufficient for WAN connections over long distances and later reverted to 10s in versions `v1.8.11`/`v1.9.5`. As of Consul `v1.8.11`, this dial timeout is not configurable and applies to all network areas. - -When a connection is established, inbound gossip requests are handled by the area transport. This handler then passes the request off to memberlist via the [packet](https://github.com/hashicorp/memberlist/blob/838073fef1a4e1f6cb702a57a8075304098b1c31/transport.go#L49) or [stream](https://github.com/hashicorp/memberlist/blob/838073fef1a4e1f6cb702a57a8075304098b1c31/transport.go#L61) channels. diff --git a/docs/cluster-membership/README.md b/docs/cluster-membership/README.md deleted file mode 100644 index 3edb5846eb9..00000000000 --- a/docs/cluster-membership/README.md +++ /dev/null @@ -1,12 +0,0 @@ -# Cluster membership - -This section is a work in progress. It will contain topics like the following: - - - hashicorp/serf - - hashicorp/memberlist - - network coordinates - - consul events - - consul exec - - -Both client and server mode agents participate in a [Gossip Protocol](https://developer.hashicorp.com/docs/internals/gossip.html) which provides two important mechanisms. First, it allows for agents to learn about all the other agents in the cluster, just by joining initially with a single existing member of the cluster. This allows clients to discover new Consul servers. Second, the gossip protocol provides a distributed failure detector, whereby the agents in the cluster randomly probe each other at regular intervals. Because of this failure detector, Consul can run health checks locally on each agent and just sent edge-triggered updates when the state of a health check changes, confident that if the agent dies altogether then the cluster will detect that. This makes Consul's health checking design very scaleable compared to centralized systems with a central polling type of design. diff --git a/docs/config/README.md b/docs/config/README.md deleted file mode 100644 index 1ec0474c5cf..00000000000 --- a/docs/config/README.md +++ /dev/null @@ -1,58 +0,0 @@ -# Agent Configuration - -The [Agent Configuration] is the primary mechanism for configuring Consul. Agent -Configuration also allows for specifying [Config Entries], [Services], and [Checks] that -will be loaded when the agent starts. - -Most configuration comes from [hcl] or `json` files, but some configuration can also be -specified using command line flags, and some can be loaded with [Auto-Config]. - -See also the [checklist for adding a new field] to the configuration. - -[hcl]: https://github.com/hashicorp/hcl/tree/hcl1 -[Agent Configuration]: https://developer.hashicorp.com/docs/agent/config -[checklist for adding a new field]: ./checklist-adding-config-fields.md -[Auto-Config]: #auto-config -[Config Entries]: https://developer.hashicorp.com/docs/agent/config/config-files#config_entries -[Services]: https://developer.hashicorp.com/docs/discovery/services -[Checks]: https://developer.hashicorp.com/docs/discovery/checks - - -## Code - -The Agent Configuration is implemented in [agent/config], and the primary entrypoint is -[Load]. Config loading is performed in phases: - -1. Command line flags are used to create a `config.LoadOpts` and passed to `Load`. -2. `Load` reads all the config files and builds an ordered list of `config.Source`. -3. Each `config.Source` is read to produce a `config.Config`. -4. Each `config.Config` is merged ontop the previous. -5. A `config.RuntimeConfig` is produced from the merged `config.Config` -6. The `config.RuntimeConfig` is validated. -7. Finally a result is returned with the `RuntimeConfig` and any warnings, or an error. - -[agent/config]: https://github.com/hashicorp/consul/tree/main/agent/config -[Load]: https://pkg.go.dev/github.com/hashicorp/consul/agent/config#Load - -If [Auto-Config] is enabled, when it receives the config from the server, the -entire process is repeated a second time with the addition config provided as another -`config.Source`. - -Default values can be specified in one of the [default sources] or set when -converting from `Config` to `RuntimeConfig` in [builder.build]. Hopefully in the future we -should remove one of those ways of setting default values. - -[default sources]: https://github.com/hashicorp/consul/blob/main/agent/config/default.go -[builder.build]: https://github.com/hashicorp/consul/blob/main/agent/config/builder.go - -## Auto-Config - -Auto-Config is enabled by the [auto_config] field in an Agent Configuration file. It is -implemented in a couple packages. - -* the server RPC endpoint is in [agent/consul/auto_config_endpoint.go] -* the client that receives and applies the config is implemented in [agent/auto-config] - -[auto_config]: https://developer.hashicorp.com/docs/agent/config/config-files#auto_config -[agent/consul/auto_config_endpoint.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/auto_config_endpoint.go -[agent/auto-config]: https://github.com/hashicorp/consul/tree/main/agent/auto-config diff --git a/docs/consul-mermaid-theme.json b/docs/consul-mermaid-theme.json deleted file mode 100644 index c3823254b1f..00000000000 --- a/docs/consul-mermaid-theme.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "theme": "default", - "themeCSS": ".node rect, .er.entityBox { fill: rgb(220, 71, 125); stroke-width: 1; stroke: black; } .node .label { color: white; }; .cluster rect { fill: #f0f0f0; stroke-width: 1px; stroke: #333}; .edgeLabel { background-color: #f0f0f0; }; .er.entityBox + .er.entityLabel { fill: white }; .er.attributeBoxEven, .er.attributeBoxOdd { fill: #fff; stroke: #777 }" -} diff --git a/docs/debug/pprof.md b/docs/debug/pprof.md deleted file mode 100644 index 75a2eb32cbb..00000000000 --- a/docs/debug/pprof.md +++ /dev/null @@ -1,128 +0,0 @@ -# pprof - -> A Profile is a collection of stack traces showing the call sequences that led to instances of a particular event, such as allocation. - -## Installation -`go install github.com/google/pprof@latest` - -### CPU Profiles -* Every 10ms the CPU profiler interrupts a thread in the program and records the stack traces of **running** goroutines. - * It does **not** track time spent sleeping or waiting for I/O. -* The duration of a sample is assumed to be 10ms, so the seconds spent in a function is calculated as: `(num_samples * 10ms)/1000` - -### Heap Profiles -* Aims to record the stack trace for every 512KB allocated, at the point of the allocation. - * Tracks allocations and when profiled allocations are freed. With both of these data points pprof can calculate allocations and memory in use. -* `alloc_objects` and `alloc_space` refer to allocations since the start of the profiling period. - * Helpful for tracking functions that produce a lot of allocations. -* `inuse_objects` and `inuse_space` refers to allocations that have not been freed. - * `inuse_objects = allocations - frees` - * Helpful for tracking sources of high memory usage. - * May not line up with OS reported memory usage! The profiler only tracks heap memory usage, and there are also cases where the Go GC will not release free heap memory to the OS. -* When allocations are made, the heap sampler makes a decision about whether to sample the allocation or not. If it decides to sample it, it records the stack trace, counts the allocation, and counts the number of bytes allocated. - * When an object's memory is freed, the heap profiler checks whether that allocation was sampled, and if it was, it counts the bytes freed. - -### Goroutine Profiles -* Shows the number of times each stack trace was seen across goroutines. - -## Common commands -* Open a profile in the terminal: -`go tool pprof profile.prof` -`go tool pprof heap.prof` -`go tool pprof goroutine.prof` - -* Open a profile in a web browser: -`go tool pprof -http=:8080 profile.prof` - -* If the correct source isn't detected automatically, specify the location of the source associated with the profile: -`go tool pprof -http=:9090 -source_path=/Users/freddy/go/src/github.com/hashicorp/consul-enterprise profile.prof` - -Useful in annotated `Source` view which shows time on a line-by-line basis. - -**Important:** Ensure that the source code matches the version of the binary! The source view relies on line numbers for its annotations. - -* Compare two profiles: -`go tool pprof -http=:8080 -base before/profile.prof after/profile.prof` -This comparison will subtract the `-base` profile from the given profile. In this case, "before" is subtracted from "after". - -* Useful commands when profile is opened in terminal: - * `top` lists the top 10 nodes by value - * `list ` lists the top matches to the pattern - -## Graph view -* **Node Values:** - * Package - * Function - * Flat value: the value in the function itself. - * Cumulative value: the sum of the flat value and all its descendants. - * Percentage of flat and cumulative values relative to total samples. Total sample time is visible with `top` command in terminal view. - -Example: -``` -func foo(){ - a() // step 1 takes 1s - do something directly. // step 2 takes 3s - b() // step 3 takes 1s -} -``` - -`flat` is the time spent on step 2 (3s), while `cum` is time spent on steps 1 to 3. - -* **Path Value** - * The cumulative value of the following node. - -* **Node Color**: - * large positive cumulative values are red. - * cumulative values close to zero are grey. - * large negative cumulative values are green; negative values are most likely to appear during profile comparison. - -* **Node Font Size**: - * larger font size means larger absolute flat values. - * smaller font size means smaller absolute flat values. - -* **Edge Weight**: - * thicker edges indicate more resources were used along that path. - * thinner edges indicate fewer resources were used along that path. - -* **Edge Color**: - * large positive values are red. - * large negative values are green. - * values close to zero are grey. - -* **Dashed Edges**: some locations between the two connected locations were removed. - -* **Solid Edges**: one location directly calls the other. - -* **"(inline)" Edge Marker**: the call has been inlined into the caller. More on inlining: [Inlining optimisations in Go | Dave Cheney](https://dave.cheney.net/2020/04/25/inlining-optimisations-in-go) - -Example graph: -![Nodes](./pprof_cpu_nodes.png) - -* 20.11s is spent doing direct work in `(*Store).Services()` -* 186.88s is spent in this function **and** its descendants -* `(*Store).Services()` has both large `flat` and `cumulative` values, so the font is large and the box is red. -* The edges to `mapassign_faststr` and `(radixIterator).Next()` are solid and red because these are direct calls with large positive values. - - -## Flame graph view -A collection of stack traces, where each stack is a column of boxes, and each box represents a function. - -Functions at the top of the flame graph are parents of functions below. - -The width of each box is proportional to the number of times it was observed during sampling. - -Mouse-over boxes shows `cum` value and percentage, while clicking on boxes lets you zoom into their stack traces. - -### Note -* The background color is **not** significant. -* Sibling boxes are not necessarily in chronological order. - - -## References: -* [Diagnostics - The Go Programming Language](https://go.dev/doc/diagnostics) -* [Profiling Go programs with pprof](https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/) -* [pprof/README.md](https://github.com/google/pprof/blob/master/doc/README.md) -* [GitHub - DataDog/go-profiler-notes: felixge's notes on the various go profiling methods that are available.](https://github.com/DataDog/go-profiler-notes) -* [The Flame Graph - ACM Queue](https://queue.acm.org/detail.cfm?id=2927301) -* [High Performance Go Workshop](https://dave.cheney.net/high-performance-go-workshop/dotgo-paris.html#pprof) -* [Pprof and golang - how to interpret a results?](https://stackoverflow.com/a/56882137) \ No newline at end of file diff --git a/docs/debug/pprof_cpu_nodes.png b/docs/debug/pprof_cpu_nodes.png deleted file mode 100644 index 77a587d494e..00000000000 Binary files a/docs/debug/pprof_cpu_nodes.png and /dev/null differ diff --git a/docs/faq.md b/docs/faq.md deleted file mode 100644 index 890acefcfdd..00000000000 --- a/docs/faq.md +++ /dev/null @@ -1,21 +0,0 @@ -# FAQ - -This section addresses some frequently asked questions about Consul's architecture. - -### How does eventually-consistent gossip relate to the Raft consensus protocol? - -When you query Consul for information about a service, such as via the [DNS interface](https://developer.hashicorp.com/docs/discovery/dns), the agent will always make an internal RPC request to a Consul server that will query the consistent state store. Even though an agent might learn that another agent is down via gossip, that won't be reflected in service discovery until the current Raft leader server perceives that through gossip and updates the catalog using Raft. You can see an example of where these layers are plumbed together here - https://github.com/hashicorp/consul/blob/v1.0.5/agent/consul/leader.go#L559-L602. - -## Why does a blocking query sometimes return with identical results? - -Consul's [blocking queries](https://developer.hashicorp.com/api/index.html#blocking-queries) make a best-effort attempt to wait for new information, but they may return the same results as the initial query under some circumstances. First, queries are limited to 10 minutes max, so if they time out they will return. Second, due to Consul's prefix-based internal immutable radix tree indexing, there may be modifications to higher-level nodes in the radix tree that cause spurious wakeups. In particular, waiting on things that do not exist is not very efficient, but not very expensive for Consul to serve, so we opted to keep the code complexity low and not try to optimize for that case. You can see the common handler that implements the blocking query logic here - https://github.com/hashicorp/consul/blob/v1.0.5/agent/consul/rpc.go#L361-L439. For more on the immutable radix tree implementation, see https://github.com/hashicorp/go-immutable-radix/ and https://github.com/hashicorp/go-memdb, and the general support for "watches". - -### Do the client agents store any key/value entries? - -No. These are always fetched via an internal RPC request to a Consul server. The agent doesn't do any caching, and if you want to be able to fetch these values even if there's no cluster leader, then you can use a more relaxed [consistency mode](https://developer.hashicorp.com/api/index.html#consistency-modes). You can see an example where the `/v1/kv/` HTTP endpoint on the agent makes an internal RPC call here - https://github.com/hashicorp/consul/blob/v1.0.5/agent/kvs_endpoint.go#L56-L90. - -### I don't want to run a Consul agent on every node, can I just run servers with a load balancer in front? - -We strongly recommend running the Consul agent on each node in a cluster. Even the key/value store benefits from having agents on each node. For example, when you lock a key it's done through a [session](https://developer.hashicorp.com/docs/internals/sessions.html), which has a lifetime that's by default tied to the health of the agent as determined by Consul's gossip-based distributed failure detector. If the agent dies, the session will be released automatically, allowing some other process to quickly see that and obtain the lock without having to wait for an open-ended TTL to expire. If you are using Consul's service discovery features, the local agent runs the health checks for each service registered on that node and only needs to send edge-triggered updates to the Consul servers (because gossip will determine if the agent itself dies). Most attempts to avoid running an agent on each node will face solving issues that are already solved by Consul's design if the agent is deployed as intended. - -For cases where you really cannot run an agent alongside a service, such as for monitoring an [external service](https://developer.hashicorp.com/docs/guides/external.html), there's a companion project called the [Consul External Service Monitor](https://github.com/hashicorp/consul-esm) that may help. diff --git a/docs/http-api/README.md b/docs/http-api/README.md deleted file mode 100644 index 223c984b646..00000000000 --- a/docs/http-api/README.md +++ /dev/null @@ -1,9 +0,0 @@ -# HTTP API - -Work in progress. This section will eventually contain docs about: - -* the HTTP "framework" -* HTTP endpoints -* the [api](https://github.com/hashicorp/consul/tree/main/api) client library - the `api` package - provides an official Go API client for Consul, which is also used by Consul's - [CLI](https://developer.hashicorp.com/docs/commands/index.html) commands to communicate with the local Consul agent. diff --git a/docs/overview.mmd b/docs/overview.mmd deleted file mode 100644 index f55f94795a6..00000000000 --- a/docs/overview.mmd +++ /dev/null @@ -1,30 +0,0 @@ -graph TD - - ServiceMesh[Service Mesh] - ServiceDiscovery[Service Discovery] - ClusterMembership[Cluster Membership] - KV[Key/Value Store] - MultiClusterFederation[Multi-Cluster Federation] - - ACL - AgentConfiguration[Agent Configuration] - ClientAgent[Client Agent] - RPC - ClusterPersistence[Cluster Persistence] - CLI - HTTPAPI[HTTP API] - - CLI --> HTTPAPI - HTTPAPI --> ClientAgent - HTTPAPI --> ACL - - AgentConfiguration --> ClientAgent - ClientAgent --> RPC - ClientAgent --> ACL - RPC --> ClusterPersistence - RPC --> ACL - - MultiClusterFederation --> ClusterMembership - MultiClusterFederation --> RPC - ServiceMesh --> ServiceDiscovery - diff --git a/docs/overview.svg b/docs/overview.svg deleted file mode 100644 index 3de2f78fa69..00000000000 --- a/docs/overview.svg +++ /dev/null @@ -1 +0,0 @@ -
Service Mesh
Service Discovery
Cluster Membership
Key/Value Store
Multi-Cluster Federation
ACL
Agent Configuration
Client Agent
RPC
Cluster Persistence
CLI
HTTP API
 \ No newline at end of file diff --git a/docs/persistence/README.md b/docs/persistence/README.md deleted file mode 100644 index 6ed808b95d8..00000000000 --- a/docs/persistence/README.md +++ /dev/null @@ -1,121 +0,0 @@ -# Cluster Persistence - -> **Note** -> While the content of this document is still accurate, it doesn't cover the new -> generic resource-oriented storage layer introduced in Consul 1.16. Please see -> [Resources](../v2-architecture/controller-architecture) for more information. - -The cluser persistence subsystem runs entirely in Server Agents. It handles both read and -write requests from the [RPC] subsystem. See the [Consul Architecture Guide] for an -introduction to the Consul deployment architecture and the [Consensus Protocol] used by -the cluster persistence subsystem. - -[RPC]: ../rpc -[Consul Architecture Guide]: https://developer.hashicorp.com/docs/architecture -[Consensus Protocol]: https://developer.hashicorp.com/docs/architecture/consensus - - -![Overview](./overview.svg) - -[source](./overview.mmd) - - -## Raft and FSM - -[hashicorp/raft] is at the core of cluster persistence. Raft requires an [FSM], a -finite-state machine implementation, to persist state changes. The Consul FSM is -implemented in [agent/consul/fsm] as a set of commands. - -[FSM]: https://pkg.go.dev/github.com/hashicorp/raft#FSM -[hashicorp/raft]: https://github.com/hashicorp/raft -[agent/consul/fsm]: https://github.com/hashicorp/consul/tree/main/agent/consul/fsm - -Raft also requires a [LogStore] to persist logs to disk. Consul uses [hashicorp/raft-boltdb] -which implements [LogStore] using [boltdb]. In the near future we should be updating to -use [bbolt]. - - -[LogStore]: https://pkg.go.dev/github.com/hashicorp/raft#LogStore -[hashicorp/raft-boltdb]: https://github.com/hashicorp/raft-boltdb -[boltdb]: https://github.com/boltdb/bolt -[bbolt]: https://github.com/etcd-io/bbolt - -See [diagrams](#diagrams) below for more details on the interaction. - -## State Store - -Consul stores the full state of the cluster in memory using the state store. The state store is -implemented in [agent/consul/state] and uses [hashicorp/go-memdb] to maintain indexes of -data stored in a set of tables. The main entrypoint to the state store is [NewStateStore]. - -[agent/consul/state]: https://github.com/hashicorp/consul/tree/main/agent/consul/state -[hashicorp/go-memdb]: https://github.com/hashicorp/go-memdb -[NewStateStore]: https://github.com/hashicorp/consul/blob/main/agent/consul/state/state_store.go - -### Tables, Schemas, and Indexes - -The state store is organized as a set of tables, and each table has a set of indexes. -`newDBSchema` in [schema.go] shows the full list of tables, and each schema function shows -the full list of indexes. - -[schema.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/state/schema.go - -There are two styles for defining table indexes. The original style uses generic indexer -implementations from [hashicorp/go-memdb] (ex: `StringFieldIndex`). These indexes use -[reflect] to find values for an index. These generic indexers work well when the index -value is a single value available directly from the struct field, and there are no -ce/enterprise differences. - -The second style of indexers are custom indexers implemented using only functions and -based on the types defined in [indexer.go]. This style of index works well when the index -value is a value derived from one or multiple fields, or when there are ce/enterprise -differences between the indexes. - -[reflect]: https://golang.org/pkg/reflect/ -[indexer.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/state/indexer.go - - -## Snapshot and Restore - -Snapshots are the primary mechanism used to backup the data stored by cluster persistence. -If all Consul servers fail, a snapshot can be used to restore the cluster back -to its previous state. - -Note that there are two different snapshot and restore concepts that exist at different -layers. First there is the `Snapshot` and `Restore` methods on the raft [FSM] interface, -that Consul must implement. These methods are implemented as mostly passthrough to the -state store. These methods may be called internally by raft to perform log compaction -(snapshot) or to bootstrap a new follower (restore). Consul implements snapshot and -restore using the `Snapshot` and `Restore` types in [agent/consul/state]. - -Snapshot and restore also exist as actions that a user may perform. There are [CLI] -commands, [HTTP API] endpoints, and [RPC] endpoints that allow a user to capture an -archive which contains a snapshot of the state, and restore that state to a running -cluster. The [consul/snapshot] package provides some of the logic for creating and reading -the snapshot archives for users. See [commands/snapshot] for a reference to these user -facing operations. - -[CLI]: ../cli -[HTTP API]: ../http-api -[commands/snapshot]: https://developer.hashicorp.com/commands/snapshot -[consul/snapshot]: https://github.com/hashicorp/consul/tree/main/snapshot - -Finally, there is also a [snapshot agent] (enterprise only) that uses the snapshot API -endpoints to periodically capture a snapshot, and optionally send it somewhere for -storage. - -[snapshot agent]: https://developer.hashicorp.com/commands/snapshot/agent - -## Raft Autopilot - -[hashicorp/raft-autopilot] is used by Consul to automate some parts of the upgrade process. - - -[hashicorp/raft-autopilot]: https://github.com/hashicorp/raft-autopilot - -## Diagrams -### High-level life of a write -![Overview](./write-overview.png) - -### Deep-dive into write through Raft -![Deep dive](./write-deep-dive.png) \ No newline at end of file diff --git a/docs/persistence/overview.mmd b/docs/persistence/overview.mmd deleted file mode 100644 index ab93188bfad..00000000000 --- a/docs/persistence/overview.mmd +++ /dev/null @@ -1,34 +0,0 @@ -graph TB - - requestLeader[request] --> RPCLeader - requestFollower[request] --> RPCFollower - - class requestLeader,requestFollower req; - classDef req fill:transparent,color:#000,stroke-width:1; - - subgraph Leader - RPCLeader[RPC] - RaftLeader[Raft] - StateStoreLeader[State Store] - FSMLeader[FSM] - end - - RPCLeader -->|write| RaftLeader - RPCLeader -->|read| StateStoreLeader - RaftLeader ---> FSMLeader - FSMLeader --> StateStoreLeader - - subgraph Follower - RPCFollower[RPC] - RaftFollower[Raft] - StateStoreFollower[State Store] - FSMFollower[FSM] - end - - RaftLeader <-.->|consensus and replication| RaftFollower - - RPCFollower -->|forward write to leader| RPCLeader - RPCFollower -->|read| StateStoreFollower - RaftFollower --> FSMFollower - FSMFollower --> StateStoreFollower - diff --git a/docs/persistence/overview.svg b/docs/persistence/overview.svg deleted file mode 100644 index 346c63f818a..00000000000 --- a/docs/persistence/overview.svg +++ /dev/null @@ -1 +0,0 @@ -
Follower
Leader
write
read
consensus and replication
forward write to leader
read
RPC
Raft
State Store
FSM
RPC
Raft
State Store
FSM
request
request
 \ No newline at end of file diff --git a/docs/persistence/write-deep-dive.png b/docs/persistence/write-deep-dive.png deleted file mode 100644 index b3872fb6551..00000000000 Binary files a/docs/persistence/write-deep-dive.png and /dev/null differ diff --git a/docs/persistence/write-overview.png b/docs/persistence/write-overview.png deleted file mode 100644 index e7238c13a20..00000000000 Binary files a/docs/persistence/write-overview.png and /dev/null differ diff --git a/docs/rpc/README.md b/docs/rpc/README.md deleted file mode 100644 index 647b2a90c73..00000000000 --- a/docs/rpc/README.md +++ /dev/null @@ -1,92 +0,0 @@ -# RPC - -Consul uses two RPC systems for communication between components within the -cluster and with other clients such as Envoy: [gRPC](https://grpc.io/) -and Go's [`net/rpc`](https://pkg.go.dev/net/rpc) package. - -Communication between client agents and servers uses a mix of both gRPC and -`net/rpc`. Generally, gRPC is preferred because it supports modern features -such as context deadlines/cancellation, streaming, and middleware - but Consul -has been around for a while so the majority of RPC endpoints still use `net/rpc`. - -## Multiplexed "Server" Port - -Most in-cluster communication happens over the multiplexed "server" TCP port -(default: 8300). Consul servers implement a custom protocol for serving -different kinds of traffic on the same port, whereby the first byte sent -indicates the protocol (e.g. gRPC, `net/rpc`, Raft). - -Servers also implement [TLS ALPN](https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation) -on this port, for situations where wrapping the real protocol with a byte prefix -isn't practical (e.g. cross-DC traffic over mesh gateways). - -The diagram below shows all the possible routing flows: - -[server port]: https://developer.hashicorp.com/docs/reference/agent/configuration-files/general#server_rpc_port - -![RPC Routing](./routing.svg) - -[source](./routing.mmd) - -The main entrypoint to connection routing is `handleConn` in [agent/consul/rpc.go]. - -[agent/consul/rpc.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/rpc.go - -### Development - -Multiplexing several protocols over a single server port helps to reduce our -network requirements, but also makes interacting with Consul using local -development tools such as [grpcurl] difficult. - -[grpcurl]: https://github.com/fullstorydev/grpcurl - -You can get a "plain" TCP connection to the gRPC server using this proxy script: - -``` -$ go run tools/internal-grpc-proxy/main.go localhost:8300 -Proxying connections to Consul's internal gRPC server -Use this address: 127.0.0.1:64077 -``` - -Pass the returned proxy address to your tool of choice. - -## Private vs Public vs Internal vs External -When working on Consul's gRPC endpoints you may notice we use private/public and -internal/external slightly differently. - -Private and public refer to whether an API is suitable for consumption by -clients other than Consul's core components. - -Private gRPC APIs are defined in the `proto` directory, and should only be used -by Consul servers and agents. Public gRPC APIs are defined in the `proto-public` -directory and may be used by 3rd-party applications. - -Internal and external refer to how the gRPC APIs are exposed. - -Internal gRPC APIs are exposed on the multiplexed "server" port, whereas -external APIs are exposed on a dedicated gRPC port (default: 8502). - -The reason for this differentiation is that some private APIs are exposed on the -external port, such as peer streaming/replication; this API isn't (yet) suitable -for consumption by 3rd-party applications but must be accessible from outside -the cluster, and present a TLS certificate signed by a public CA, which the -multiplexed port cannot. - -## RPC Endpoints - -This section is a work in progress, it will eventually cover topics like: - -- net/rpc - (in the stdlib) -- new grpc endpoints -- [Streaming](./streaming) -- [agent/structs](https://github.com/hashicorp/consul/tree/main/agent/structs) - contains definitions of all the internal RPC protocol request and response structures. - - -## RPC connections and load balancing - -This section is a work in progress, it will eventually cover topics like: - -Routing RPC request to Consul servers and for connection pooling. - -- [agent/router](https://github.com/hashicorp/consul/tree/main/agent/router) -- [agent/pool](https://github.com/hashicorp/consul/tree/main/agent/pool) diff --git a/docs/rpc/routing.mmd b/docs/rpc/routing.mmd deleted file mode 100644 index bed74c02400..00000000000 --- a/docs/rpc/routing.mmd +++ /dev/null @@ -1,33 +0,0 @@ -graph LR - - handleConn - - handleConn -->|RPCConsul| handleConsulConn - handleConn -->|RPCRaft| raftLayer - handleConn -->|RPCTLS| handleConn - handleConn -->|RPCMultiplexV2| handleMultiplexV2 - handleConn -->|RPCSnapshot| handleSnapshotConn - handleConn -->|RPCTLSInsecure| handleInsecureConn - handleConn -->|RPCGossip| handleGossipConn - - handleConsulConn --> RPCServer - handleMultiplexV2 --> handleConsulConn - - %% new after 1.6.9 - - handleConn -->|PeekForTLS| handleNativeTLS - - handleNativeTLS -->|ALPN_RPCConsul| handleConsulConn - handleNativeTLS -->|ALPN_RPCRaft| raftLayer - handleNativeTLS -->|ALPN_RPCMultiplexV2| handleMultiplexV2 - handleNativeTLS -->|ALPN_RPCSnapshot| handleSnapshotConn - handleNativeTLS -->|ALPN_RPCGRPC| grpcHandler - handleNativeTLS -->|ALPN_WANGossipPacket| handleWANGossipPacket - handleNativeTLS -->|ALPN_WANGossipStream | handleWANGossipStream - handleNativeTLS -->|ALPN_RPCGossip| handleGossipConn - - handleMultiplexV2 -->|RPCGossip| handleGossipConn - handleConn -->|RPCGRPC| grpcHandler - - - diff --git a/docs/rpc/routing.svg b/docs/rpc/routing.svg deleted file mode 100644 index 0a63bd5d777..00000000000 --- a/docs/rpc/routing.svg +++ /dev/null @@ -1 +0,0 @@ -
RPCConsul
RPCRaft
RPCTLS
RPCMultiplexV2
RPCSnapshot
RPCTLSInsecure
RPCGossip
PeekForTLS
ALPN_RPCConsul
ALPN_RPCRaft
ALPN_RPCMultiplexV2
ALPN_RPCSnapshot
ALPN_RPCGRPC
ALPN_WANGossipPacket
ALPN_WANGossipStream
ALPN_RPCGossip
RPCGossip
RPCGRPC
handleConn
handleConsulConn
raftLayer
handleMultiplexV2
handleSnapshotConn
handleInsecureConn
handleGossipConn
RPCServer
handleNativeTLS
grpcHandler
handleWANGossipPacket
handleWANGossipStream
 \ No newline at end of file diff --git a/docs/rpc/streaming/README.md b/docs/rpc/streaming/README.md deleted file mode 100644 index dd6873cde8c..00000000000 --- a/docs/rpc/streaming/README.md +++ /dev/null @@ -1,100 +0,0 @@ - -# Event Streaming - -Event streaming is a new asynchronous RPC mechanism that is being added to Consul. Instead -of synchronous blocking RPC calls (long polling) to fetch data when it changes, streaming -sends events as they occur, and the client maintains a materialized view of the events. - -At the time of writing only the service health endpoint uses streaming, but more endpoints -will be added in the future. - -See [adding a topic](./adding-a-topic.md) for a guide on adding new topics to streaming. - -## Overview - -The diagram below shows the components that are used in streaming, and how they fit into -the rest of Consul. - -![Streaming Overview](./overview.svg) - -[source](./overview.mmd) - -Read requests are received either from the HTTP API or from a DNS request. They use -[rpcclient/health.Health] -to query the cache. The [StreamingHealthServices cache-type] uses a [materialized view] -to manage subscriptions and store the aggregated events. On the server, the -[SubscribeEndpoint] subscribes and receives events from [EventPublisher]. - -Writes will likely enter the system through the client as well, but to make the diagram -less complicated the write flow starts when it is received by the RPC endpoint. The -endpoint calls raft.Apply, which if successful will save the new data in the state.Store. -When the [state.Store commits] it produces an event which is managed by the [EventPublisher] -and sent to any active subscriptions. - -[rpcclient/health.Health]: https://github.com/hashicorp/consul/blob/main/agent/rpcclient/health/health.go -[StreamingHealthServices cache-type]: https://github.com/hashicorp/consul/blob/main/agent/cache-types/streaming_health_services.go -[materialized view]: https://github.com/hashicorp/consul/blob/main/agent/submatview/materializer.go -[SubscribeEndpoint]: https://github.com/hashicorp/consul/blob/main/agent/grpc-internal/services/subscribe/subscribe.go -[EventPublisher]: https://github.com/hashicorp/consul/blob/main/agent/consul/stream/event_publisher.go -[state.Store commits]: https://github.com/hashicorp/consul/blob/main/agent/consul/state/memdb.go - - -## Event Publisher - -The [EventPublisher] is at the core of streaming. It receives published events, and -subscription requests, and forwards events to the appropriate subscriptions. The diagram -below illustrates how events are stored by the [EventPublisher]. - -![Event Publisher layout](./event-publisher-layout.svg) - -[source](./event-publisher-layout.mmd) - -When a new subscription is created it will create a snapshot of the events required to -reflect the current state. This snapshot is cached by the [EventPublisher] so that other -subscriptions can re-use the snapshot without having to recreate it. - -The snapshot always points at the first item in the linked list of events. A subscription -will initially point at the first item, but the pointer advances each time -`Subscribe.Next` is called. The topic buffers in the EventPublisher always point at the -latest item in the linked list, so that new events can be appended to the buffer. - -When a snapshot cache TTL expires, the snapshot is removed. If there are no other -subscriptions holding a reference to those items, the items will be garbage collected by -the Go runtime. This setup allows EventPublisher to keep some events around for a short -period of time, without any hard coded limit on the number of events to cache. - - -## Subscription events - -A subscription provides a stream of events on a single topic. Most of the events contain -data for a change in state, but there are a few special "framing" events that are used to -communicate something to the client. The diagram below helps illustrate the logic in -`EventPublisher.Subscribe` and the [materialized view]. - - -![Framing events](./framing-events.svg) - -[source](./framing-events.mmd) - - -Events in the `Snapshot` contain the same data as those in the `EventStream`, the only -difference is that events in the `Snapshot` indicate the current state not a change in -state. - -`NewSnapshotToFollow` is a framing event that indicates to the client that their existing -view is out of date. They must reset their view and prepare to receive a new snapshot. - -`EndOfSnapshot` indicates to the client that the snapshot is complete. Any future events -will be changes in state. - - -## Event filtering - -As events pass through the system from the `state.Store` to the client they are grouped -and filtered along the way. The diagram below helps illustrate where each of the grouping -and filtering happens. - - -![event filtering](./event-filtering.svg) - -[source](./event-filtering.mmd) diff --git a/docs/rpc/streaming/adding-a-topic.md b/docs/rpc/streaming/adding-a-topic.md deleted file mode 100644 index 7f474c1f4d4..00000000000 --- a/docs/rpc/streaming/adding-a-topic.md +++ /dev/null @@ -1,35 +0,0 @@ - -# Adding a new topic to streaming - -This document is a guide for adding a new streaming topic. - -1. Add the name of the topic to the [proto/pbsubscribe/subscribe.proto Topic enum][1]. - Run `make proto` to generate the Go code from the protobuf. -2. Add a `FromChanges` function to the list of change functions in - [agent/consul/state.processDBChanges][2]. The `FromChanges` function should examine the - list of `Changes` and return a list of events that subscriptions would need to update - their view. -3. Add a snapshot function to [agent/consul/state.newSnapshotHandlers][3]. The snapshot - function should produce a set of events to create the initial state of the view. - Generally these are all "create" events. -4. Add a new `Payload` type, similar to [agent/consul/state.EventPayloadCheckServiceNode][6]. - This type will be used in the `Payload` field of the event. -5. Create the protobuf for the payload, and add the `Payload` type to the `oneof` in - [proto/pbsubscrube/subscribe.proto Event.Payload][7]. This may require creating other - protobuf types as well, to encode anything in the payload. Run `make proto` to generate - the Go code from the protobuf. -6. Add another case to [agent/rpc/subscribe.setPayload][8] to convert from the Payload - type in `state`, to the protobuf type. This may require either writing or generating a - function to convert between the types. -7. Add a new cache-type that uses [agent/submatview.Materializer][4] similar to - [agent/cache-types/streaming_health_services.go][5]. - - -[1]: https://github.com/hashicorp/consul/blob/v1.9.4/proto/pbsubscribe/subscribe.proto#L37-L45 -[2]: https://github.com/hashicorp/consul/blob/v1.9.4/agent/consul/state/memdb.go#L188-L192 -[3]: https://github.com/hashicorp/consul/blob/v1.9.4/agent/consul/state/memdb.go#L205-L209 -[4]: https://github.com/hashicorp/consul/blob/v1.9.4/agent/submatview/materializer.go#L76 -[5]: https://github.com/hashicorp/consul/blob/v1.9.4/agent/cache-types/streaming_health_services.go -[6]: https://github.com/hashicorp/consul/blob/v1.9.4/agent/consul/state/catalog_events.go#L12-L46 -[7]: https://github.com/hashicorp/consul/blob/v1.9.4/proto/pbsubscribe/subscribe.proto#L95-L117 -[8]: https://github.com/hashicorp/consul/blob/v1.9.4/agent/rpc/subscribe/subscribe.go#L161-L168 diff --git a/docs/rpc/streaming/event-filtering.mmd b/docs/rpc/streaming/event-filtering.mmd deleted file mode 100644 index 620698c733a..00000000000 --- a/docs/rpc/streaming/event-filtering.mmd +++ /dev/null @@ -1,11 +0,0 @@ -graph TD - - state.Store -->|events in different topics| EventPublisher.Publish - EventPublisher.Publish -->|group by topic| EventPublisher.topicBuffer - - EventPublisher.topicBuffer --> Subscription - Subscription -->|filter by key and namespace| SubscribeEndpoint - SubscribeEndpoint -->|"filter by auth (acl token)"| ProtobufEvents[/ grpc /] - ProtobufEvents -->|filter with bexpr| MaterializedView - MaterializedView --> HTTPEndpoint - diff --git a/docs/rpc/streaming/event-filtering.svg b/docs/rpc/streaming/event-filtering.svg deleted file mode 100644 index ee6e5f19814..00000000000 --- a/docs/rpc/streaming/event-filtering.svg +++ /dev/null @@ -1 +0,0 @@ -
events in different topics
group by topic
filter by key and namespace
filter by auth (acl token)
filter with bexpr
state.Store
EventPublisher.Publish
EventPublisher.topicBuffer
Subscription
SubscribeEndpoint
grpc
MaterializedView
HTTPEndpoint
 \ No newline at end of file diff --git a/docs/rpc/streaming/event-publisher-layout.mmd b/docs/rpc/streaming/event-publisher-layout.mmd deleted file mode 100644 index b2986aae746..00000000000 --- a/docs/rpc/streaming/event-publisher-layout.mmd +++ /dev/null @@ -1,36 +0,0 @@ -graph TB - - subgraph ep[ ] - EventPublisher - subscriptions - snapshots - topicBuffers - end - - EventPublisher --> snapshots & subscriptions & topicBuffers - - Subscription - Snapshot - Item0 - Item1 - Item2 - Item3 - Item4 - - topicBuffers ----->|head| Item4 - subscriptions --> Subscription - - snapshots --> Snapshot - - Subscription -->|next| Item0 - Item0 --> Item1 - Item1 --> Item2 - Item2 --> Item3 - Item3 --> Item4 - Snapshot -->|first| Item0 - - Subscription -..->|next| Item1 - Subscription -..->|next| Item2 - Subscription -..->|next| Item3 - Subscription -..->|next| Item4 - diff --git a/docs/rpc/streaming/event-publisher-layout.svg b/docs/rpc/streaming/event-publisher-layout.svg deleted file mode 100644 index 2f16c26d00b..00000000000 --- a/docs/rpc/streaming/event-publisher-layout.svg +++ /dev/null @@ -1 +0,0 @@ -
head
next
first
next
next
next
next
EventPublisher
subscriptions
snapshots
topicBuffers
Subscription
Snapshot
Item0
Item1
Item2
Item3
Item4
 \ No newline at end of file diff --git a/docs/rpc/streaming/framing-events.mmd b/docs/rpc/streaming/framing-events.mmd deleted file mode 100644 index fd091cb045d..00000000000 --- a/docs/rpc/streaming/framing-events.mmd +++ /dev/null @@ -1,17 +0,0 @@ -graph TD - - SubscribeIndex0[Subscribe, index = 0, no snapshot] - SubscribeIndexNot0[Subscribe, index > 0, with snapshot] - - SubscribeIndex0 --->|if events in topic| Snapshot - Snapshot --> EndOfSnapshot - SubscribeIndex0 ------->|no events in topic| EndOfSnapshot - EndOfSnapshot --> EventStream - - SubscribeIndexNot0 -->|if index != TopicBuffer.Head| NewSnapshotToFollow - NewSnapshotToFollow ---> Snapshot - - SubscribeIndexNot0 -->|if index == TopicBuffer.Head| EventStream - - class EndOfSnapshot,NewSnapshotToFollow framing - classDef framing fill:#FFD700,stroke:#333 diff --git a/docs/rpc/streaming/framing-events.svg b/docs/rpc/streaming/framing-events.svg deleted file mode 100644 index 885b8719467..00000000000 --- a/docs/rpc/streaming/framing-events.svg +++ /dev/null @@ -1 +0,0 @@ -
if events in topic
no events in topic
if index != TopicBuffer.Head
if index == TopicBuffer.Head
Subscribe, index = 0, no snapshot
Subscribe, index > 0, with snapshot
Snapshot
EndOfSnapshot
EventStream
NewSnapshotToFollow
 \ No newline at end of file diff --git a/docs/rpc/streaming/overview.mmd b/docs/rpc/streaming/overview.mmd deleted file mode 100644 index a5d61548ae4..00000000000 --- a/docs/rpc/streaming/overview.mmd +++ /dev/null @@ -1,36 +0,0 @@ -graph TD - - subgraph ClientAgent[Client Agent] - HTTPEndpoint - DNSEndpoint - rpcClient.Health - AgentCache - MaterializedView - end - - subgraph ServerAgent[Server Agent] - RPCEndpoint - raft.Apply - FSM.applyRegistration - state.Store.Register - SubscribeEndpoint - EventPublisher - end - - Read --> HTTPEndpoint & DNSEndpoint - - HTTPEndpoint & DNSEndpoint --> rpcClient.Health - rpcClient.Health --> AgentCache - AgentCache --> MaterializedView - MaterializedView --> SubscribeEndpoint - SubscribeEndpoint -->|Subscribe to topic| EventPublisher - - Write --> RPCEndpoint - RPCEndpoint --> raft.Apply - raft.Apply --> FSM.applyRegistration - FSM.applyRegistration --> state.Store.Register - state.Store.Register -->|Publish event| EventPublisher - - class Read,Write start - classDef start fill:transparent,stroke:transparent - diff --git a/docs/rpc/streaming/overview.svg b/docs/rpc/streaming/overview.svg deleted file mode 100644 index f1a0b9874b0..00000000000 --- a/docs/rpc/streaming/overview.svg +++ /dev/null @@ -1 +0,0 @@ -
Server Agent
Client Agent
Subscribe to topic
Publish event
RPCEndpoint
raft.Apply
FSM.applyRegistration
state.Store.Register
SubscribeEndpoint
EventPublisher
HTTPEndpoint
DNSEndpoint
rpcClient.Health
AgentCache
MaterializedView
Read
Write
 \ No newline at end of file diff --git a/docs/service-discovery/README.md b/docs/service-discovery/README.md deleted file mode 100644 index 4787ce3eafe..00000000000 --- a/docs/service-discovery/README.md +++ /dev/null @@ -1,7 +0,0 @@ -# Service Discovery - -This section is still a work in progress. - -1. [Catalog](./catalog.md) -1. [DNS Interface](./dns.md) -1. [Health Checks](./health-checks.md) diff --git a/docs/service-discovery/catalog-2.mmd b/docs/service-discovery/catalog-2.mmd deleted file mode 100644 index 2db8092d70f..00000000000 --- a/docs/service-discovery/catalog-2.mmd +++ /dev/null @@ -1,36 +0,0 @@ -erDiagram - - CheckServiceNode - Node - NodeService - ServiceNode - HealthCheck - - CheckServiceNode ||--|| Node: has - CheckServiceNode ||--|| NodeService: has - CheckServiceNode ||--o{ HealthCheck: has - - Store ||--o{ Node: "stored in the node table" - Store ||--o{ ServiceNode: "stored in the service table" - Store ||--o{ HealthCheck: "stored in the checks table" - - ServiceNode ||--|| Node: references - HealthCheck ||--o| Node: references - HealthCheck ||--o| Service: references - - RegisterRequest ||--o| Node: has - RegisterRequest ||--o| NodeService: has - RegisterRequest ||--o{ HealthCheck: has - - - CheckDefinition - HealthCheckDefinition - CheckType - - HealthCheck ||--|| HealthCheckDefinition: has - - ServiceDefinition ||--|| NodeService: "is essentially a" - ServiceDefinition ||--o{ CheckType: "has" - - Config ||--o{ CheckDefinition: "has" - Config ||--o{ ServiceDefinition: "has" diff --git a/docs/service-discovery/catalog.md b/docs/service-discovery/catalog.md deleted file mode 100644 index 3aa2a80bd71..00000000000 --- a/docs/service-discovery/catalog.md +++ /dev/null @@ -1,6 +0,0 @@ -# Catalog - -This section is a work in progress. - -The catalog is at the core of both Service Discovery and Service Mesh. It accepts -registrations and deregistrations of Services, Nodes, and Checks. diff --git a/docs/service-discovery/catalog.mmd b/docs/service-discovery/catalog.mmd deleted file mode 100644 index b1a85c3b644..00000000000 --- a/docs/service-discovery/catalog.mmd +++ /dev/null @@ -1,24 +0,0 @@ -erDiagram - - CheckServiceNode - Node - NodeService - ServiceNode - HealthCheck - - CheckServiceNode ||--|| Node: has - CheckServiceNode ||--|| NodeService: has - CheckServiceNode ||--o{ HealthCheck: has - - Store ||--o{ Node: "stored in the node table" - Store ||--o{ ServiceNode: "stored in the service table" - Store ||--o{ HealthCheck: "stored in the checks table" - - ServiceNode ||--|| Node: references - HealthCheck ||--o| Node: references - HealthCheck ||--o| Service: references - - RegisterRequest ||--o| Node: has - RegisterRequest ||--o| NodeService: has - RegisterRequest ||--o{ HealthCheck: has - diff --git a/docs/service-discovery/dns.md b/docs/service-discovery/dns.md deleted file mode 100644 index 357ecc57dc3..00000000000 --- a/docs/service-discovery/dns.md +++ /dev/null @@ -1,49 +0,0 @@ -# DNS Interface - -The DNS interface allows users to find the IP address and port of services, using DNS -queries. The DNS interface is in many ways similar to an HTTP API. The major difference is -in the DNS protocol. - -There are lots of guides to DNS, the following list is a short reference that should help you -understand the parts that are relevant to the DNS interface in Consul. Full details about -the DNS protocol can be found in the RFCs: [RFC 1035], [RFC 6891], [RFC 2782], and others. - -[RFC 1035]: https://tools.ietf.org/html/rfc1035 -[RFC 6891]: https://tools.ietf.org/html/rfc6891 -[RFC 2782]: https://tools.ietf.org/html/rfc2782 - -* [wikipedia: DNS message format](https://en.wikipedia.org/wiki/Domain_Name_System#DNS_message_format) - is a quick introduction to the format used for queries and replies -* [RFC 1035 Section 4.1.1](https://datatracker.ietf.org/doc/html/rfc1035#section-4.1.1) - is a good reference for when to use specific response codes and what the different header - bits refer to. - - -## DNS Server - -The DNS interface is implemented as a DNS server using [miekg/dns] and the handlers for -requests are in `agent/dns.go`. - -[miekg/dns]: https://github.com/miekg/dns - -## DNS Queries and Records - -The DNS interface handles queries where OPCODE=0 (standard query). The following table describe the current -DNS behaviour for different record types and domain names. The Domain names (along the -top) are always in the form `..`, where `domain` is -generally documented as `consul` but can be set using the `domain` or `alt_domain` config -fields. The `prefix` is an identifier (service name, node name, prepared query name, etc). - -| Type | service | connect | ingress | node | query | addr | -|---------|-------------------------|-------------------------|-------------------------|------------------------|-----------------|------------------------------| -| SOA | Supported | Supported | Supported | Supported | Supported | Supported | -| NS | Supported | Supported | Supported | Supported | Supported | Supported | -| AXFR | Not Implemented | Not Implemented | Not Implemented | Not Implemented | Not Implemented | Not Implemented | -| A/AAAA | Supported | Supported | Supported | Supported | | Supported | -| ANY | Supported (return A) | Supported (return A) | Supported (return A) | Supported | | Supported (return A) | -| CNAME | Supported (node cname) | Supported (node cname) | Supported (node cname) | Supported (node cname) | | return empty with A as extra | -| OPT | Supported (node OPT) | Supported (node OPT) | Supported (node OPT) | Supported (node OPT) | | return empty with A as extra | -| PTR | Supported (node PTR) | Supported (node PTR) | Supported (node PTR) | Supported (node PTR) | | return empty with A as extra | -| SRV | Supported (service SRV) | Supported (service SRV) | Supported (service SRV) | No error but empty | | return empty with A as extra | -| TXT | Answer A record (????) | Answer A record (????) | Answer A record (????) | Supported | | return empty with A as extra | - diff --git a/docs/service-discovery/health-checks.md b/docs/service-discovery/health-checks.md deleted file mode 100644 index 98c8b32b216..00000000000 --- a/docs/service-discovery/health-checks.md +++ /dev/null @@ -1,37 +0,0 @@ -# Health Checks - -This section is still a work in progress. - -[agent/checks](https://github.com/hashicorp/consul/tree/main/agent/checks) contains the logic for -performing active [health checking](https://developer.hashicorp.com/docs/agent/checks.html). - - -## Check Registration flows - -There are many paths to register a check. Many of these use different struct -types, so to properly validate and convert a check, all of these paths must -be reviewed and tested. - -1. API [/v1/catalog/register](https://developer.hashicorp.com/api-docs/catalog#register-entity) - the `Checks` - field on `structs.RegisterRequest`. The entrypoint is `CatalogRegister` in - [agent/catalog_endpoint.go]. -2. API [/v1/agent/check/register](https://developer.hashicorp.com/api-docs/agent/check#register-check) - the entrypoint - is `AgentRegisterCheck` in [agent/agent_endpoint.go] -3. API [/v1/agent/service/register](https://developer.hashicorp.com/api-docs/agent/service#register-service) - - the `Check` or `Checks` fields on `ServiceDefinition`. The entrypoint is `AgentRegisterService` - in [agent/agent_endpoint.go]. -4. Config [Checks](https://developer.hashicorp.com/docs/discovery/checks) - the `Checks` and `Check` fields - on `config.Config` in [agent/config/config.go]. -5. Config [Service.Checks](https://developer.hashicorp.com/docs/discovery/services) - the - `Checks` and `Check` fields on `ServiceDefinition` in [agent/config/config.go]. -6. The returned fields of `ServiceDefinition` in [agent/config/builder.go]. -7. CLI [consul services register](https://developer.hashicorp.com/commands/services/register) - the - `Checks` and `Check` fields on `api.AgentServiceRegistration`. The entrypoint is - `ServicesFromFiles` in [command/services/config.go]. -8. API [/v1/txn](https://developer.hashicorp.com/api-docs/txn) - the `Transaction` API allows for registering a check. - - -[agent/catalog_endpoint.go]: https://github.com/hashicorp/consul/blob/main/agent/catalog_endpoint.go -[agent/agent_endpoint.go]: https://github.com/hashicorp/consul/blob/main/agent/agent_endpoint.go -[agent/config/config.go]: https://github.com/hashicorp/consul/blob/main/agent/config/config.go -[command/services/config.go]: https://github.com/hashicorp/consul/blob/main/command/services/config.go diff --git a/docs/service-mesh/README.md b/docs/service-mesh/README.md deleted file mode 100644 index e7876f3e526..00000000000 --- a/docs/service-mesh/README.md +++ /dev/null @@ -1,84 +0,0 @@ -# Service Mesh (Connect) -## Terminology -### Data plane -The data plane refers to gateways, sidecar proxies, or native application libraries that are in the request path of applications and embed service mesh logic around routing, authorization, and observability. - -For production deployments we primarily support [Envoy](https://www.envoyproxy.io/) proxy. Active development of service mesh functionality is focused on Envoy. - - -### Control plane -At a high level, the primary goal of the control plane in a service mesh is to provide configuration for the data plane. Consul's service mesh is composed of: server agents, client agents, and [consul-dataplane](https://github.com/hashicorp/consul-dataplane) proxies. - -The control plane allows users to configure policies for the service mesh, and then translates these into configuration that the data plane components will use to execute the intended functionality. - -A key distinction from the data plane is that the control plane is largely not in the request path of service to service traffic. The notable exception to this rule is the [/agent/connect/authorize](https://developer.hashicorp.com/consul/api-docs/agent/connect#authorize), discussed in the Native integration below. - - -### Connect Native -Consul's service mesh supports a "native" app integration. In this setup users must explicitly request leaf and root certificates from Consul for use in service-to-service mTLS. Additionally, to consider intentions for authorization, applications can issue an authorization check to a Consul agent. - -The Go client library for this integration exists in the [connect](https://github.com/hashicorp/consul/tree/main/connect) package. - -**APIs:** -* [/agent/connect/authorize](https://developer.hashicorp.com/consul/api-docs/agent/connect#authorize) can be used to evaluate whether intentions allow connections by a client to some target service. -* [/agent/connect/ca/leaf/:service](https://developer.hashicorp.com/consul/api-docs/agent/connect#service-leaf-certificate) can be used to request a leaf certificate for a service instance. This is the certificate to present during the mTLS handshake -* [/agent/connect/ca/roots](https://developer.hashicorp.com/consul/api-docs/agent/connect#certificate-authority-ca-roots) can be used to request the trusted certificate authority's root certificates. These are the certificates used to verify leaf certificates presented in the mTLS handshake. - - -### Built-in Proxy -Consul's service mesh was released with a built-in proxy. This proxy provides basic functionality as outlined in its [documentation](https://developer.hashicorp.com/consul/docs/connect/proxies/built-in). This proxy is not supported for production deployments and has not been under active development for several years. - -The core of the built-in proxy is implemented in the [connect/proxy](https://github.com/hashicorp/consul/tree/main/connect/proxy) package, and is launched by the [command/connect/proxy](https://github.com/hashicorp/consul/tree/main/command/connect/proxy) package. - - -## Configuration Lifecycle -![Configuring Envoy](./configuring-envoy.png) - -The high-level flow of configuring Envoy is: -1. The initial "bootstrap" configuration is generated for an Envoy proxy by a consul-dataplane instance or a Consul client agent. -2. Envoy dials the xDS server, requesting configuration to act as a particular proxy or gateway instance. The xDS server will either be a Consul server or a Consul client agent. -3. Consul initializes internal watches for the snapshot of data necessary to configure Envoy. This snapshot will contain data as collected from Consul's state. -4. As these snapshots are generated and updated, Consul will generate and push Envoy configuration for the various xDS resource types if there were changes. - -### Bootstrapping Envoy proxies -Consul generates the initial "bootstrap" configuration file for Envoy proxy instances, and can optionally launch the Envoy process itself. - -The basic information provided in Envoy's bootstrap configuration contains: -* The listener address and port for [Envoy's administration interface](https://www.envoyproxy.io/docs/envoy/latest/operations/admin). -* The ID, namespace, and admin partition of the corresponding sidecar proxy registration in Consul's catalog. -* Configuration on how to reach Consul, and the Consul ACL token to present. - -This process is handled by two different components depending on whether Consul client agents are in use: -* [consul-dataplane](https://github.com/hashicorp/consul-dataplane) is used in "agentless" Consul. -* [command/connect/envoy](https://github.com/hashicorp/consul/tree/main/command/connect/envoy) is used with Consul agents. - - -### Internal resource watches -The `proxycfg-*` family of packages drive the process of generating snapshots containing all of the data necessary to configure an Envoy proxy. This snapshot is populated via internal watches to resources such as configuration entries and service registrations. - -When initialized on a client agent these watches flow through the agent cache, which manages the associated blocking queries. On the other hand, when initialized on a Consul server these watches are done directly against the server's in-memory state store. - -For additional details see: [proxycfg](./proxycfg.md). - - -### Generating xDS Configuration -The `agent/xds` package implements the gRPC service used by Envoy to fetch configuration. At the core of the package is [delta.go](https://github.com/hashicorp/consul/blob/main/agent/xds/delta.go), which contains the implementation of the **Incremental ADS** protocol variant. With this variant there is a single stream between Consul and an Envoy proxy, and on that stream we send configuration diffs based on Envoy's current state. - -This package also contains files that generate xDS resources such as Clusters, Endpoints, Listeners, and Routes from snapshots generated by `proxycfg`. These files handle the conversion from Consul's data model to Envoy's. - -For additional details see: [xDS Server](./xds.md) and [Envoy's documentation](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol) on the xDS protocol. - - -## Additional Components -### Certificate Authority -Consul's certificate authority is the component responsible for certificate management in the service mesh. Certificates issued by the certificate authority are used for two primary reasons: -* mTLS between mesh-enabled applications. -* Intention enforcement. -* TLS between components of the control plane, such as: client agent to server agent when using [auto-config](https://developer.hashicorp.com/consul/tutorials/security-operations/docker-compose-auto-config), or leader server agent to the leader server of a peer cluster. - -For additional details see: [Certificate Authority](./ca) and the [public documentation](https://developer.hashicorp.com/consul/docs/connect/ca). - -### Configuration Entries -Configuration entries are the primary way to apply configuration or policies uniformly across the mesh. They are stored centrally on Consul's servers, and can be scoped to a service, namespace, admin partition, or a federation of datacenters. - -For additional details see: [Configuration Entries](./config-entries) and the [public documentation](https://developer.hashicorp.com/consul/docs/connect/config-entries). diff --git a/docs/service-mesh/ca/README.md b/docs/service-mesh/ca/README.md deleted file mode 100644 index 1b72f30b49d..00000000000 --- a/docs/service-mesh/ca/README.md +++ /dev/null @@ -1,114 +0,0 @@ -# Certificate Authority (Connect CA) - -The Certificate Authority Subsystem manages a CA trust chain for issuing certificates to -services and client agents (via auto-encrypt and auto-config). - -The code for the Certificate Authority is in the following packages: -1. most of the core logic is in [agent/consul/leader_connect_ca.go] -2. the providers are in [agent/connect/ca] -3. the RPC interface is in [agent/consul/connect_ca_endpoint.go] - - -[agent/consul/leader_connect_ca.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/leader_connect_ca.go -[agent/connect/ca]: https://github.com/hashicorp/consul/blob/main/agent/connect/ca/ -[agent/consul/connect_ca_endpoint.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/connect_ca_endpoint.go - - -## Architecture - -### High level overview - -In Consul the leader is responsible for handling the CA management. -When a leader election happen, and the elected leader do not have any root CA available it will start a process of creating a set of CA certificate. -Those certificates will be used to authenticate/encrypt communication between services (service mesh) or between `Consul client agent` (auto-encrypt/auto-config). This process is described in the following diagram: - -![CA creation](./hl-ca-overview.svg) - -[source](./hl-ca-overview.mmd) - -The features that benefit from Consul CA management are: -- [service Mesh/Connect](https://developer.hashicorp.com/docs/connect) -- [auto encrypt](https://developer.hashicorp.com/docs/agent/options#auto_encrypt) - - -### CA and Certificate relationship - -This diagram shows the relationship between the CA certificates in Consul primary and -secondary. - -![CA relationship](./cert-relationship.svg) - -[source](./cert-relationship.mmd) - - -In most cases there is an external root CA that provides an intermediate CA that Consul -uses as the Primary Root CA. The only except to this is when the Consul CA Provider is -used without specifying a `RootCert`. In this one case Consul will generate the Root CA -from the provided primary key, and it will be used in the primary as the top of the chain -of trust. - -In the primary datacenter, the Consul and AWS providers use the Primary Root CA to sign -leaf certificates. The Vault provider uses an intermediate CA to sign leaf certificates. - -Leaf certificates are created for two purposes: -1. the Leaf Cert Service is used by envoy proxies in the mesh to perform mTLS with other - services. -2. the Leaf Cert Client Agent is created by auto-encrypt and auto-config. It is used by - client agents for HTTP API TLS, and for mTLS for RPC requests to servers. - -Any secondary datacenters receive an intermediate certificate, signed by the Primary Root -CA, which is used as the CA certificate to sign leaf certificates in the secondary -datacenter. - -## Operations - -When trying to learn the CA subsystem it can be helpful to understand the operations that -it can perform. The sections below are the complete set of read, write, and periodic -operations that provide the full behaviour of the CA subsystem. - -### Periodic Operations - -Periodic (or background) opeartions are started automatically by the Consul leader. They run at some interval (often 1 hour). - -- `CAManager.InitializeCA` - attempts to initialize the CA when a leader is ellected. If the synchronous InitializeCA fails, `CAManager.backgroundCAInitialization` runs `InitializeCA` periodically in a goroutine until it succeeds. -- `CAManager.RenewIntermediate` - (called by `CAManager.intermediateCertRenewalWatch`) runs in the primary if the provider uses a separate signing cert (the Vault provider). The operation always runs in the secondary. Renews the signing cert once half its lifetime has passed. -- `CAManager.secondaryCARootWatch` - runs in secondary only. Performs a blocking query to the primary to retrieve any updates to the CA roots and stores them locally. -- `Server.runCARootPruning` - removes non-active and expired roots from state.CARoots - -### Read Operations - -- `RPC.ConnectCA.ConfigurationGet` - returns the CA provider configuration. Only called by user, not by any internal subsystems. -- `RPC.ConnectCA.Roots` - returns all the roots, the trust domain ID, and the ID of the active root. Each "root" also includes the signing key/cert, and any intermediate certs in the chain. It is used (via the cache) by all the connect proxy types. - -### Write Operations - -- `CAManager.UpdateConfiguration` - (via `RPC.ConnectCA.ConfigurationSet`) called by a user when they want to change the provider or provider configuration (ex: rotate root CA). -- `CAManager.Provider.SignIntermediate` - (via `RPC.ConnectCA.SignIntermediate`) called from the secondary DC: - 1. by `CAManager.RenewIntermediate` to sign the new intermediate when the old intermediate is about to expire - 2. by `CAMananger.initializeSecondary` when setting up a new secondary, when the provider is changed in the secondary - by a user action, or when the primary roots changed and the secondary needs to generate a new intermediate for the new - primary roots. -- `CAMananger.SignCertificate` - is used by: - 1. (via `RPC.ConnectCA.Sign`) - called by client agents to sign a leaf cert for a connect proxy (via `agent/cache-types/connect_ca_leaf.go`) - 2. (via in-process call to `RPC.ConnectCA.Sign`) - called by auto-encrypt to sign a leaf cert for a client agent - 3. called by Auto-Config to sign a leaf cert for a client agent - -## detailed call flow -![CA Leader Sequence](./ca-leader-sequence.svg) - -[source](./ca-leader-sequence.mmd) - -####TODO: -- sequence diagram for leaf signing -- sequence diagram for CA cert rotation - -## CAManager states - -This section is a work in progress - -TODO: style the diagram to match the others, and add some narative text to describe the -diagram. - -![CA Mananger states](./state-machine.svg) - - diff --git a/docs/service-mesh/ca/ca-leader-sequence.mmd b/docs/service-mesh/ca/ca-leader-sequence.mmd deleted file mode 100644 index f81e66b7ab5..00000000000 --- a/docs/service-mesh/ca/ca-leader-sequence.mmd +++ /dev/null @@ -1,19 +0,0 @@ -sequenceDiagram -Participant Provider -Participant PL As Primary Leader -Participant SL As Secondary Leader -Alt Primary don't have a valid CA -PL->>Provider:initializeRootCA (fetch root and sign intermediate) -Provider->>PL:root + intermediate -PL->>PL:RPC ConnectCA.Roots (fetch primary root and store it) -end -SL->>PL: RPC ConnectCA.Roots (fetch primary root and store it) -PL->>SL: Root + intermediate -Alt Secondary needs a new intermediate (check if current intermediate is signed by primary root) -SL->>Provider: Generate CSR -Provider->>SL: CSR -SL->>PL: ConnectCA.SignIntermediate (CSR) -PL->>SL: Intermediate CA (secondary) -SL->>Provider: Set Intermediate (secondary CA) + root (primary CA) -SL->>SL: Store certs in RAFT (primary root + secondary intermediate) -end \ No newline at end of file diff --git a/docs/service-mesh/ca/ca-leader-sequence.svg b/docs/service-mesh/ca/ca-leader-sequence.svg deleted file mode 100644 index d19b957e444..00000000000 --- a/docs/service-mesh/ca/ca-leader-sequence.svg +++ /dev/null @@ -1 +0,0 @@ -ProviderPrimary LeaderSecondary LeaderinitializeRootCA (fetch root and sign intermediate)root + intermediateRPC ConnectCA.Roots (fetch primary root and store it)alt[Primary don't have a valid CA]RPC ConnectCA.Roots (fetch primary root and store it)Root + intermediateGenerate CSRCSRConnectCA.SignIntermediate (CSR)Intermediate CA (secondary)Set Intermediate (secondary CA) + root (primary CA)Store certs in RAFT (primary root + secondary intermediate)alt[Secondary needs a new intermediate (check if current intermediate is signed by primary root)]ProviderPrimary LeaderSecondary Leader \ No newline at end of file diff --git a/docs/service-mesh/ca/cert-relationship.mmd b/docs/service-mesh/ca/cert-relationship.mmd deleted file mode 100644 index 85ae848dcb5..00000000000 --- a/docs/service-mesh/ca/cert-relationship.mmd +++ /dev/null @@ -1,31 +0,0 @@ -graph TD - - ExternalRootCA["External RootCA (optional)"] - - subgraph "Consul Primary" - PrimaryRootCA["Primary Root CA"] - PrimarySigningCA["Primary Signing CA (conditional)"] - end - - subgraph "Consul Secondary" - SeconarySigningCA["Seconary Signing CA"] - end - - LeafCertAgentPrimary[Leaf Cert Client Agent] - LeafCertServicePrimary[Leaf Cert Service] - - LeafCertAgentSecondary[Leaf Cert Client Agent] - LeafCertServiceSecondary[Leaf Cert Service] - - - ExternalRootCA -.-> PrimaryRootCA - PrimaryRootCA -.-> PrimarySigningCA - - PrimaryRootCA --> SeconarySigningCA - - PrimarySigningCA --> LeafCertAgentPrimary - PrimarySigningCA --> LeafCertServicePrimary - - SeconarySigningCA --> LeafCertAgentSecondary - SeconarySigningCA --> LeafCertServiceSecondary - diff --git a/docs/service-mesh/ca/cert-relationship.svg b/docs/service-mesh/ca/cert-relationship.svg deleted file mode 100644 index a32f838d198..00000000000 --- a/docs/service-mesh/ca/cert-relationship.svg +++ /dev/null @@ -1 +0,0 @@ -
Consul Secondary
Consul Primary
Seconary Signing CA
Primary Root CA
Primary Signing CA (conditional)
External RootCA (optional)
Leaf Cert Client Agent
Leaf Cert Service
Leaf Cert Client Agent
Leaf Cert Service
 \ No newline at end of file diff --git a/docs/service-mesh/ca/hl-ca-overview.mmd b/docs/service-mesh/ca/hl-ca-overview.mmd deleted file mode 100644 index 952f64b98f7..00000000000 --- a/docs/service-mesh/ca/hl-ca-overview.mmd +++ /dev/null @@ -1,43 +0,0 @@ -graph TD - subgraph "Primary DC" - leaderP["Leader"] - rootCAI["Root CA "] - rootCA["Root CA "] - Provider["Consul/AWS providers"] - IntermediateProvider["Vault provider"] - intermediateCAP["Intermediate CA "] - leafP["Leaf certificates"] - end - - subgraph "Secondary DC" - leaderS["Leader"] - intermediateCAS["Intermediate CA"] - leafS["Leaf certificates"] - ProviderS["Consul/AWS/Vault providers"] - end - - consulCAS["Consul client Agents"] - servicesS["Mesh services"] - - consulCAP["Consul client Agents"] - servicesP["Mesh services"] - - leaderP -->|use|Provider - leaderP-->|use|IntermediateProvider - Provider--> |fetch/self sign|rootCA - IntermediateProvider --> |fetch/self sign|rootCAI - rootCAI -->|sign| intermediateCAP - intermediateCAP -->|sign| leafP - rootCA -->|sign| leafP - - leaderS -->|use| ProviderS - ProviderS --> |generate csr| intermediateCAS - rootCA -->|sign| intermediateCAS - rootCAI -->|sign| intermediateCAS - intermediateCAS --> |sign| leafS - - leafS -->|auth/encrypt| servicesS - leafS -->|auth/encrypt| consulCAS - leafP -->|auth/encrypt| servicesP - leafP -->|auth/encrypt| consulCAP - diff --git a/docs/service-mesh/ca/hl-ca-overview.svg b/docs/service-mesh/ca/hl-ca-overview.svg deleted file mode 100644 index 76a61662451..00000000000 --- a/docs/service-mesh/ca/hl-ca-overview.svg +++ /dev/null @@ -1 +0,0 @@ -
Secondary DC
Primary DC
use
use
fetch/self sign
fetch/self sign
sign
sign
sign
use
generate csr
sign
sign
sign
auth/encrypt
auth/encrypt
auth/encrypt
auth/encrypt
Leader
Intermediate CA
Leaf certificates
Consul/AWS/Vault providers
Leader
Root CA
Root CA
Consul/AWS providers
Vault provider
Intermediate CA
Leaf certificates
Consul client Agents
Mesh services
Consul client Agents
Mesh services
 \ No newline at end of file diff --git a/docs/service-mesh/ca/state-machine.mmd b/docs/service-mesh/ca/state-machine.mmd deleted file mode 100644 index 214e3116793..00000000000 --- a/docs/service-mesh/ca/state-machine.mmd +++ /dev/null @@ -1,23 +0,0 @@ -stateDiagram-v2 - - [*] --> Uninitialized - Uninitialized --> Initializing : InitializeCA - Uninitialized --> Reconfig : UpdateConfiguration - Reconfig --> Uninitialized : return - - # Initialized can transition to any state - Initialized --> Renew : RenewIntermediate - Initialized --> Uninitialized : Stop - Initialized --> Reconfig : UpdateConfiguration - Initialized --> Initializing : INVALID - - # Initialized is set using validate=false - Uninitialized --> Initialized : INVALID - Reconfig --> Initialized : return - Initializing --> Initialized : InitializeCA complete - Renew --> Initialized : return - - # Uninitialized is set using validate=false - Renew --> Uninitialized : Stop - Reconfig --> Uninitialized : Stop - Initializing --> Uninitialized : Stop diff --git a/docs/service-mesh/ca/state-machine.svg b/docs/service-mesh/ca/state-machine.svg deleted file mode 100644 index f63e880e080..00000000000 --- a/docs/service-mesh/ca/state-machine.svg +++ /dev/null @@ -1 +0,0 @@ -
InitializeCA
UpdateConfiguration
return
RenewIntermediate
Stop
UpdateConfiguration
INVALID
INVALID
return
InitializeCA complete
return
Stop
Stop
Stop
Uninitialized
Initializing
Reconfig
Initialized
Renew
 \ No newline at end of file diff --git a/docs/service-mesh/config-entries/README.md b/docs/service-mesh/config-entries/README.md deleted file mode 100644 index 1fdab309a38..00000000000 --- a/docs/service-mesh/config-entries/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Configuration Entries -## Background -Consul's catalog was designed to store node, service, and check registrations. An instance of a service is defined by a combination of a node and a service registered against that node. Each of these service registrations is considered to be an _instance_ of a logical service. - -Logical services in consul are implicit. There is no true logical service "object" that defines the existence of a service. In Consul, a service is a collection of service instances. When the first instance of a service is registered, the service becomes known, and when the last instance is deregistered then the service is mostly forgotten. - -The lack of a first-class service object makes it so that the source of truth for the configuration of a service is the configuration of its service instances. Registering services at the edge has the side-effect of making it more challenging to enforce uniform configuration across service or proxy instances. - -## Overview -Configuration entries are the primary way to apply configuration or policies uniformly across the mesh. They are stored centrally on Consul's servers, and can be scoped to a service, namespace, a cluster, or a federation of datacenters. - -With WAN-federation, the scope of config entries spans across Consul datacenters. All config entry writes are forwarded to the primary datacenter, and each secondary datacenter has blocking queries to the primary datacenter so that they can replicate in the latest entries. The implications of their global scope means that a config entry referencing service "web" in namespace "frontend" will apply to **all** services named "web" in a namespace called "frontend" across federated datacenters. - -Since the introduction of admin partitions in Consul v1.11.0, the broadest scope for a config entry is a partition. A service a namespace of one partition is logically distinct from a service with the same name and namespace in another partition. - -This restriction also applies when peering two clusters. As of Consul 1.14, configuration entries are not replicated to peer clusters. Configuration entries applied in one cluster do not apply to services in a peer cluster because services in different clusters that share the same name are considered to be logically different. - -The diagram below shows how various resources are scoped: -* Service and gateway config entries exist within a namespace but can span across WAN federated datacenters, just as namespaces can. -* The proxy-defaults config entry is not namespaced, but can similarly span across datacenters. -* Config entries in admin partition "default" are logically distinct from config entries in other admin partitions like "team1". - -![World of Data](./world-of-data.png) - -For sidecar proxies the fundamental config entries are [service-defaults](https://developer.hashicorp.com/consul/docs/connect/config-entries/service-defaults) and [proxy-defaults](https://developer.hashicorp.com/consul/docs/connect/config-entries/proxy-defaults). These config entries can be thought of as default values for proxy or gateway service registrations. Their data gets merged into proxy service registrations themselves. - -As defaults, their data is of lower precedence compared to data stored in proxy registrations. In Consul, data present in individual proxy registrations **always** has a higher precedence than the equivalent stored in a configuration entry. - -## Additional Information: -- [Config Resolution](config-resolution.md): Summary of the mechanics of how configuration entries are resolved for sidecar proxies. - -## Lifecycle - -The diagram below shows the lifecycle of a configuration entry along with the locations where the concrete types are stored and common tests. The steps highlighted along these paths are ones that will likely require code or test updates when modifying a config entry or adding a new config entry kind. - -![Life of a Config Entry](./life-of-a-config-entry.png) diff --git a/docs/service-mesh/config-entries/config-resolution.md b/docs/service-mesh/config-entries/config-resolution.md deleted file mode 100644 index 58a77bcc861..00000000000 --- a/docs/service-mesh/config-entries/config-resolution.md +++ /dev/null @@ -1,25 +0,0 @@ -# Configuration Entries -## Merging Service Config -Merging data from defaults in config entries is roughly a two-step process: -1. Flatten central configuration for proxies at a Consul server. Resolving config is done on behalf of a specific service, and this merge considers multiple config entries, such as proxy-defaults and service-defaults. -2. Merge flattened central defaults into sidecar proxy registrations. This occurs at a client agent in the agentful case, or on a server in the agentless case. - -Any time the centralized defaults are updated, internal watches will fire and their data will be merged into the registration for the given sidecar proxy. You can ensure that a config entry has applied by querying the service: -* Agentful: `/v1/agent/service/:service_id` -* Agentless: `/v1/health/connect/:service_name?merge-central-config` - -### Agentless -The agentless flow relies on [configentry.MergeNodeServiceWithCentralConfig](https://github.com/hashicorp/consul/blob/0402fd23a349513d3e8d137ddbffcdefcc89838b/agent/configentry/merge_service_config.go) to merge default configuration. This helper is called both when bootstrapping a new dataplane proxy and when an Envoy proxy first dials the `xds` server on a Consul server. - -This merge does not directly update the proxy registration in the catalog. The merged proxy registration is passed to the consuming `proxycfg` package. - -### Agentful -The [agent.ServiceManager](https://github.com/hashicorp/consul/blob/0402fd23a349513d3e8d137ddbffcdefcc89838b/agent/service_manager.go#LL18) is responsible for ensuring that central configuration is merged down into proxy registrations on **client agents**. Any time a proxy is registered or updated, the service manager will set up a watch through the agent cache to fetch the central configuration. - -When the central configuration is fetched, the configuration is merged and persisted to the agent state in [serviceConfigWatch.handleUpdate](https://github.com/hashicorp/consul/blob/0402fd23a349513d3e8d137ddbffcdefcc89838b/agent/service_manager.go#L256). - -## Consuming Central Config -Once centralized defaults have been merged into service registrations, the data becomes available to consume by the `proxycfg` package. This package assembles the necessary Consul data to generate Envoy configuration for a given proxy. - -The `proxycfg` package does not need explicit watches for the service or proxy-defaults that apply to a sidecar proxy because of how these defaults are merged into proxy registrations. When defaults change, a fresh `proxycfg` snapshot is built based on the latest registration. - diff --git a/docs/service-mesh/config-entries/life-of-a-config-entry.png b/docs/service-mesh/config-entries/life-of-a-config-entry.png deleted file mode 100644 index 0ac518fb6c6..00000000000 Binary files a/docs/service-mesh/config-entries/life-of-a-config-entry.png and /dev/null differ diff --git a/docs/service-mesh/config-entries/world-of-data.png b/docs/service-mesh/config-entries/world-of-data.png deleted file mode 100644 index 35e202cd09e..00000000000 Binary files a/docs/service-mesh/config-entries/world-of-data.png and /dev/null differ diff --git a/docs/service-mesh/configuring-envoy.png b/docs/service-mesh/configuring-envoy.png deleted file mode 100644 index c8ba50d12b3..00000000000 Binary files a/docs/service-mesh/configuring-envoy.png and /dev/null differ diff --git a/docs/service-mesh/proxycfg-snapshot-building.png b/docs/service-mesh/proxycfg-snapshot-building.png deleted file mode 100644 index dda4f0cb3d9..00000000000 Binary files a/docs/service-mesh/proxycfg-snapshot-building.png and /dev/null differ diff --git a/docs/service-mesh/proxycfg-snapshot-sharing.png b/docs/service-mesh/proxycfg-snapshot-sharing.png deleted file mode 100644 index 9897da68301..00000000000 Binary files a/docs/service-mesh/proxycfg-snapshot-sharing.png and /dev/null differ diff --git a/docs/service-mesh/proxycfg.md b/docs/service-mesh/proxycfg.md deleted file mode 100644 index de340abbdb5..00000000000 --- a/docs/service-mesh/proxycfg.md +++ /dev/null @@ -1,45 +0,0 @@ -# Proxycfg -The `proxycfg-*` family of packages drive the process of generating snapshots containing all of the data necessary to configure an Envoy proxy. This snapshot is populated via internal watches to resources such as configuration entries and service registrations. - -When initialized on a client agent these watches flow through the agent cache, which manages the associated blocking queries. On the other hand, when initialized on a Consul server these watches are done directly against the server's in-memory state store. - -## agent/proxycfg -### Manager -The `proxycfg.Manager` is responsible for initializing or tearing down the machinery required to watch the internal data required to configure an Envoy proxy. This includes initializing the snapshots of internal data for proxy configuration, kicking off the long-running update routines, and managing the delivery of snapshots to the xDS server. - -![Snapshot sharing](./proxycfg-snapshot-sharing.png) - -### State management -Building a snapshot of data to configure a proxy is done with a long-running event-processing state machine. When a proxy is first registered with the manager we initialize the known watches that are needed based on the kind of proxy or gateway being watched. Each of these watches will contain the necessary request type, as well as a `CorrelationID`, which acts as a key for the watch. If a watch will not exist for the duration of a proxy instance, we also store a context cancellation function so that the watch can be torn down later. - -The results of these watches are then consumed as a stream of update events to a channel. Any time a new event is received, the `handleUpdate` function is called, which contains kind-specific logic. For each new event the `CorrelationID` is inspected to determine what watch the event corresponds to. From an event we may store the data directly or initialize/destroy additional watches. - -Since the event updates are processed concurrently, the way to ensure ordering is via chained watches. For example, the discovery chain dictates what upstream instances need to be watched for a logical upstream. Once a discovery chain update is received we then kick off a service discovery watch for the appropriate targets. - -![Snapshot building](./proxycfg-snapshot-building.png) - -## agent/proxycfg-glue -The dependencies to watch data on Consul's servers are encoded in `proxycfg.DataSources`. For any given resource to watch there is a corresponding data source, which is contained in the `DataSources` type as an interface. These interfaces are uniform: - -```go -type interface { - Notify(ctx context.Context, req *structs.ServiceDumpRequest, correlationID string, ch chan<- UpdateEvent) error -} -``` - -Implementations for these interfaces exist within the `proxycfg-glue` package. When using the agentless consul-dataplane the implementation names have the structure: `Server`, and when using client agents these implementations have the structure: `Cache`. - -For each resource there are parallel implementations that use the agent's cache as the data source or the server's state store. Requests to the state store may use subscriptions to Consul's internal event publisher, or a memdb WatchSet. For more information about the event publisher see the [streaming documentation](/docs/rpc/streaming). - -If the event publisher contains the necessary data it is preferable to use that as the server datasource over a memdb WatchSet. Memdb's watch sets are susceptible to spurious wake-ups and may lead to doing more work than strictly necessary when a change occurs. The event publisher watches memdb tables for changes and broadcasts incremental events based on the data that changed. It explicitly avoids re-generating all the data for the key being watched. - -## agent/proxycfg-sources -Contains implementations of the `agent/xds/ProxyConfigSource` interface, which ensures that proxy instances are registered or deregistered with the `proxycfg.Manager`. - -There are two distinct implementations split across two packages. Both of these registers, re-registers, or deregisters watches with the `proxycfg.Manager`. -* `/agent/proxycfg-sources/local`: Path exercised by Consul client agents. -* `/agent/proxycfg-sources/catalog`: Path exercised by Consul server agents. - -The primary reason why these two implementations are separate is due to how proxy service registrations are handled in agentless and agentful deployments: -* Server agents watch the catalog for proxy registration changes, while client agents watch their local state -* Server agents merge data from service-defaults and proxy-defaults configuration entries at the `catalog.ConfigSource` sync function, while client agents merge them by hooking into the service registration code path. diff --git a/docs/service-mesh/xds.md b/docs/service-mesh/xds.md deleted file mode 100644 index e876a336277..00000000000 --- a/docs/service-mesh/xds.md +++ /dev/null @@ -1,60 +0,0 @@ -# xDS Server -The `agent/xds` package implements the streaming `DeltaAggregatedResources` gRPC service used by Envoy to fetch configuration. At the core of the package is [delta.go](https://github.com/hashicorp/consul/blob/main/agent/xds/delta.go), which contains the implementation of the **Incremental ADS** protocol variant. With this variant there is a single stream between Consul and an Envoy proxy, and on that stream we send configuration diffs based on Envoy's current state. - -The remainder of this package contains the logic necessary for generating xDS configuration such as Clusters, Endpoints, Listeners, and Routes from snapshots generated by `proxycfg`. - -## Authorization -The xDS server authorizes requests by looking at the proxy ID in the request and ensuring the ACL token has `service:write` access to either the destination service (for kind=ConnectProxy), or the gateway service (for other kinds). - -This authorization strategy is based on the assumption of how the corresponding -`proxycfg.ConfigSnapshot` is constructed. Most interfaces (HTTP, DNS, RPC) -authorize requests by authorizing the data in the response, or by filtering -out data that the requester is not authorized to view. - -This authorization strategy requires that [agent/proxycfg](https://github.com/hashicorp/consul/blob/main/agent/proxycfg) only fetches data using a -token with the same permissions, and that it only stores data by proxy ID. We assume -that any data in the snapshot was already filtered, which allows this authorization at the -xDS server to only perform a shallow check against the proxy ID. - - -## Config Generation -The xDS types that Consul supports as of v1.14 are: Clusters, Endpoints, Listeners, and Routes. For each of these resource types there is a corresponding file such as [listeners.go](https://github.com/hashicorp/consul/blob/main/agent/xds/listeners.go). There, the entry-point will take a proxycfg snapshot and generate xDS configuration depending on the kind of proxy being configured. There are diverging paths depending on whether a sidecar is being configured, or a gateway. - - -## Testing -Testing changes to this package is generally done at two layers: -- Against golden files, where each test case tests against a fixed file containing the JSON representation of an xDS resource. -- In integration tests, which spin up live instances of Consul and Envoy and make assertions against Envoy's metrics or configuration. - -### Golden files -Tests against golden files exists in functions with names such as `TestAllResourcesFromSnapshot`, `TestListenersFromSnapshot`, etc. These tests generate xDS configuration from a `proxycfg.ConfigSnapshot`, mimicking how we generate configuration for Envoy. - -The primary source for the test snapshots is `proxycfg.TestConfigSnapshot`. This function will construct a snapshot from a list of events by calling `initialize` and `handleUpdate` as we do in production code. You can attach new update events to the snapshot, or override existing events by emitting a replacement event for an existing `CorrelationID`. - -When a new test case is added, the corresponding Golden files can be generated using: -``` -go test ./agent/xds -update -run TestAllResourcesFromSnapshot -``` - -The new golden files then must be **manually** inspected to ensure that the Envoy configuration was generated as expected. Tests against golden files do not assert that the configuration works as intended, but rather that it _looks_ as intended. - -### Integration tests -#### New consul-container integration tests -TODO - -#### Legacy bash-driven integration tests -Updating one of these integration tests may be appropriate if fixing a bug in functionality that is tested there, or making an improvement to functionality tested there. If the new test involves significant modifications to the bash helpers you should consider using adding a `consul-container` integration test instead. - -For more information refer to their [documentation](test/integration/connect/envoy/). - -## Delta (Incremental) xDS Protocol -Consul's implementation of the incremental xDS protocol exists in the file [delta.go](https://github.com/hashicorp/consul/blob/main/agent/xds/delta.go). The interactions between Envoy follow the general guidance from [Envoy's documentation](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol), - -The xDS stream is a bidirectional stream, and messages from Envoy can be either a: subscription request for resources, an ACK for resources it successfully stored, or a NACK for resources that it did not store. ACKs and NACKs can be associated with specific messages by tracking a "nonce", which Consul generates and pushes in every response sent to Envoy. Consul will only ever send xDS resources, into the stream and will only send resources if it believes that Envoy does not already have them. - -For mock examples of how interactions over the stream can play out, refer to the tests in [delta_test.go](https://github.com/hashicorp/consul/blob/main/agent/xds/delta_test.go). - -To achieve the aim of only sending incremental diffs to Envoy there are maps tracking the state of both Envoy and Consul. The [xDSDeltaType](https://github.com/hashicorp/consul/blob/c7ef04c5979dbc311ff3c67b7bf3028a93e8b0f1/agent/xds/delta.go#L459) struct contains this data for each xDS resource type: -* `subscriptions` tracks the names that Envoy is interested in for a given xDS type. For example, if Envoy is subscribed to Routes, then this map would contain the names of the routes that Envoy is interested in receiving updates about. -* `resourceVersions` tracks the hash of resources that Envoy has ACK'd. This way we can avoid sending data that Envoy already has. -* `pendingUpdates` holds a map used for tracking data that has been sent to Envoy but has not been ACK'd yet. Certain xDS types have strict ordering requirements to avoid dropping traffic. Tracking pending updates allows us to block sending another update of the same type or sending resources that depend on a previous one being acknowledged. \ No newline at end of file diff --git a/docs/v2-architecture/controller-architecture/README.md b/docs/v2-architecture/controller-architecture/README.md deleted file mode 100644 index 18eb9d1102a..00000000000 --- a/docs/v2-architecture/controller-architecture/README.md +++ /dev/null @@ -1,116 +0,0 @@ -# Overview - -> **Note** -> Looking for guidance on adding new resources and controllers to Consul? Check -> out the [developer guide](guide.md). - -Consul 1.16 introduced a set of [generic APIs] for managing resources, and a -[controller runtime] for building functionality on top of them. - -[generic APIs]: ../../../proto-public/pbresource/resource.proto -[controller runtime]: ../../../internal/controller - -Previously, adding features to Consul involved making changes at every layer of -the stack, including: HTTP handlers, RPC handlers, MemDB tables, Raft -operations, and CLI commands. - -This architecture made sense when the product was maintained by a small core -group who could keep the entire system in their heads, but presented significant -collaboration, ownership, and onboarding challenges when our contributor base -expanded to many engineers, across several teams, and the product grew in -complexity. - -In the new model, teams can work with much greater autonomy by building on top -of a shared platform and own their resource types and controllers. - -## Architecture Overview - -![architecture diagram](architecture-overview.png) - -[source](https://whimsical.com/state-store-v2-UKE6SaEPXNc4UrZBrZj4Kg) - -Our resource-oriented architecture comprises the following components: - -#### Resource Service - -[Resource Service](../../../proto-public/pbresource/resource.proto) is a gRPC -service that contains the shared logic for creating, reading, updating, -deleting, and watching resources. It will be consumed by controllers, our -Kubernetes integration, the CLI, and mapped to an HTTP+JSON API. - -#### Type Registry - -[Type Registry](../../../internal/resource/registry.go) is where teams register -their resource types, along with hooks for performing structural validation, -authorization, etc. - -#### Storage Backend - -[Storage Backend](../../../internal/storage/storage.go) is an abstraction over -low-level storage primitives. Today, there are two implementations (Raft and -an in-memory backend for tests) but in the future, we envisage external storage -systems such as the Kubernetes API or an RDBMS could be supported which would -reduce operational complexity for our customers. - -#### Controllers - -[Controllers](../../../internal/controller/api.go) implement Consul's business -logic using asynchronous control loops that respond to changes in resources. -Please see [Controller docs](controllers.md) for more details about controllers - -## Raft Storage Backend - -Our [Raft Storage Backend](../../../internal/storage/raft/backend.go) integrates -with the existing Raft machinery (e.g. FSM) used by the [old state store]. It -also transparently forwards writes and strongly consistent reads to the leader -over gRPC. - -There's quite a lot going on here, so to dig into the details, let's take a look -at how a write operation is handled. - -[old state store]: ../persistence/ - -![raft storage backend diagram](raft-backend.png) - -[source](https://whimsical.com/state-store-v2-UKE6SaEPXNc4UrZBrZj4Kg) - -#### Steps 1 & 2 - -User calls the resource service's `Write` endpoint, on a Raft follower, which -in-turn calls the storage backend's `WriteCAS` method. - -#### Steps 3 & 4 - -The storage backend determines that the current server is a Raft follower, and -forwards the operation to the leader via a gRPC [forwarding service] listening -on the multiplexed RPC port ([`ports.server`]). - -[forwarding service]: ../../../proto/private/pbstorage/raft.proto -[`ports.server`]: https://developer.hashicorp.com/consul/docs/reference/agent/configuration-files/general#server_rpc_port - -#### Step 5 - -The leader's storage backend serializes the operation to protobuf and applies it -to the Raft log. As we need to share the Raft log with the old state store, we go -through the [`consul.raftHandle`](../../../agent/consul/raft_handle.go) and -[`consul.Server`](../../agent/consul/server/server.go) which applies a msgpack -envelope and type byte prefix. - -#### Step 6 - -Raft consensus happens! Once the log has been committed, it is applied to the -[FSM](../../../agent/consul/fsm/fsm.go) which calls the storage backend's `Apply` -method to apply the protobuf-encoded operation to the [`inmem.Store`]. - -[`inmem.Store`]: ../../../internal/storage/inmem/store.go - -#### Steps 7, 8, 9 - -At this point, the operation is complete. The forwarding service returns a -successful response, as does the follower's storage backend, and the user -gets a successful response too. - -#### Steps 10 & 11 - -Asynchronously, the log is replicated to followers and applied to their storage -backends. diff --git a/docs/v2-architecture/controller-architecture/architecture-overview.png b/docs/v2-architecture/controller-architecture/architecture-overview.png deleted file mode 100644 index 656844393e1..00000000000 Binary files a/docs/v2-architecture/controller-architecture/architecture-overview.png and /dev/null differ diff --git a/docs/v2-architecture/controller-architecture/controllers.md b/docs/v2-architecture/controller-architecture/controllers.md deleted file mode 100644 index e521a07208c..00000000000 --- a/docs/v2-architecture/controller-architecture/controllers.md +++ /dev/null @@ -1,268 +0,0 @@ -# Controllers - -This page describes how to write controllers in Consul's new controller architecture. - --> **Note**: This information is valid as of Consul 1.17 but some portions may change in future releases. - -## Controller Basics - -A controller consists of several parts: - -1. **The watched type** - This is the main type a controller is watching and reconciling. -2. **Additional watched types** - These are additional types a controller may care about in addition to the main watched type. -3. **Additional custom watches** - These are the watches for things that aren't resources in Consul. -4. **Reconciler** - This is the instance that's responsible for reconciling requests whenever there's an event for the main watched type or for any of the watched types. -5. **Initializer** - This is responsible for anything that needs to be executed when the controller is started. - -A basic controller setup could look like this: - -```go -func barController() controller.Controller { - return controller.NewController("bar", pbexample.BarType). - WithReconciler(barReconciler{}) -} -``` - -barReconciler needs to implement the `Reconcile` method of the `Reconciler` interface. -It's important to note that the `Reconcile` method only gets the request with the `ID` of the main -watched resource and so it's up to the reconcile implementation to fetch the resource and any relevant information needed -to perform the reconciliation. The most basic reconciler could look as follows: - -```go -type barReconciler struct {} - -func (b *barReconciler) Reconcile(ctx context.Context, rt Runtime, req Request) error { - ... -} -``` - -## Watching Additional Resources - -Most of the time, controllers will need to watch more resources in addition to the main watched type. -To set up an additional watch, the main thing we need to figure out is how to map additional watched resource to the main -watched resource. Controller-runtime allows us to implement a mapper function that can take the additional watched resource -as the input and produce reconcile `Requests` for our main watched type. - -To figure out how to map the two resources together, we need to think about the relationship between the two resources. - -There are several common relationship types between resources that are being used currently: -1. Name-alignment: this relationship means that resources are named the same and live in the same tenancy, but have different data. Examples: `Service` and `ServiceEndpoints`, `Workload` and `ProxyStateTemplate`. -2. Selector: this relationship happens when one resource selects another by name or name prefix. Examples: `Service` and `Workload`, `ProxyConfiguration` and `Workload`. -3. Owner: in this relationship, one resource is the owner of another resource. Examples: `Service` and `ServiceEndpoints`, `HealthStatus` and `Workload`. -4. Arbitrary reference: in this relationship, one resource may reference another by some sort of reference. This reference could be a single string in the resource data or a more composite reference containing name, tenancy, and type. Examples: `Workload` and `WorkloadIdentity`, `HTTPRoute` and `Service`. - -Note that it's possible for the two watched resources to have more than one relationship type simultaneously. -For example, `FailoverPolicy` type is name-aligned with a service to which it applies, however, it also contains -references to destination services, and for a controller that reconciles `FailoverPolicy` and watches `Service` -we need to account for both type 1 and type 4 relationship whenever we get an event for a `Service`. - -### Simple Mappers - -Let's look at some simple mapping examples. - -#### Name-aligned resources -If our resources only have a name-aligned relationship, we can map them with a built-in function: - -```go -func barController() controller.Controller { - return controller.NewController("bar", pbexample.BarType). - WithWatch(pbexample.FooType, controller.ReplaceType(pbexample.BarType)). - WithReconciler(barReconciler{}) -} -``` - -Here, all we need to do is replace the type of the `Foo` resource whenever we get an event for it. - -#### Owned resources - -Let's say our `Foo` resource owns `Bar` resources, where any `Foo` resource can own multiple `Bar` resources. -In this case, whenever we see a new event for `Foo`, all we need to do is get all `Bar` resources that `Foo` currently owns. -For this, we can also use a built-in function to set up our watch: - -```go -func MapOwned(ctx context.Context, rt controller.Runtime, res *pbresource.Resource) ([]controller.Request, error) { - resp, err := rt.Client.ListByOwner(ctx, &pbresource.ListByOwnerRequest{Owner: res.Id}) - if err != nil { - return nil, err - } - - var result []controller.Request - for _, r := range resp.Resources { - result = append(result, controller.Request{ID: r.Id}) - } - - return result, nil -} - -func barController() controller.Controller { - return controller.NewController("bar", pbexample.BarType). - WithWatch(pbexample.FooType, MapOwned). - WithReconciler(barReconciler{}) -} -``` - -### Advanced Mappers and Caches - -For selector or arbitrary reference relationships, the mapping that we choose may need to be more advanced. - -#### Naive mapper implementation - -Let's first consider what a naive mapping function could look like in this case. Let's say that the `Bar` resource -references `Foo` resource by name in the data. Now to watch and map `Foo` resources, we need to be able to find all relevant `Bar` resources -whenever we get an event for a `Foo` resource. - -```go -func MapFoo(ctx context.Context, rt controller.Runtime, res *pbresource.Resource) ([]controller.Request, error) { - resp, err := rt.Client.List(ctx, &pbresource.ListRequest{Type: pbexample.BarType, Tenancy: res.Id.Tenancy}) - if err != nil { - return nil, err - } - - var result []controller.Request - for _, r := range resp.Resources { - decodedResource, err := resource.Decode[*pbexample.Bar](r) - if err != nil { - return nil, err - } - - // Only add Bar resources that match Foo by name. - if decodedResource.GetData().GetFooName() == res.Id.Name { - result = append(result, controller.Request{ID: r.Id}) - } - } -} -``` - -This approach is fine for cases when the number of `Bar` resources in a cluster is relatively small. If it's not, -then we'd be doing a large `O(N)` search on each `Bar` event which could be too expensive. - -#### Caching Mappers - -For cases when `N` is too large, we'd want to use a caching layer to help us make lookups more efficient so that they -don't require an `O(N)` search of potentially all cluster resources. - -The controller runtime contains a controller cache and the facilities to keep the cache up to date in response to watches. Additionally there are dependency mappers provided for querying the cache. - -_While it is possible to not use the builtin cache and manage state in dependency mappers yourself, this can get quite complex and reasoning about the correct times to track and untrack relationships is tricky to get right. Usage of the cache is therefore the advised approach._ - -At a high level, the controller author provides the indexes to track for each watchedtype and can then query thosfunc fooFromArgs(args ...any) ([]byte, error)e indexes in the { - -}future. The querying can occur during both dependency mapping and during resource reconciliation. - -The following example shows how to configure the "bar" controller to rereconcile a Bar resource whenever a Foo resource is changed that references the Bar - -```go -func fooReferenceFromBar(r *resource.DecodedResource[*pbexample.Bar]) (bool, []byte, error) { - idx := index.IndexFromRefOrID(&pbresource.ID{ - Type: pbexample.FooType, - Tenancy: r.Id.Tenancy, - Name: r.Data.GetFooName(), - }) - - return true, idx, nil -} - -func barController() controller.Controller { - fooIndex := indexers.DecodedSingleIndexer( - "foo", - index.ReferenceOrIDFromArgs, - fooReferenceFromBar, - ) - - return controller.NewController("bar", pbexample.BarType, fooIndex). - WithWatch( - pbexample.FooType, - dependency.CacheListMapper(pbexample.BarType, fooIndex.Name()), - ). - WithReconciler(barReconciler{}) -} -``` - -The controller will now reconcile Bar type resources whenever the Foo type resources they reference are updated. No further tracking is necessary as changes to all Bar types will automatically update the cache. - -One limitation of the cache is that it only has knowledge about the current state of resources. That specifically means that the previous state is forgotten once the cache observes a write. This can be problematic when you want to reconcile a resource to no longer take into account something that previously reference it. - -Lets say there are two types: `Baz` and `ComputedBaz` and a controller that will aggregate all `Baz` resource with some value into a single `ComputedBaz` object. When -a `Baz` resource gets updated to no longer have a value, it should not be represented in the `ComputedBaz` resource. The typical way to work around this is to: - -1. Store references to the resources that were used during reconciliation within the computed/reconciled resource. For types computed by controllers and not expected to be written directly by users a `bound_references` field should be added to the top level resource types message. For other user manageable types the references may need to be stored within the Status field. - -2. Add a cache index to the watch of the computed type (usually the controllers main managed type). This index can use one of the indexers specified within the [`internal/controller/cache/indexers`](../../../internal/controller/cache/indexers/) package. That package contains some builtin functionality around reference indexing. - -3. Update the dependency mappers to query the cache index *in addition to* looking at the current state of the dependent resource. In our example above the `Baz` dependency mapper could use the [`MultiMapper`] to combine querying the cache for `Baz` types that currently should be associated with a `ComputedBaz` and querying the index added in step 2 for previous references. - -#### Footgun: Needing Bound References - -When an interior (mutable) foreign key pointer on watched data is used to -determine the resources's applicability in a dependency mapper, it is subject -to the "orphaned computed resource" problem. - -(An example of this would be a ParentRef on an xRoute, or the Destination field -of a TrafficPermission.) - -When you edit the mutable pointer to point elsewhere, the DependencyMapper will -only witness the NEW value and will trigger reconciles for things derived from -the NEW pointer, but side effects from a prior reconcile using the OLD pointer -will be orphaned until some other event triggers that reconcile (if ever). - -This applies equally to all varieties of controller: - -- creates computed resources -- only updates status conditions on existing resources -- has other external side effects (xDS controller writes envoy config over a stream) - -To solve this we need to collect the list of bound references that were -"ingredients" into a computed resource's output and persist them on the newly -written resource. Then we load them up and index them such that we can use them -to AUGMENT a mapper event with additional maps using the OLD data as well. - -We have only actively worked to solve this for the computed resource flavor of -controller: - -1. The top level of the resource data protobuf needs a - `BoundReferences []*pbresource.Reference` field. - -2. Use a `*resource.BoundReferenceCollector` to capture any resource during - `Reconcile` that directly contributes to the final output resource data - payload. - -3. Call `brc.List()` on the above and set it to the `BoundReferences` field on - the computed resource before persisting. - -4. Use `indexers.BoundRefsIndex` to index this field on the primary type of the - controller. - -5. Create `boundRefsMapper := dependency.CacheListMapper(ZZZ, boundRefsIndex.Name())` - -6. For each watched type, wrap its DependencyMapper with - `dependency.MultiMapper(boundRefsMapper, ZZZ)` - -7. That's it. - -This will cause each reconcile to index the prior list of inputs and augment -the results of future mapper events with historical references. - -### Custom Watches - -In some cases, we may want to trigger reconciles for events that aren't generated from CRUD operations on resources, for example -when Envoy proxy connects or disconnects to a server. Controller-runtime allows us to setup watches from -events that come from a custom event channel. Please see [xds-controller](https://github.com/hashicorp/consul/blob/ecfeb7aac51df8730064d869bb1f2c633a531522/internal/mesh/internal/controllers/xds/controller.go#L40-L41) for examples of custom watches. - -## Statuses - -In many cases, controllers would need to update statuses on resources to let the user know about the successful or unsuccessful -state of a resource. - -These are the guidelines that we recommend for statuses: - -* While status conditions is a list, the Condition type should be treated as a key in a map, meaning a resource should not have two status conditions with the same type. -* Controllers need to both update successful and unsuccessful conditions states. This is because we need to make sure that we clear any failed status conditions. -* Status conditions should be named such that the `True` state is a successful state and `False` state is a failed state. - -## Best Practices - -Below is a list of controller best practices that we've learned so far. Many of them are inspired by [kubebuilder](https://book.kubebuilder.io/reference/good-practices). - -* Avoid monolithic controllers as much as possible. A single controller should only manage a single resource to avoid complexity and race conditions. -* If using cached mappers, aim to write (update or delete entries) to mappers in the `Reconcile` method and read from them in the mapper functions used by watches. -* Fetch all data in the `Reconcile` method and avoid caching it from the mapper functions. This ensures that we get the latest data for each reconciliation. diff --git a/docs/v2-architecture/controller-architecture/guide.md b/docs/v2-architecture/controller-architecture/guide.md deleted file mode 100644 index 1c26838a6a6..00000000000 --- a/docs/v2-architecture/controller-architecture/guide.md +++ /dev/null @@ -1,564 +0,0 @@ -# Resource and Controller Developer Guide - -This is a whistle-stop tour through adding a new resource type and controller to -Consul 🚂 - -## Resource Schema - -Adding a new resource type begins with defining the object schema as a protobuf -message, in the appropriate package under [`proto-public`](../../../proto-public). - -```shell -$ mkdir proto-public/pbfoo/v1alpha1 -``` - -```proto -// proto-public/pbfoo/v1alpha1/foo.proto -syntax = "proto3"; - -import "pbresource/resource.proto"; -import "pbresource/annotations.proto"; - -package hashicorp.consul.foo.v1alpha1; - -message Bar { - option (hashicorp.consul.resource.spec) = {scope: SCOPE_NAMESPACE}; - - string baz = 1; - hashicorp.consul.resource.ID qux = 2; -} -``` - -```shell -$ make proto -``` - -Next, we must add our resource type to the registry. At this point, it's useful -to add a package (e.g. under [`internal`](../../../internal)) to contain the logic -associated with this resource type. - -The convention is to have this package export variables for its type identifiers -along with a method for registering its types: - -```Go -// internal/foo/types.go -package foo - -import ( - "github.com/hashicorp/consul/internal/resource" - pbv1alpha1 "github.com/hashicorp/consul/proto-public/pbfoo/v1alpha1" - "github.com/hashicorp/consul/proto-public/pbresource" -) - -func RegisterTypes(r resource.Registry) { - r.Register(resource.Registration{ - Type: pbv1alpha1.BarType, - Scope: resource.ScopePartition, - Proto: &pbv1alpha1.Bar{}, - }) -} -``` -Note that Scope reference the scope of the new resource, `resource.ScopePartition` -mean that resource will be at the partition level and have no namespace, while `resource.ScopeNamespace` mean it will have both a namespace -and a partition. - -Update the `NewTypeRegistry` method in [`type_registry.go`] to call your -package's type registration method: - -[`type_registry.go`]: ../../../agent/consul/type_registry.go - -```Go -import ( - // … - "github.com/hashicorp/consul/internal/foo" - // … -) - -func NewTypeRegistry() resource.Registry { - // … - foo.RegisterTypes(registry) - // … -} -``` - -That should be all you need to start using your new resource type. Test it out -by starting an agent in dev mode: - -```shell -$ make dev -$ consul agent -dev -``` - -You can now use [grpcurl](https://github.com/fullstorydev/grpcurl) to interact -with the [resource service](../../../proto-public/pbresource/resource.proto): - -```shell -$ grpcurl -d @ \ - -plaintext \ - -protoset pkg/consul.protoset \ - 127.0.0.1:8502 \ - hashicorp.consul.resource.ResourceService.Write \ -< **Warning** -> Writing a status to the resource will cause it to be re-reconciled. To avoid -> infinite loops, we recommend dirty checking the status before writing it with -> [`resource.EqualStatus`]. - -[`resource.EqualStatus`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/resource#EqualStatus - -### Watching Other Resources - -In addition to watching their "managed" resources, controllers can also watch -resources of different, related, types. For example, the service endpoints -controller also watches workloads and services. - -```Go -func barController() controller.Controller { - return controller.NewController("bar", pbv1alpha1.BarType). - WithWatch(pbv1alpha1.BazType, controller.MapOwner) - WithReconciler(barReconciler{}) -} -``` - -The second argument to `WithWatch` is a [dependency mapper] function. Whenever a -resource of the watched type is modified, the dependency mapper will be called -to determine which of the controller's managed resources need to be reconciled. - -[`dependency.MapOwner`] is a convenience function which causes the watched -resource's [owner](#ownership--cascading-deletion) to be reconciled. - -[dependency mapper]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#DependencyMapper -[`dependency.MapOwner`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller/dependency#MapOwner - -### Placement - -By default, only a single, leader-elected, replica of each controller will run -within a cluster. Sometimes it's necessary to override this, for example when -you want to run a copy of the controller on each server (e.g. to apply some -configuration to the server whenever it changes). You can do this by changing -the controller's placement. - -```Go -func barController() controller.Controller { - return controller.NewController("bar", pbv1alpha1.BarType). - WithPlacement(controller.PlacementEachServer) - WithReconciler(barReconciler{}) -} -``` - -> **Warning** -> Controllers placed with [`controller.PlacementEachServer`] generally shouldn't -> modify resources (as it could lead to race conditions). - -[`controller.PlacementEachServer`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#PlacementEachServer - -### Initializer - -If your controller needs to execute setup steps when the controller -first starts and before any resources are reconciled, you can add an -Initializer. - -If the controller has an Initializer, it will not start unless the -Initialize method is successful. The controller does not have retry -logic for the initialize method specifically, but the controller -is restarted on error. When restarted, the controller will attempt -to execute the initialization again. - -The example below has the controller creating a default resource as -part of initialization. - -```Go -package foo - -import ( - "context" - - "github.com/hashicorp/consul/internal/controller" - pbv1alpha1 "github.com/hashicorp/consul/proto-public/pbfoo/v1alpha1" - "github.com/hashicorp/consul/proto-public/pbresource" -) - -func barController() controller.Controller { - return controller.ForType(pbv1alpha1.BarType). - WithReconciler(barReconciler{}). - WithInitializer(barInitializer{}) -} - -type barInitializer struct{} - -func (barInitializer) Initialize(ctx context.Context, rt controller.Runtime) error { - _, err := rt.Client.Write(ctx, - &pbresource.WriteRequest{ - Resource: &pbresource.Resource{ - Id: &pbresource.ID{ - Name: "default", - Type: pbv1alpha1.BarType, - }, - }, - }, - ) - if err != nil { - return err - } - - return nil -} -``` - -### Finalizer - -A finalizer allows a controller to execute teardown logic before a -resource is deleted. This can be useful to perform cleanup or block -deletion until certain conditions are met. - -Finalizers are encoded as keys within a resource's metadata map. It -is the responsibility of each controller that adds a finalizer to a -resource to remove the finalizer when it is marked for deletion. -Once a resource has no finalizers present, it is deleted by the -resource service. - -When the `Delete` endpoint is called on a resource with one or more -finalizers, the resource is marked for deletion by adding an immutable -`deletionTimestamp` key to the resource's metadata map. The resource is -now effectively frozen and will only accept subsequent `Write`s -that remove finalizers. `WriteStatus` is still allowed. - -The `resource` package API can be used to manage finalizers and -check whether a resource has been marked for deletion. You would -typically use this API within the logic of your controller's -`Reconcile` method to either put a finalizer in place or perform -cleanup and then remove a finalizer. Don't forget to `Write` your -changes once you add or remove finalizers. - -```Go -package resource - -// IsMarkedForDeletion returns true if a resource has been marked for deletion, -// false otherwise. -func IsMarkedForDeletion(res *pbresource.Resource) bool { ... } - -// HasFinalizers returns true if a resource has one or more finalizers, false otherwise. -func HasFinalizers(res *pbresource.Resource) bool { ... } - -// HasFinalizer returns true if a resource has a given finalizer, false otherwise. -func HasFinalizer(res *pbresource.Resource, finalizer string) bool { ... } - -// AddFinalizer adds a finalizer to the given resource. -func AddFinalizer(res *pbresource.Resource, finalizer string) { ... } - -// RemoveFinalizer removes a finalizer from the given resource. -func RemoveFinalizer(res *pbresource.Resource, finalizer string) { ... } - -// GetFinalizers returns the set of finalizers for the given resource. -func GetFinalizers(res *pbresource.Resource) mapset.Set[string] { ... } -``` - -Example flow in a controller's `Reconcile` method -```Go -const finalizer = "consul.io/bar-finalizer" - -func (barReconciler) Reconcile(ctx context.Context, rt controller.Runtime, req controller.Request) error { - ... - // Check if resource is marked for deletion. If yes, perform cleanup, remove finalizer, and Write the resource - if resource.IsMarkedForDeletion(res) { - // Perform some cleanup... - return EnsureFinalizerRemoved(ctx, rt, res, finalizer) - } - - // Check if resource has finalizer. If not, add it and Write the resource - if err := EnsureHasFinalizer(ctx, rt, res, finalizer); err != nil { - return err - } -} -``` - -## Ownership & Cascading Deletion - -The resource service implements a lightweight `1:N` ownership model where, on -creation, you can mark a resource as being "owned" by another resource. When the -owner is deleted, the owned resource will be deleted too. - -```Go -client.Write(ctx, &pbresource.WriteRequest{ - Resource: &pbresource.Resource{, - Owner: ownerID, - // … - }, -}) -``` - -## Testing - -Now that you have created your controller its time to test it. The types of tests each controller should have and boiler plat for test files is documented [here](./testing.md) diff --git a/docs/v2-architecture/controller-architecture/raft-backend.png b/docs/v2-architecture/controller-architecture/raft-backend.png deleted file mode 100644 index 42499fa86a5..00000000000 Binary files a/docs/v2-architecture/controller-architecture/raft-backend.png and /dev/null differ diff --git a/docs/v2-architecture/controller-architecture/testing.md b/docs/v2-architecture/controller-architecture/testing.md deleted file mode 100644 index 6c49c40ecee..00000000000 --- a/docs/v2-architecture/controller-architecture/testing.md +++ /dev/null @@ -1,219 +0,0 @@ -# Controller Testing - -For every controller we want to enable 3 types of testing. - -1. Unit Tests - These should live alongside the controller and utilize mocks and the controller.TestController. Where possible split out controller functionality so that other functions can be independently tested. -2. Lightweight integration tests - These should live in an internal//test package. These tests utilize the in-memory resource service and the standard controller manager. There are two types of tests that should be created. - * Lifecycle Integration Tests - These go step by step to modify resources and check what the controller did. They are meant to go through the lifecycle of resources and how they are reconciled. Verifications are typically intermingled with resource updates. - * One-Shot Integration Tests - These tests publish a bunch of resources and then perform all the verifications. These mainly are focused on the controller eventually converging given all the resources thrown at it and aren't as concerned with any intermediate states resources go through. -3. Container based integration tests - These tests live along with our other container based integration tests. They utilize a full multi-node cluster (and sometimes client agents). There are 3 types of tests that can be created here: - * Lifecycle Integration Tests - These are the same as for the lighweight integration tests. - * One-shot IntegrationTests - These are the same as for the lightweight integration tests. - * Upgrade Tests - These are a special form of One-shot Integration tests where the cluster is brought up with some original version, data is pushed in, an upgrade is done and then we verify the consistency of the data post-upgrade. - - -Between the lightweight and container based integration tests there is a lot of duplication in what is being tested. For this reason these integration test bodies should be defined as exported functions within the apigroups test package. The container based tests can then import those packages and invoke the same functionality with minimal overhead. - -For one-shot integration tests, functions to do the resource publishing should be split from functions to perform the verifications. This allows upgrade tests to publish the resources once pre-upgrade and then validate that their correctness post-upgrade without requiring rewriting them. - -Sometimes it may also be a good idea to export functions in the test packages for running a specific controllers integration tests. This is a good idea when the controller will use a different version of a dependency in Consul Enterprise to allow for the enterprise implementations package to invoke the integration tests after setting up the controller with its injected dependency. - -## Unit Test Template - -These tests live alongside controller source. - -```go -package foo - -import ( - "testing" - - "github.com/stretchr/testif/mock" - "github.com/stretchr/testif/require" - "github.com/stretchr/testif/suite" -) - -func TestReconcile(t *testing.T) { - rtest.RunWithTenancies(func(tenancy *pbresource.Tenancy) { - suite.Run(t, &reconcileSuite{tenancy: tenancy}) - }) -} - -type reconcileSuite struct { - suite.Suite - - tenancy *pbresource.Tenancy - - ctx context.Context - ctl *controller.TestController - client *rtest.Client - - // Mock objects needed for testing -} - -func (suite *reconcileSuite) SetupTest() { - suite.ctx = testutil.TestContext(suite.T()) - - // Alternatively it is sometimes useful to use a mock resource service. For that - // you can use github.com/hashicorp/consul/grpcmocks.NewResourceServiceClient - // to create the client. - client := svctest.NewResourceServiceBuilder(). - // register this API groups types. Also register any other - // types this controller depends on. - WithRegisterFns(types.Register). - WithTenancies(suite.tenancy). - Run(suite.T()) - - // Build any mock objects or other dependencies of the controller here. - - // Build the TestController - suite.ctl = controller.NewTestController(Controller(), client) - suite.client = rtest.NewClient(suite.ctl.Runtime().Client) -} - -// Implement tests on the suite as needed. -func (suite *reconcileSuite) TestSomething() { - // Setup Mock expectations - - // Push resources into the resource service as needed. - - // Issue the Reconcile call - suite.ctl.Reconcile(suite.ctx, controller.Request{}) -} -``` - -## Integration Testing Templates - -These tests should live in internal//test. For these examples, assume the API group under test is named `foo` and the latest API group version is v2. - -### `run_test.go` - -This file is how `go test` knows to execute the tests. These integration tests should -be executed against an in-memory resource service with the standard controller manager. - -```go -// Copyright (c) HashiCorp, Inc. -// SPDX-License-Identifier: BUSL-1.1 - -package footest - -import ( - "testing" - - "github.com/hashicorp/consul/internal/foo" - "github.com/hashicorp/consul/internal/controller/controllertest" - "github.com/hashicorp/consul/internal/resource/reaper" - rtest "github.com/hashicorp/consul/internal/resource/resourcetest" - "github.com/hashicorp/consul/proto-public/pbresource" -) - -var ( - // This makes the CLI options available to control timing delays of requests. The - // randomized timings helps to build confidence that regardless of resources writes - // occurring in quick succession, the controller under test will eventually converge - // on its steady state. - clientOpts = rtest.ConfigureTestCLIFlags() -) - -func runInMemResourceServiceAndControllers(t *testing.T) pbresource.ResourceServiceClient { - t.Helper() - - return controllertest.NewControllerTestBuilder(). - // Register your types for the API group and any others that these tests will depend on - WithResourceRegisterFns(types.Register). - WithControllerRegisterFns( - reaper.RegisterControllers, - foo.RegisterControllers, - ).Run(t) -} - -// The basic integration test should operate mostly in a one-shot manner where resources -// are published and then verifications are performed. -func TestControllers_Integration(t *testing.T) { - client := runInMemResourceServiceAndControllers(t) - RunFooV2IntegrationTest(t, client, clientOpts.ClientOptions(t)...) -} - -// The lifecycle integration test is typically more complex and deals with changing -// some values over time to cause the controllers to do something differently. -func TestControllers_Lifecycle(t *testing.T) { - client := runInMemResourceServiceAndControllers(t) - RunFooV2LifecycleTest(t, client, clientOpts.ClientOptions(t)...) -} - -``` - -### `test_integration_v2.go` - - -```go -package footest - -import ( - "embed" - "fmt" - "testing" - - rtest "github.com/hashicorp/consul/internal/resource/resourcetest" - "github.com/hashicorp/consul/proto-public/pbresource" -) - -var ( - //go:embed integration_test_data - testData embed.FS -) - -// Execute the full integration test -func RunFooV2IntegrationTest(t *testing.T, client pbresource.ResourceServiceClient, opts ...rtest.ClientOption) { - t.Helper - - PublishFooV2IntegrationTestData(t, client, opts...) - VerifyFooV2IntegrationTestResults(t, client) -} - -// PublishFooV2IntegrationTestData publishes all the data that needs to exist in the resource service -// for the controllers to converge on the desired state. -func PublishFooV2IntegrationTestData(t *testing.T, client pbresource.ResourceServiceClient, opts ...rtest.ClientOption) { - t.Helper() - - c := rtest.NewClient(client, opts...) - - // Publishing resources manually is an option but alternatively you can store the resources on disk - // and use go:embed declarations to embed the whole test data filesystem into the test binary. - resources := rtest.ParseResourcesFromFilesystem(t, testData, "integration_test_data/v2") - c.PublishResources(t, resources) -} - -func VerifyFooV2IntegrationTestResults(t *testing.T, client pbresource.ResourceServiceClient) { - t.Helper() - - c := rtest.NewClient(client) - - // Perform verifications here. All verifications should be retryable except in very exceptional circumstances. - // This could be in a retry.Run block or could be retryed by using one of the WaitFor* methods on the rtest.Client. - // Having them be retryable will prevent flakes especially when the verifications are run in the context of - // a multi-server cluster where a raft follower hasn't yet observed some change. -} -``` - -### `test_lifecycle_v2.go` - -```go -// Copyright (c) HashiCorp, Inc. -// SPDX-License-Identifier: BUSL-1.1 - -package footest - -import ( - "testing" - - rtest "github.com/hashicorp/consul/internal/resource/resourcetest" - "github.com/hashicorp/consul/proto-public/pbresource" -) - -func RunFooV2LifecycleIntegrationTest(t *testing.T, client pbresource.ResourceServiceClient, opts ...rtest.ClientOption) { - t.Helper() - - // execute tests. -} -``` diff --git a/docs/v2-architecture/service-mesh/README.md b/docs/v2-architecture/service-mesh/README.md deleted file mode 100644 index 4d0b6e9ff27..00000000000 --- a/docs/v2-architecture/service-mesh/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# V2 Service Mesh Architecture - -In Consul 1.16 and 1.17 releases, Consul's service mesh has been rewritten to use the controller architecture and the -resource APIs. - -At a high level, the service mesh consists of resources and controllers living in three groups: `catalog`, `mesh`, -and `auth`. - -![controllers diagram](controllers.png) - -The controllers in each groups are responsible for producing an output that may then be used by other controllers. - --> **Note:** This diagram is valid as of Consul 1.17. It may change in the future releases. - -## Catalog controllers - -Catalog controllers are responsible for reconciling resources in the `catalog` API group. - -1. **FailoverPolicy** controller validates `FailoverPolicy` resource has valid service references and updating the - status the `FailoverPolicy` resource with the result. -2. **WorkloadHealth** controller takes in workloads and any relevant health statuses and updates the status of a - workload with the combined health status. -3. **NodeHealth** controller takes in nodes and any relevant health statuses and updates the status of a node with the - combined health status. -4. **ServiceEndpoints** controller generates a `ServiceEndpoints` object that is name-aligned with a service and - contains workload addresses and ports that constitute a service. - -## Mesh Controllers - -1. **ProxyConfiguration** controller generates a `ComputedProxyConfiguration` resource that is name-aligned with a - workload. `ComputedProxyConfiguration` contains all merged `ProxyConfiguration` resources that apply to a specific - workload. -2. **Routes** controller generates a `ComputedRoutes` resource that is name-aligned with a service. It contains merged - configuration from all xRoutes objects as well as `FailoverPolicy` and `DestinationPolicy`. -3. **ExplicitDestinations** controller generates a `ComputedExplicitDestinations` resource that is name-aligned with a - workload. It contains merged `Destinations` resources that apply to a specific workload. -4. **SidecarProxy** controller takes in the results of the previous three controllers as well as some user-provided - resources and generates `ProxyStateTemplate` resource which serves as the representation of Envoy configuration for - sidecar proxies. -5. **XDSController** takes in `ProxyStateTemplate` resources, fills in missing endpoints references as well as - certificates and CA roots and sends them over to another component that sends this information over to Envoy. - -## Auth Controllers - -1. **TrafficPermissions** controller generates `ComputedTrafficPermissions` resource that is name-aligned - with `WorkloadIdentity`. This computed resource contains all traffic permissions that apply to a specific workload - identity. \ No newline at end of file diff --git a/docs/v2-architecture/service-mesh/controllers.png b/docs/v2-architecture/service-mesh/controllers.png deleted file mode 100644 index 76efac1d059..00000000000 Binary files a/docs/v2-architecture/service-mesh/controllers.png and /dev/null differ