feat: loki integration#118
Open
SimonRichardson wants to merge 7 commits into
Open
Conversation
jujubot
added a commit
to juju/juju
that referenced
this pull request
May 19, 2026
#22414 This is the logging domain which we will track the loki endpoint for loki_push_api changes from. It will take values from the juju-controller charm and push them into the database. From there it will dispatch events to a worker to replace the underlying loggers with the correct slog handler, which can then pipe to loki. Requires juju/juju-controller#118 ## QA steps This isn't wired up yet. ## Links **Jira card:** [JUJU-9763](https://warthogs.atlassian.net/browse/JUJU-9763) [JUJU-9763]: https://warthogs.atlassian.net/browse/JUJU-9763?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
Uses loki integration to push the loki endpoint to juju database, so it can send all the information to all the agents. This will allow the a much more distributed logging approach.
iyiguncevik
reviewed
Jun 19, 2026
gfouillet
reviewed
Jun 19, 2026
jujubot
added a commit
to juju/juju
that referenced
this pull request
Jun 19, 2026
#22692 It was noted by @hpidcock that the absence of a CA cert doesn't actually indicate that we should skip verification of a certificate. They're orthogonal. This change fixes this, by introducing the ability to set that field independently of the certificate. There will be a follow up patch in the controller charm that will allow you to skip the verification. Requires juju/juju-controller#118 ## QA steps Ensure that you've got the PR charm packed in the right location juju/juju-controller#118 ```sh $ juju bootstrap lxd src --bootstrap-base=ubuntu@24.04 --controller-charm-path=./juju-controller_ubuntu@24.04-amd64.charm --build-agent $ juju switch controller $ juju config controller loki-insecure-skip-verify=true ``` Pack the charm and deploy: ``` $ cd ./testcharms/charms/observability $ charmcraft pack $ juju deploy observability ``` Add the relation: ``` $ juju config observability loki-url="http://foo.com" $ juju integrate controller:loki-push-api observability:loki-push-api $ juju ssh -m controller 0 $ juju_db_repl repl (controller)> SELECT prdesc FROM logging_loki_config uuid endpoint ca_cert insecure_skip_verify 0f6a774f-7504-4c8e-86c3-5afbffcc663f http://foo.com true ``` Remove the relation: ``` $ juju remove-relation controller:loki-push-api observability:loki-push-api $ juju ssh -m controller 0 $ juju_db_repl repl (controller)> SELECT prdesc FROM logging_loki_config uuid endpoint ca_cert insecure_skip_verify ``` ## Links It's not quite the following task, but will be part of it for now, as it's just a fix-up to existing code. **Jira card:** [JUJU-9980](https://warthogs.atlassian.net/browse/JUJU-9980) [JUJU-9980]: https://warthogs.atlassian.net/browse/JUJU-9980?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
iyiguncevik
approved these changes
Jun 22, 2026
jujubot
added a commit
to juju/juju
that referenced
this pull request
Jun 25, 2026
#22717 Prior to this, we just managed to set the loki endpoints into the database. This how ever, now sets the loki values in the agent config. Thus, when the agent comes up, it will talk to the right loki endpoint. We do not currently ensure that the endpoint is correct if it's changed, that'll be follow up work. Requires juju/juju-controller#118 ## QA steps Ensure that you've got the PR charm packed in the right location juju/juju-controller#118 ```sh $ juju bootstrap lxd src --bootstrap-base=ubuntu@24.04 --controller-charm-path=./juju-controller_ubuntu@24.04-amd64.charm --build-agent $ juju switch controller $ juju config controller loki-insecure-skip-verify=true ``` Pack the charm and deploy: ``` $ cd ./testcharms/charms/observability $ charmcraft pack $ juju deploy observability ``` Add the relation: ``` $ juju config observability loki-url="http://foo.com" $ juju integrate controller:loki-push-api observability:loki-push-api $ juju ssh -m controller 0 $ juju_db_repl repl (controller)> SELECT prdesc FROM logging_loki_config uuid endpoint ca_cert insecure_skip_verify 0f6a774f-7504-4c8e-86c3-5afbffcc663f http://foo.com true ``` Deploy a charm to another model and ensure the endpoint is there: ```sh $ juju add-model foo $ juju deploy juju-qa-test $ juju ssh 0 $ sudo cat /var/lib/juju/agents/machine-0/agent.conf | grep "loki" lokiendpoint: http://foo.com lokiinsecureskipverify: true ``` ## Links **Jira card:** [JUJU-9983](https://warthogs.atlassian.net/browse/JUJU-9983) [JUJU-9983]: https://warthogs.atlassian.net/browse/JUJU-9983?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
We need an org id for some loki setups, so it needs to be set in the controller config.
iyiguncevik
approved these changes
Jun 29, 2026
jujubot
added a commit
to juju/juju
that referenced
this pull request
Jul 1, 2026
#22744 ~~Requires #22741 to land first.~~ Provides loki integration for IAAS models. I say IAAS models, as I've not yet tested on CAAS, which will come in a follow up PR, as this is already getting large. The major change is the introduction of an orgID (aka tenant), which is required for some loki deployments. This can be configured with the controller charm. It's empty by default. What you should see is once the loki integrate is created, is all logs **except** for model logs being forwarded to loki. There will be a follow patch for the model logs issue. ## QA steps This setups a loki locally, work on COS stack is in progress, expect that to follow: https://grafana.com/docs/loki/latest/setup/install/docker/ You'll need juju/juju-controller#118 built and in the root of juju, along with the observability charm found in `testcharms/charms`. ```sh $ juju bootstrap lxd src --bootstrap-base=ubuntu@24.04 --controller-charm-path=./juju-controller_ubuntu@24.04-amd64.charm --build-agent $ juju switch controller $ juju config controller loki-org-id=tenant1 $ juju config controller loki-insecure-skip-verify=true $ juju deploy ./observability_amd64.charm $ juju config observability loki-url="http://<IP ADDRESS>:3100/loki/api/v1/push" $ juju integrate controller:loki-push-api observability:loki-push-api ``` See: <img width="1987" height="1038" alt="Screenshot from 2026-06-29 17-22-10" src="https://github.com/user-attachments/assets/2bb04524-67cb-4608-b18b-821714de17c2" /> ## Links **Jira card:** [JUJU-10075](https://warthogs.atlassian.net/browse/JUJU-10075) [JUJU-10075]: https://warthogs.atlassian.net/browse/JUJU-10075?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
jujubot
added a commit
to juju/juju
that referenced
this pull request
Jul 3, 2026
#22770 This change makes model-scoped logging a first-class path in Juju. The problem we needed to solve was that logging is no longer tied to a single fixed destination. A model's logs may need to move between the controller log sink, Loki, or drain mode as configuration changes. Model workers should not need to know about those backend changes, and they must keep the correct model identity attached to every record. Model loggers exist to solve that. They give model-scoped workers a stable logging interface, while the logrouter and logsink handle where the logs actually go. That keeps the responsibility in the right place: workers log to "the model", and the logging infrastructure decides how that model's logs are routed. Unfortunately, there is a problem - a circular dependency that prevents us from using the `logrouter` directly inside of the `logsink`. The dependency graph is as follows: log-router -> api-caller -> api-server -> log-sink -> log-router What is more of a challenge, is the duality of the jujud binary. It can serve both the controller and non-controller agents alike. This will change in the future, but we don't want to depend on that change for 4.1 (there might be complications). On a non-controller machine, the normal path is fine: model logs go through the local logrouter, then over the API to the controller, which writes them to the active destination. The API server is remote, so there is no feedback loop. On a controller machine, that same shape is unsafe. The controller hosts the API server and the log sink locally, so sending controller-local model logs through the API-backed path can loop back into the same logging pipeline. That is why the controller needs a separate local path. In logsink mode, controller model logs must be able to go directly to a controller-local backend instead of re-entering through the API surface. The logsinkproxy exists to hide that split. The rest of the system still wants one dependency: “give me the model logger/log sink for this machine.” Without the proxy, every consumer would need to know whether it was running on a controller and pick different manifolds and interfaces accordingly. The proxy keeps the dual wiring in one place and presents one stable output. Thus ifController and ifNotController let the machine manifolds wire different logging workers depending on whether the agent is running on a controller machine. That split is necessary because controllers need a controller-local logging path to avoid the API/logsink loop, while non-controllers can safely use the normal API-backed path. ```mermaid flowchart LR subgraph Left["Problem: controller using API-backed path"] A1[Controller model worker] --> B1[Model logger] B1 --> C1[Logrouter] C1 --> D1[API caller] D1 --> E1[Controller API server] E1 --> F1[Log sink] F1 --> C1 end subgraph Right["Fix: split controller and non-controller paths"] A2[Model worker] --> B2[Model logger] B2 --> C2[Controller logrouter] C2 --> D2[Controller-local log sink or Loki/drain backend] B2 --> E2[Non-controller logrouter] E2 --> F2[API caller] F2 --> G2[Controller API server] G2 --> H2[Log sink or Loki/drain backend] end Left ~~~ Right ``` ## QA steps This setups a loki locally, work on COS stack is in progress, expect that to follow: https://grafana.com/docs/loki/latest/setup/install/docker/ You'll need juju/juju-controller#118 built and in the root of juju, along with the observability charm found in `testcharms/charms`. ```sh $ juju bootstrap lxd src --bootstrap-base=ubuntu@24.04 --controller-charm-path=./juju-controller_ubuntu@24.04-amd64.charm --build-agent $ juju switch controller $ juju config controller loki-org-id=tenant1 $ juju config controller loki-insecure-skip-verify=true $ juju deploy ./observability_amd64.charm $ juju config observability loki-url="http://<IP ADDRESS>:3100/loki/api/v1/push" $ juju integrate controller:loki-push-api observability:loki-push-api ``` You should see the model logs. ## Links <!-- Place JIRA number in both places below. --> **Jira card:** [JUJU-10075](https://warthogs.atlassian.net/browse/JUJU-10075) [JUJU-10075]: https://warthogs.atlassian.net/browse/JUJU-10075?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requires #107 to land first. Although, this isn't explicitly tied to s3 integration, it does build upon some of the charm logic within it.Uses the loki_push_api requirer integration to send the loki push URL to the controller charm. Once it has that, it will be up to the controller to distribute the loki endpoint to all agents, so that we can send logging to loki, instead of jujud-controller. This should elevate a lot of load in busy controllers.
Note
Don't review the
lib/charms/loki_k8s/v1/loki_push_api.pyfile, it's from upstream.