SEP for observability integration (monitoring controller)#2028
SEP for observability integration (monitoring controller)#2028mayleighnmyers wants to merge 1 commit into
Conversation
Signed-off-by: Mayleigh Tjapkes <mamyers@redhat.com>
|
Hi @mayleighnmyers. Thanks for your PR. I'm waiting for a istio-ecosystem or istio member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
| - [x] Unit tests for controller and relabeling package | ||
| - [ ] External CRD/API group detection (`monitoring.coreos.com` vs `monitoring.rhobs`) | ||
| - [ ] Kubernetes PodMonitor strategy using `namespaceSelector.any: true` | ||
| - [ ] Validation when monitoring CRDs are unavailable |
There was a problem hiding this comment.
Can we add to this document design details for this? From my pov this can be discussed here to set a base. I think we can set a Condition field with Type: MonitoringAvailable, Status: False, Reason: MissingCRDs, Message: "Prometheus Operator CRDs not found" something like this will be very helpfull
There was a problem hiding this comment.
Hi @fjglira , yes, I am working on this validation change and having a draft change.
ok, we can add status condition update in the Istio custom resources. I will update this detail in SEP later today.
|
|
||
| We assume `ServiceMonitor` and `PodMonitor` CRDs are available under `monitoring.coreos.com/v1` and/or `monitoring.rhobs/v1` group. The Sail Operator ClusterRole grants permissions for both API groups' `ServiceMonitor` and `PodMonitor` custom resources. | ||
|
|
||
| A monitoring controller watches `IstioRevision` resources and reconciles `ServiceMonitor` and `PodMonitor` CRs when the boolean field `spec.monitoring.enabled` is set to `true` in the parent `Istio` CR. It also watches namespaces with the `istio-injection=enabled` label and reconciles `PodMonitor` CRs for Istio sidecar injection namespaces. |
There was a problem hiding this comment.
This means it will only reconcile the resource for namespaces with stio-injection=enabled`? What about the revision labels used?
There was a problem hiding this comment.
Hi @mayleighnmyers , here is a doc about injection label:
https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/#controlling-the-injection-policy
Currently, we only have the istio-injection label watch. There is another label we want to watch. istio.io/rev
when the istio.io/rev label has a value , it is same as the istio-injection=enabled.
I can update the SEP part, would you like to add that additional istio.io/rev label watch in the monitoring controller code ? thanks.
| * Provide a monitoring controller that reconciles `ServiceMonitor` and `PodMonitor` resources for Istio control plane and sidecar proxy metrics. | ||
| * Apply platform-appropriate default relabeling rules on Kubernetes and OpenShift. |
There was a problem hiding this comment.
If it is out of scope, we should mention this; if not, we need to plan how metrics are gathered from node level ztunnel components and namespace level waypoint proxies
There was a problem hiding this comment.
Right, ambient is out of scope in this SEP. We want to keep changes smaller in each SEP. We can have the next SEP for ambient mode metrics.
| ## Non-goals | ||
|
|
||
| * Deploying observability stack components (Prometheus, OpenShift Cluster Observability Operator, OpenShift User-Workload Monitoring, etc.). We assume those are installed separately. | ||
| * Updating `ServiceMonitor` and `PodMonitor` for user customization of scraping paths, ports, or relabeling rules via the Sail Operator API. |
There was a problem hiding this comment.
We will need to explicitly mention this in the docs:
If users require custom relabeling or scraping configs, they must leave spec.monitoring.enabled: false, and deploy their own independent PodMonitors.
|
|
||
| A broader Observability Integration API with target references to a monitoring and tracing stacks is being discussed in the initial PR comments. We plan to design and implement the monitoring controller and broader Integration API controller separately. This enhancement proposal is focused on the monitoring controller with API change in the `Istio` Custom Resource. We will work toward a broader Integration API design and implementation in next phase. | ||
|
|
||
| ### Architecture |
There was a problem hiding this comment.
How will deletion be handled? I think it worth to mention how we plan to hanlde the deletion of the resources
There was a problem hiding this comment.
There is a concern from existing environments where users have configured their custom ServiceMonitor or PodMontior resources manually. They expect those existing resources should not be deleted from Sail Operator.
We will add a paragraph about deletion in SEP. Thanks.
4 participants
What type of PR is this?
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Related Issue/PR #
#1959
Additional information: