Skip to main content

Monitoring

Signadot Operator exposes two Prometheus endpoints that can be used to collect metrics about the status of the application.

Controller Manager Metrics

The Controller Manager exports metrics corresponding to in-cluster CRDs and controllers. You can find these via the following endpoint.

https://signadot-controller-manager-metrics-service.signadot.svc:8443/metrics

This endpoint is protected by kube-rbac-proxy, to let your Prometheus server scrape it, you will need to:

  1. Grant the required permissions to the service account used by Prometheus:
kubectl create clusterrolebinding signadot-metrics-reader --clusterrole=signadot-metrics-reader --serviceaccount=<namespace>:<service-account-name>
  1. Instruct Prometheus to:

    1. Skip the validation of the server certificate for our endpoint (given it uses a self-sign certificate).

    2. Send the Authorization header with a bearer token read from /var/run/secrets/kubernetes.io/serviceaccount/token (the service account token).

There are different ways to configure the above settings.
If you are using Prometheus Operator, you can create the following ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
control-plane: controller-manager
name: controller-manager-metrics-monitor
namespace: signadot
spec:
endpoints:
- path: /metrics
port: https
scheme: https
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
control-plane: controller-manager

If you are using Datadog Agent, you apply the following patch to the signadot-controller-manager deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
name: signadot-controller-manager
spec:
template:
metadata:
annotations:
ad.datadoghq.com/kube-rbac-proxy.check_names: |
["openmetrics"]
ad.datadoghq.com/kube-rbac-proxy.init_configs: |
[{}]
ad.datadoghq.com/kube-rbac-proxy.instances: |
[{
"openmetrics_endpoint": "https://%%host%%:8443/metrics",
"namespace": "signadot",
"metrics": [".*"],
"auth_token": {
"reader": {
"type": "file",
"path": "/var/run/secrets/kubernetes.io/serviceaccount/token"
},
"writer": {
"type": "header",
"name": "Authorization",
"value": "Bearer <TOKEN>",
"placeholder": "<TOKEN>"
}
},
"tls_verify": "false"
}]

This endpoint also includes all the default metrics from kubebuilder project as documented in the metrics reference.

Tunnel Proxy Metrics

The Tunnel Proxy exposes an HTTP endpoing at:

http://tunnel-proxy.signadot.svc:8001/metrics

The following metrics are exported:

Metrics nameTypeDescription
inbound_connectionsGaugeTotal number of active inbound connections by tunnel method
inbound_revtunsGaugeTotal number of inbound reverse tunnels by tunnel method