Monitoring NGINX Ingress Controller
Introducedin GitLab 9.5
GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built in Prometheus metrics included inversion 0.9.0and above of the ingress.
Requirements
Prometheus integrationmust be active.
Metrics supported
Name | Query |
---|---|
Throughput (req/sec) | sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code) |
Latency (ms) | avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}) |
HTTP Error Rate (HTTP Errors / sec) | sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) |
Configuring NGINX ingress monitoring
If you have deployed NGINX Ingress using GitLab'sKubernetes cluster integration, it willautomatically be monitoredby Prometheus.
For other deployments, there issome configurationrequired depending on your installation:
- NGINX Ingress should be version 0.9.0 or above, with metrics enabled
- NGINX Ingress should be annotated for Prometheus monitoring
- Prometheus should be configured to monitor annotated pods
About managed NGINX Ingress deployments
NGINX Ingress is deployed into thegitlab-managed-apps
namespace, using theofficial Helm chart.NGINX Ingress will beexternally reachable via the Load Balancer's IP.
NGINX is configured for Prometheus monitoring, by setting:
enable-vts-status: "true"
, to export Prometheus metricsprometheus.io/scrape: "true"
, to enable automatic discoveryprometheus.io/port: "10254"
, to specify the metrics port
When used in conjunction with the GitLab deployed Prometheus service, response metrics will be automatically collected.
Manually setting up NGINX Ingress for Prometheus monitoring
Version 0.9.0 and above ofNGINX ingresshave built-in support for exporting Prometheus metrics. To enable, a ConfigMap setting must be passed:enable-vts-status: "true"
.一旦启用,Prometheus metrics endpoint will start running on port 10254.
Next, the ingress needs to be annotated for Prometheus monitoring. Two new annotations need to be added:
prometheus.io/scrape: "true"
prometheus.io/port: "10254"
Managing these settings depends on how NGINX ingress has been deployed. If you have deployed via theofficial Helm chart, metrics can be enabled withcontroller.stats.enabled
along with the required annotations. Alternatively it is possible edit the NGINX ingress YML directly in theKubernetes dashboard.
Specifying the Environment label
In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do this, GitLab will search for metrics with appropriate labels. In this case, theupstream
标签必须的形式
.
If you have usedAuto Deployto deploy your app, this format will be used automatically and metrics will be detected with no action on your part.