Monitoring Kubernetes with Prometheus & Grafana

1. The Anatomy of a Kubernetes Monitoring Stack

Before we deploy anything, let's understand the components we're about to install. The kube-prometheus-stack isn't just one or two tools; it's a collection of services that work together to provide end-to-end monitoring.

The key components in this architecture are:

• Prometheus: A time-series database that scrapes and stores metrics. It's the core of our monitoring solution.
• Grafana: A powerful visualization tool used to create dashboards from the data stored in Prometheus.
• Prometheus Operator: This is a crucial controller that simplifies the management of Prometheus on Kubernetes. Instead of manually editing complex configuration files, the operator allows you to manage Prometheus declaratively using Kubernetes Custom Resources like ServiceMonitor and PrometheusRule.
• Alertmanager: Handles alerts sent by Prometheus. It deduplicates, groups, and routes them to the correct receiver (like email, Slack, or PagerDuty).
• Exporters: These are services that expose metrics in a format Prometheus can understand. The stack includes two vital ones out of the box:
  • Node Exporter: A DaemonSet that runs on every node to expose machine-level metrics (CPU, memory, disk, network).
  • kube-state-metrics: A service that listens to the Kubernetes API and generates metrics about the state of objects like Deployments, Pods, and Services.

To get a detailed walkthrough of these components and how they fit together, the following video provides an excellent introduction.

2. Deploying the kube-prometheus-stack with Helm

Now, let's get our hands dirty and deploy the stack. Given your experience with Helm, we'll use the official prometheus-community/kube-prometheus-stack chart.

After running the helm install command, you can verify that all the components are starting up by running:

kubectl get pods -n monitoring -w

It may take a few minutes for all pods to be in the Running state. Once they are, you have a fully functional monitoring stack!

3. Accessing and Exploring the Stack

With the stack running, let's access the UIs for Grafana and Prometheus to see what we get out of the box. We will use kubectl port-forward for local access. In a production environment, you would typically use an Ingress.

The video you watched earlier also demonstrates this process perfectly.

Getting the Grafana Admin Password

The default password for Grafana can be retrieved reliably with the following command:

kubectl get secret --namespace monitoring prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Log in to Grafana with the username admin and the password you just retrieved. You'll be amazed at the wealth of information available in the pre-configured dashboards.

4. Customizing the Deployment

A default installation is great for a demo, but for any real-world use case, you'll need to customize the configuration. This is done by providing a custom values.yaml file to the helm install or helm upgrade command.

Let's look at how to configure a few key areas: Prometheus metrics retention and adding a custom Grafana dashboard declaratively.

4.1 Prometheus Retention and Storage

By default, Prometheus may only store metrics for a short period. For production, you'll want to configure persistence and a longer retention policy.

4.2 Configuring Custom Dashboards with ConfigMaps

You can, and should, manage your Grafana dashboards declaratively. The kube-prometheus-stack chart is configured to automatically discover dashboards from ConfigMaps that have the label grafana_dashboard: "1".

5. Monitoring Your Own Applications with ServiceMonitor

So far, we've monitored the cluster itself. But how do you tell Prometheus to scrape metrics from your own applications? This is where the Prometheus Operator and the ServiceMonitor CRD shine.

Instead of manually editing a central Prometheus configuration file, you create a ServiceMonitor resource in the same namespace as your application. This resource declaratively defines how Prometheus should find and scrape your application's metrics endpoint.

The key takeaway is this declarative, decoupled approach:

1. Your application's deployment includes a Service with a specific label (e.g., app: my-fastapi-app).
2. You create a ServiceMonitor that uses a selector to find services with that label.
3. The Prometheus Operator sees the ServiceMonitor and automatically reconfigures the underlying Prometheus instance to start scraping the endpoints of the pods behind that service.

Conclusion

In this lesson, you've deployed a production-grade monitoring stack on Kubernetes. You've seen how the combination of Helm and the Prometheus Operator provides a powerful, declarative framework for managing observability.

Key Takeaways:

• The kube-prometheus-stack Helm chart is the industry standard for deploying a comprehensive monitoring solution, including Prometheus, Grafana, and key exporters.
• The Prometheus Operator is the engine that enables declarative management of the monitoring configuration through Custom Resource Definitions (CRDs).
• ServiceMonitor is the CRD used to declaratively tell Prometheus how to discover and scrape metrics from your applications, fitting perfectly with the GitOps philosophy.
• Grafana dashboards can and should be managed declaratively as code using ConfigMaps, allowing them to be version-controlled in Git.

What's Next?

You've now mastered collecting and visualizing metrics, one of the three pillars of observability. In the next lesson, "Deploy Fluent Bit as a DaemonSet for cluster-wide log collection and forwarding," we will tackle the second pillar: logging. You'll learn how to set up a lightweight, efficient agent to collect logs from all nodes and applications in your cluster and forward them to a central location.