Monitoring

Monitoring architecture

Our monitoring architecture looks like this:

        flowchart TB
    admin(["`👤 **System admin**`"])

    subgraph alerts [Alerts]
        direction TB
        mattermost["`💬 **Mattermost**
        Team chat`"]
        pushover["`📱 **Pushover**
        Push notifications`"]
    end

    uptime["`🚦 **Uptime Kuma**
    *dockerised, central*
    Service monitoring`"]

    grafana["`📊 **Grafana**
    *dockerised, central*
    Dashboard & alert manager`"]

    prometheus["`🧮 **Prometheus**
    *dockerised, central*
    Metrics collector`"]

    loki["`📄 **Loki**
    *dockerised, central*
    Log aggregator`"]

    alloy["`🧲 **Alloy**
    *dockerised, per-host*
    Log & metrics collector`"]

    node_exporter["`🖥 **Node Exporter**
    *native, per-host*
    Machine metrics exporter`"]

    admin -. runs queries .-> grafana
    uptime -- sends alerts --> alerts
    grafana -- sends alerts --> alerts
    grafana -. PromQL ..-> prometheus
    grafana -. LogQL ..-> loki
    prometheus -- scrapes metrics --> node_exporter
    prometheus <-- sends metrics --> alloy
    loki <-- sends logs --> alloy

    classDef notification fill:#62f69b,stroke:#2e9e5b,color:#000
    classDef docker fill:#acd2e0,stroke:#4f8aa3,color:#000
    classDef native fill:#f4b4b9,stroke:#b5666d,color:#000

    class mattermost,pushover notification
    class uptime,grafana,prometheus,loki,alloy docker
    class node_exporter native

    %% Alloy sends to Prometheus & Loki, but sits on the per-host level, so only show the start arrows.
    linkStyle 6,7 marker-end:none
    

Monitoring alerts

Monitoring alerts of the Service monitoring & Infrastructure monitoring will be sent to the Monitoring channel in Mattermost, and via Pushover.

Hint

Alerts are deactivated / silenced during the 🗓 Maintenance windows.

Links for the Pushover subscriptions can be found in the Uptime monitoring alerts & Grafana monitoring alerts chapters.

Service monitoring

Uptime Kuma

We’re using Uptime Kuma to monitor all of our services.

Important

Please note we’ve a dedicated public status site based on Uptime Kuma.

Infrastructure monitoring

Grafana

We’re using Grafana OSS as dashboard & alert manager to:

• Access Prometheus metrics via PromQL

• Access Loki logs via LogQL

• Visualise metrics & logs

• Use it as alert manager to send notifications to Mattermost

Prometheus

We’re using Prometheus as metrics collector to:

• Scrape OS, and 🐳 Docker metrics

• Scrape application metrics

• Persist those metrics

• Query, aggregate, and correlate metrics via Grafana

Loki

We’re using Grafana Loki as log receiver to:

• Retrieve OS, and 🐳 Docker logs

• Retrieve application logs

• Persist those logs

• Query, aggregate, and correlate logs via Grafana

Alloy

We’re using Grafana Alloy collectors to:

• Collect OS, and 🐳 Docker logs

• Retrieve 🐳 Docker metrics

• Forward them to Loki, and Prometheus

Node Exporter

We’re using Node Exporters to:

• Expose OS metrics to Prometheus

Note

The deployment, and all documentation for the Node Exporter can be found in the Ansible node_exporter role.