Skip to main content

🚀 Prometheus Alertmanager Setup Guide

🛠️ Installation​

sudo apt update && sudo apt install -y prometheus-alertmanager

✅ Verify Installation​

prometheus-alertmanager --version

📝 Configuring Alertmanager​

Edit the Alertmanager configuration file:

sudo nano /etc/prometheus/alertmanager.yml

🔧 alertmanager.yml Configuration​

global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'your-app-password'  # ⚠️ Do not use plain text passwords, use environment variables!

templates:
  - '/etc/prometheus/alertmanager_templates/*.tmpl'

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: team-X-mails

  routes:
  - match:
      service: my-email-test
    receiver: email-notifications

  - match_re:
      service: ^(foo1|foo2|baz)$
    receiver: team-X-mails
    routes:
    - match:
        severity: critical
      receiver: team-X-pager

  - match:
      service: files
    receiver: team-Y-mails
    routes:
    - match:
        severity: critical
      receiver: team-Y-pager

  - match:
      service: database
    receiver: team-DB-pager
    group_by: [alertname, cluster, database]
    routes:
    - match:
        owner: team-X
      receiver: team-X-pager
    - match:
        owner: team-Y
      receiver: team-Y-pager

inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  equal: ['alertname', 'cluster', 'service']

receivers:
- name: 'team-X-mails'
  email_configs:
  - to: '[email protected]'

- name: 'team-X-pager'
  email_configs:
  - to: '[email protected]'
  pagerduty_configs:
  - service_key: <team-X-key>

- name: 'team-Y-mails'
  email_configs:
  - to: '[email protected]'

- name: 'team-Y-pager'
  pagerduty_configs:
  - service_key: <team-Y-key>

- name: 'team-DB-pager'
  pagerduty_configs:
  - service_key: <team-DB-key>

- name: 'email-notifications'
  email_configs:
  - to: '[email protected]'
    from: '[email protected]'
    smarthost: 'smtp.gmail.com:587'
    auth_username: '[email protected]'
    auth_identity: '[email protected]'
    auth_password: 'your-app-password'
    require_tls: true

🔄 Restart Alertmanager​

sudo systemctl restart prometheus-alertmanager

📜 Alert Rules Configuration​

Edit the alert rules file:

sudo nano /etc/prometheus/alert.rules.yml

⚠️ alert.rules.yml​

groups:
  - name: example-alert-rules
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
          service: my-email-test
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "No data received from {{ $labels.instance }} for over 1 minute"

🔧 Configuring Prometheus to Use Alertmanager​

Edit the Prometheus configuration file:

sudo nano /etc/prometheus/prometheus.yml

📢 prometheus.yml​

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 'localhost:9093'

rule_files:
  - "/etc/prometheus/alert.rules.yml"

🔄 Restart Prometheus​

sudo systemctl restart prometheus

🌍 Access Prometheus Alerts​

📌 Open the following URL in your browser: http://15.207.86.27:9090/alerts

🎯 Key Takeaways​

✅ Installed and configured Prometheus Alertmanager ✅ Set up SMTP for email notifications 📧 ✅ Defined alerting routes and inhibition rules ✅ Created an alert rule to detect down instances 🚨 ✅ Configured Prometheus to use Alertmanager

Now your system is ready to send alerts when an issue arises! 🎉