Setting Up Notification Channels for Email and Slack in Grafana

1. Prerequisites

Before we begin, ensure the following:

  • Grafana is installed and running.
  • We have administrative access to the Grafana server.
  • For Email notifications, an SMTP server is available and configured.
  • For Slack notifications, we have a Slack workspace with permission to create an Incoming Webhook.

2. Configuring the Notification Channel for Email

Step 1: Configure SMTP Settings in Grafana

To send emails, Grafana must be configured with an SMTP server.

  1. Edit the Grafana Configuration File: Locate the grafana.ini file (typically in /etc/grafana/ on Linux systems).

  2. Set SMTP Configuration: Modify the [smtp] section of the grafana.ini file:

    [smtp]
enabled = true
host = smtp.example.com:587  # Replace with our SMTP server and port
user = our-email@example.com  # SMTP username
password = our-password  # SMTP password
from_address = grafana@example.com  # The email address Grafana will use
from_name = Grafana Alerts
startTLS_policy = Yes  # Enable STARTTLS if supported by our SMTP server

  3. Restart Grafana: Apply the changes by restarting the Grafana service:

    sudo systemctl restart grafana-server

  4. Test SMTP Configuration: Use the test option when creating a notification channel (covered below).

Step 2: Create an Email Notification Channel

  1. Open Notification Channels:

    • Navigate to Grafana’s sidebar.
    • Go to Alerting > Notification channels.

  2. Create a New Channel:

    • Click New Channel.

  3. Configure Email Settings:

    • Name: Enter a name (e.g., Email Alerts).
    • Type: Select Email.
    • Email addresses: Enter recipient email addresses, separated by commas.
    • Customizations:
      • Configure templates or add additional options if needed.

  4. Test Notification:

    • Use the Send Test button to ensure the email setup works.

  5. Save the Channel:

    • Click Save.

3. Configuring the Notification Channel for Slack

Step 1: Create a Slack Incoming Webhook

  1. Log In to Slack:

    • Go to our Slack workspace and log in with an account that has admin privileges.

  2. Create a Webhook:

    • Visit the Slack API page for Incoming Webhooks.
    • Click Create an App and give it a name (e.g., Grafana Alerts).
    • Choose the workspace where we want to send alerts.

  3. Set Up Webhook URL:

    • In the app configuration page, go to Incoming Webhooks and click Activate Incoming Webhooks.
    • Click Add New Webhook to Workspace.
    • Select the channel where alerts will be sent (e.g., #alerts).
    • Copy the Webhook URL.

Step 2: Create a Slack Notification Channel in Grafana

  1. Open Notification Channels:

    • Navigate to Grafana’s sidebar.
    • Go to Alerting > Notification channels.

  2. Create a New Channel:

    • Click New Channel.

  3. Configure Slack Settings:

    • Name: Enter a name (e.g., Slack Alerts).
    • Type: Select Slack.
    • Webhook URL: Paste the Webhook URL obtained from Slack.
    • Optional Fields:
      • Mention Users: Use @username to mention specific users.
      • Icon Emoji: Customize the icon for alerts using Slack emojis (e.g., :bell:).

  4. Test Notification:

    • Use the Send Test button to ensure the Slack setup works.

  5. Save the Channel:

    • Click Save.

4. Applying Notification Channels to Alerts

Step 1: Create or Edit an Alert Rule

  1. Navigate to the dashboard where we want to create an alert.
  2. Edit the panel, go to the Alert tab, and click Create Alert.
  3. Set conditions for the alert (e.g., trigger when CPU usage exceeds 80%).
  4. Add the notification channel (Email or Slack) created earlier.

Step 2: Test the Alert

  • Ensure the conditions are met to trigger the alert.
  • Verify the notification is received via Email or Slack.

5. Troubleshooting

Email Notifications:

  • Ensure SMTP settings in grafana.ini are correct.
  • Check the Grafana logs for errors (/var/log/grafana/grafana.log).

Slack Notifications:

  • Verify the Webhook URL is correct.
  • Ensure the Grafana server has internet access to send requests to Slack.

By following these steps, we can set up and test notification channels for Email and Slack in Grafana, ensuring we receive timely alerts for our monitoring needs.