[Howto] Icinga2 with Graphite and Grafana on Debian

In this blog, we describe the integration of Icinga2 into Graphite and Grafana on Debian.

What is Graphite

Graphite stores performance data over a configurable period. Services can send metrics to Graphite via a defined interface, which are then stored in a structured manner for the desired period. Possible examples of such metrics include CPU utilization or web server access numbers. Graphs can now be generated from the various metrics via Graphite’s integrated web interface. This allows us to detect and observe changes in values over different periods. A good example of such a trend analysis is disk space utilization. With the help of a trend graph, it is easy to see at what rate the space requirement is growing and approximately when a storage replacement will be necessary.

What is Grafana

Although Graphite offers its own web interface, it is not particularly attractive or flexible. This is where Grafana steps in.

Grafana is a frontend for various metric storage systems. For example, it supports Graphite, InfluxDB, and OpenTSDB. Grafana offers an intuitive interface for creating representative graphs from metrics. It also has a variety of functions to optimize the appearance and display of graphs. Subsequently, graphs can be grouped into dashboards. Parameterization of graphs is also possible. This also allows you to display only a graph from a specific host.

Installing Icinga2

At this point, only the installation required for Graphite is described. Current versions of Icinga2 packages for Debian can be obtained directly from the Debmon Project. The Debmon Project, run by official Debian package maintainers, provides current versions of various monitoring tools for Debian releases in a timely manner. To integrate these packages, the following commands are required:

# add debmon
cat  <<EOF  >/etc/apt/sources.list.d/debmon.list
deb http://debmon.org/debmon debmon-jessie main
EOF

# add debmon key
wget -O - http://debmon.org/debmon/repo.key 2>/dev/null | apt-key add -

# update repos
apt-get update

Next, we can install Icinga2:

apt-get install icinga2

Installing Graphite and Graphite-Web

After Icinga2 is installed, Graphite and Graphite-web can also be installed.

# install packages for icinga2 and graphite-web and carbon

apt-get install icinga2 graphite-web graphite-carbon libapache2-mod-wsgi apache2

Configuring Icinga2 with Graphite

Icinga2 must be configured to export all collected metrics to Graphite. The Graphite component that receives this data is called “Carbon”. In our example installation, Carbon runs on the same host as Icinga2 and also uses the default port. For this reason, no further configuration of Icinga2 is necessary; it is sufficient to enable the export.

The command does this. icinga2 feature enable graphite
Next, Icinga2 must be restarted: service icinga2 restart

If the Carbon server runs on a different host or a different port, the Icinga2 configuration can be adjusted in the file /etc/icinga2/features-enabled/graphite.conf. Details can be found in the Icinga2 documentation.

If the configuration was successful, a number of files should appear shortly in “/var/lib/graphite/whisper/icinga“. If this is not the case, you should check the Icinga2 log file (located in “/var/log/icinga2/icinga2.log“).

Configuring Graphite-web

Grafana uses Graphite’s web frontend as an interface for the metrics stored by Graphite. For this reason, it is necessary to configure Graphite-web correctly. For performance reasons, we operate Graphite-web as a WSGI module. A number of configuration steps are required for this:

1. First, we create a user database for Graphite-web. Since we will not have many users, we use sqlite as the backend for our user data at this point. For this purpose, we execute the following commands, which initialize the user database and assign it to the user under which the web frontend runs:

   graphite-manage syncdb
   chown _graphite:_graphite /var/lib/graphite/graphite.db

2. Next, we activate the WSGI module in Apache: a2enmod wsgi
3. For simplicity, the web interface should run in its own virtual host and on its own port. To ensure Apache also listens on this port, we add the line “Listen 8000” to the file “/etc/apache2/ports.conf“.
4. The Graphite Debian package already provides an Apache configuration file that we can use for our purposes, with slight modifications. cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available/graphite.conf To ensure the virtual host also uses port 8000, we must replace the line

   <VirtualHost *:80>

   with

   <VirtualHost *:8000>

   .

5. Then we activate the new virtual host via a2ensite graphite and restart Apache: systemctl restart apache2
6. Graphite-web should now be accessible at http://YOURIP:8000/. If this is not the case, the Apache log files under “/var/log/apache2/” could provide valuable information.

Configuring Grafana

Grafana is currently not included in Debian. However, the author offers an Apt repository through which Grafana can be installed. Even if the repository refers to Wheezy, the packages also work under Debian Jessie.

The repository is only accessible via HTTPS. For this reason, HTTPS support for apt must first be installed: apt-get install apt-transport-https

Next, the repository can be integrated.

# add repo (package for wheezy works on jessie)
cat  <<EOF  >/etc/apt/sources.list.d/grafana.list
deb https://packagecloud.io/grafana/stable/debian/ wheezy main
EOF
 
# add key
curl -s https://packagecloud.io/gpg.key | sudo apt-key add -
 
# update repos
apt-get update

Subsequently, the package can be installed: apt-get install grafana. For Grafana to run, we still need to enable the service systemctl enable grafana-server.service and start it systemctl start grafana-server.

Grafana is now accessible at http://YOURIP:3000/. The default username and password in our example is “admin”. This password should, of course, be replaced with a secure password at the next opportunity.

Next, Grafana must be configured to use Graphite as a data source. For simplicity, the configuration is explained via a screencast.

After successfully integrating Graphite as a data source, we can create our first graph. There is also a short screencast for this here.

Congratulations, you have now successfully installed and configured Icinga2, Graphite, and Grafana. For all further steps, please refer to the documentation of the respective projects:

• Icinga2
• Graphite
• Grafana