Monitoring Archive

There are two ways to authenticate yourself as a client to Icinga2. On the one hand there is the possibility to authenticate yourself by username and password. The other option is authentication using client certificates. With the automated query of the Icinga2 API, the setup of client certificates is not only safety-technically advantageous, but also in the implementation on the client side much more practical.

Unfortunately, the official Icinga2 documentation does not provide a description of the exact certificate creation process. Therefore here is a short manual:

After installing Icinga2 the API feature has to be activated first:

icinga2 feature enable api

The next step is to configure the Icinga2-node as master, the easiest way to do this is with the “node-wizard” program:

icinga2 node wizard

Icinga2 creates the necessary CA certificates with which the client certificates still to be created must be signed. Now the client certificate is created:

icinga2 pki new-cert --cn  --key .key --csr .csr

The parameter cn stands for the so-called common-name. This is the name used in the Icinga2 user configuration to assign the user certificate to the user. Usually the common name is the FQDN. In this scenario, however, this name is freely selectable. All other names can also be freely chosen, but it is recommended to use a name that suggests that the three files belong together.

Now the certificate has to be signed by the CA, Icinga2:

icinga2 pki sign-csr --csr .csr --cert .crt

Finally, the API user must be created in the file “api-user.conf”. This file is located in the subfolder of each Icinga2 configuration:

object ApiUser {
client_cn = 
permissions = []
}

For a detailed explanation of the user’s assignment of rights, it is worth taking a look at the documentation.

Last but not least Icinga2 has to be restarted. Then the user can access the Icinga2 API without entering a username and password, if he passes the certificates during the query.

You can read up on the services we provide for Icinga2 righthere.

This post was originally written by Bernd Borowski.

In this post we describe how to integrate Icinga2 with Graphite and Grafana on Debian stable (jessie).

What is Graphite?

Graphite stores performance data on a configurable period. Via a defined interface, services can send metrics to Graphite, which are then stored for the designated period. Possible examples of the type of metrics that it might be useful for are CPU utilization or traffic of a web server. Graphs can be generated from the various metrics via the integrated web interface in Graphite. This allows changes in values to be observed across different time periods. A good example of where this type of trend analysis could be useful is monitoring levels on a hard drive. By using a Trend Graph, it’s clear to see the growth rate of the footprint and anticipate when an exchange of storage will become necessary.

What is Grafana?

Graphite does have a private web interface but it is neither attractive nor flexible. This is where Grafana comes in.

Grafana is a frontend for various storage metrics, supporting Graphite, InfluxDB and OpenTSDB, among others. Grafana offers an intuitive interface through which you can create graphs to represent the metrics and a variety of functions to optimise their appearance and presentation. The graphs can then be summarized in dashboards; you can also opt to display one from a specific host only, through parameterization.

Installation of Icinga2

Here we describe the only installation which is essential for Graphite. Icinga2 packages in the current version of Debian can be obtained directly from Debmon projekt.
At the Debmon project up to date versions of the different monitoring tools software for the different Debian releases are provided by the official Debian package maintainers. To include these packages, use the following commands:

# add debmon
cat <<EOF >/etc/apt/sources.list.d/debmon.list
deb http://debmon.org/debmon debmon-jessie main
EOF

# add debmon key
wget -O - http://debmon.org/debmon/repo.key 2>/dev/null | apt-key add -

# update repos
apt-get update

Finally we can install Icinga2:

apt-get install icinga2

Installation of Graphite and Graphite-Web

Once Icinga2 is installed, Graphite and Graphite-Web can also be installed.

# install packages for icinga2 and graphite-web and carbon

apt-get install icinga2 graphite-web graphite-carbon libapache2-mod-wsgi apache2

Configuration of Icinga2 with Graphit

Icinga2 must be configured so that all defined metrics are exported to Graphite. The Graphite component receives this data called “Carbon”. In our sample installation Carbon runs on the same host as Icinga2 and uses the default port, meaning no further configuration of Icinga2 is necessary – it is enough to export on.

To do this, simply enter the command: icinga2 feature enable graphite
Subsequently Icinga2 must be restarted: service icinga2 restart

If the Carbon Server is running on a different host or a different port, the configuration of the file /etc/icinga2/features-enabled/graphite.conf Icinga2 should be adjusted. Details can be found in the Icinga2 documentation.

If the configuration was successful, then after a short time a number of files should appear in “/var/lib/graphite/whisper/icinga“. If they don’t, then you should take a look in the log file of Icinga2 (located in “/var/log/icinga2/icinga2.log“)

Configuration of Graphite – Web

Grafana uses the frontend of Graphite as an interface for the stored metrics of Graphite so correct configuration of Graphite-web is very important. For performance reasons, we operate Graphite-web as a WSGI module. A number of configuration steps are necessary:

First, we create a user database for Graphite-web. Since we will not have many users, at this point we use SQLite as the backend for our user data. We do this using the following commands which initialize the user database and transfer ownership to the user under which the Web front-end runs:
```
graphite-manage syncdb
chown _graphite:_graphite /var/lib/graphite/graphite.db
```
Then we activate the WSGI module in Apache: a2enmod wsgi
For simplicity the web interface should run in a separate virtual host and on its own port. So that Apache listens to this port, we add the line “Listen 8000” to the file “/etc/apache2/ports.conf“
The Graphite Debian package already provides a configuration file for Apache, which is fine to use with slight adaptations. cp /usr/share/graphite-web/apache2- graphite.conf /etc/apache2/sites-available/graphite.conf. In order for the virtual host to also use port 8000, we replace the line:
```
<VirtualHost *:80>
```
with:
```
<VirtualHost *:8000>
```
We can now activate the new virtual host via a2ensite graphite and restart Apache: systemctl restart apache2
You should now be able to reach Graphite-web at http://YOURIP:8000/. If you cannot, the Apache log files in “/var/log/apache2/” can provide valuable information.

Configuration of Grafana

Grafana is not currently included in Debian. However, the author offers an Apt repository from which you can install Grafana. Even if the repository points to Wheezy, the packages should still function under Debian Jessie.

The repository is only accessible via https – so first, you need to install https support for apt: apt-get install apt-transport-https

Next, the repository can be integrated.

# add repo (package for wheezy works on jessie)
cat </etc/apt/sources.list.d/grafana.list
deb https://packagecloud.io/grafana/stable/debian/ wheezy main
EOF
 
# add key
curl -s https://packagecloud.io/gpg.key | sudo apt-key add -
 
# update repos
apt-get update

After this, the package can be installed: apt-get install grafana For Grafana to run we still need to activate the service systemctl enable grafana-server.service and start systemctl start grafana-server.

Grafana is now accessible at http://YOURIP:3000/. The default user name and password in our example is ‘admin’. This password should, of course, be replaced by a secure password at the earliest opportunity.

Grafana must then be configured so that it uses Graphite as a data source. For simplicity, the configuration is explained in the following screencast:

After setting up Graphite as the data source, we can create our first graph. Here is another short screencast to illustrate this:

Congratulations! You have successfully installed and configured Icinga2, Graphite and Grafana. For subsequent steps, please refer to the documentation for the specific projects:

To read more about credativ’s work with Debian, please read our Debian blogs.