Server Monitoring with Zabbix

For a long time, I have stuck on many hosted monitoring softwares to avoid management effort on
a self-hosted monitoring system. However, free stuffs cannot go long so I must move from services to services. This makes me tired! Thus I decided to return to a self-hosted solution so that I do not need to move from time to time. Currently, the number of servers/hosts that I need to manage is gradually over 100, so I must come to a decision soon. After reviewing many current solutions, my favorite Zabbix does still stand on a top position of every recommended list. So the decision is quite simple: Go monitoring with Zabbix 🙂

Long story short: This entry is for installing and configuration Zabbix for my self-hosted monitoring purpose.

I. Zabbix Server & Zabbix Frontend for centralized monitoring purpose

I decided to go with Debian 8, a lightweight, enterprise-reader OS. And the most important reason: the server I choose to go is with Debian 8, and why do I need to waste time to reinstall it :lol ?

• Install packages for Debian 8 (replace jessie – Debian 8 with stretch – Debian 9 or wheezy – Debian 7 ):

  [bash]wget http://repo.zabbix.com/zabbix/3.4/debian/pool/main/z/zabbix-release/zabbix-release_3.4-1+jessie_all.deb
  dpkg -i zabbix-release_3.4-1+jessie_all.deb
  apt-get update
  apt-get install -y sudo nmap[/bash]

• Install Zabbix Server and Zabbix Front-end (I use MySQL as the database):

  [bash]apt-get install -y zabbix-server-mysql zabbix-frontend-php[/bash]

• Add zabbix user to /etc/sudoers (you can use visudo to edit):

  [bash]zabbix ALL=(ALL) NOPASSWD: ALL[/bash]

• Create MySQL database for Zabbix services (replace your password in ):

  [bash]mysql -uroot -p
  create database zabbix character set utf8 collate utf8_bin;
  grant all privileges on zabbix.* to zabbix@localhost identified by ‘<password>’;
  quit;[/bash]

• Import initial Zabbix schema (you will need to input your password):

  [bash]zcat /usr/share/doc/zabbix-server-mysql/create.sql.gz | mysql -uzabbix -p zabbix[/bash]

• Update Zabbix configuration at /etc/zabbix/zabbix_server.conf:

  [bash]DBHost=localhost
  DBName=zabbix
  DBUser=zabbix
  DBPassword=<password>[/bash]

• Change the timezone configuration at /etc/apache2/conf-enabled/zabbix.conf
• Start and enable Zabbix services:

  [bash]service zabbix-server start
  update-rc.d zabbix-server enable
  systemctl restart apache2[/bash]

• Login to web interface at http://IP/zabbix/ with default user Admin and password is zabbix

II. Install Zabbix agent on nodes to be monitored

1. Install Zabbix Agent on agent nodes

   Again, I chose Debian (Ubuntu will be the same), so if you install on other OS, remember to replace according commands (e.g. replace apt-get with yum for CentOS / Fedora)

   • To install Zabbix agent, run

     [bash]wget http://repo.zabbix.com/zabbix/3.4/debian/pool/main/z/zabbix-release/zabbix-release_3.4-1+jessie_all.deb
     dpkg -i zabbix-release_3.4-1+jessie_all.deb
     apt-get update
     apt-get install -y zabbix-agent[/bash]

   • We will use Pre-Shared Keys (PSK) to secure the connection between the server and agent. So let’s generate a key (we will need the key to configure the host later):

     [bash]openssl rand -hex 32 > /etc/zabbix/zabbix_agentd.psk[/bash]

   • We need to edit the Zabbix agent settings (/etc/zabbix/zabbix_agentd.conf) to set up its secure connection to the Zabbix server. We will need to edit the following configuration items (be sure to note TLSPSKIdentity and the value of TLSPSKFile to use later on Zabbix Web Interface):

     [bash]Server=ZABBIX_SERVER_IP_ADDRESS
     Hostname=nocix64g2670v2-01
     TLSConnect=psk
     TLSAccept=psk
     TLSPSKIdentity=PSK-nocix64g2670v2-01
     TLSPSKFile=/etc/zabbix/zabbix_agentd.psk[/bash]

   • Start and enable Zabbix agent, run

     [bash]systemctl start zabbix-agent
     systemctl enable zabbix-agent[/bash]

2. Add the new host to the Zabbix server

   Installing Zabbix agent is just a halfway to get things done. We will also need to add the according host to our Zabbix server.

   • Access to your Zabbix web interface
   • Locate to Configuration >> Host >> Create host, then
     • Input the Hostname and IP ADDRESS params to reflect your client machine. You can also create Group to group many servers there
     • Click the Templates sub-menu, Type “Template OS Linux” and select it in “Link new templates“. Remember to click on the Add button to add it to the template list.
     • Next, navigate to the Encryption tab, select PSK for “Connections to host” and “Connections from host“, then
       • set PSK identity to the value that we set in TLSPSKIdentity in the previous step (PSK-nocix64g2670v2-01 in the above case)
       • set PSK value to the key you generated for the Zabbix agent in the previous set(at /etc/zabbix/zabbix_agentd.psk)
   • Finally, click Add button and wait for several minutes 🙂

III. Secure Zabbix server with Let’s Encrypt

• Install packages:

  [bash]echo ‘deb http://ftp.debian.org/debian jessie-backports main’ | tee /etc/apt/sources.list.d/backports.list
  apt-get update
  apt-get install -y python-certbot-apache -t jessie-backports[/bash]

• Set Up the Apache ServerName and ServerAlias: edit /etc/apache2/sites-available/000-default.conf and put ServerName and ServerAlias information. For example:

  [bash] ServerName tienle.com
  ServerAlias www.tienle.com[/bash]

• Restart apache:

  [bash]service apache2 restart[/bash]

• Setup the SSL certification process:

  [bash]certbot –apache[/bash]

• After finishing, remember to setup crontab to auto-renew the certificate for each month / week with the certbot renew command line.

IV. To be considered: Zabbix Proxy

Zabbix Proxy can help off-load Zabbix servers in case we monitor thousands of nodes, so I do not really need it this case. In case you decide to go with Zabbix Proxy, one note worth remembering is to use a separated database from your Zabbix server if you install them on the same server.

V. Zabbix Notes

• Remember to associate a user with a Media type (e.g. SMS notification, Email notification, etc.
  which can be configured at Administration >> Media Types menu) so that alert notifications can be sent
• You can change anything of a template (triggers, applications, etc.) via the menu Configuration >> Templates >> select the associated template
• Lazy setup on Guest machines with ZonePing? Run

  [bash]wget –no-check-certificate -N https://www.tienle.com/tools/tool-script-install-zabbix-agent-zoneping.sh && bash tool-script-install-zabbix-agent-zoneping.sh
  [/bash]

• Wanna monitor disk performance? Use templates from zabbix-disk-performance as follows:
  • On the ZB Agent nodes:

    # low level discovery script 
    wget https://raw.githubusercontent.com/grundic/zabbix-disk-performance/master/lld-disks.py -O /usr/local/bin/lld-disks.py
    chmod +x /usr/local/bin/lld-disks.py
    # diskstats user parameters config
    mkdir -p /etc/zabbix/zabbix_agentd.d/
    wget https://raw.githubusercontent.com/grundic/zabbix-disk-performance/master/userparameter_diskstats.conf -O /etc/zabbix/zabbix_agentd.d/userparameter_diskstats.conf
    # restart service
    service zabbix-agent restart

  • Login to the ZB Web Interface and import the Template Disk Performance.
  • Then add the template to your according hosts:
    • Go to Configuration >> Templates >> Template Disk Performance, then add according hosts to the field “Hosts / templates” >> Update.
    • OR Go to Configuration >> Hosts >> Your host, and then go to Templates tab, Add Template Disk Performance to the current host.
  • Done, wait several minutes before checking  associated graphs.
• Reserved for later