Beats is a companion product of Elasticsearch. It is defined as single purpose, lightweight data shippers. Each type of beat is designed to load a very specific data type into Elasticsearch. The Beats family includes Filebeat, which I covered in more detail in this post, Metricbeat, Packetbeat and Winlogbeat. Those are the official beats from Elasticsearch. But this is an open architecture and there are plenty of other beat created by the community that cover many other types of system. Those can be seen here.
This post will focus on Metricbeat, which is designed to collect metrics from various systems and visualize them as Kibana dashboards to make it easier to monitor those systems. Metricbeat was evolved from the Topbeat which was able to visualize the results of the top Linux/Unix command, but Metricbeat can do much more thanks to its modular design. It can collect metric data from a variety of sources and push them to Elasticsearch. There are modules for system data, Apache, Docker, Kafka, MySQL, MongoDB, PostgreSQL, Zookeeper and many more. Later this data is visualized as Kibana dashboards.
Assuming you already have the elasticsearch yum repository set up from previous step, run:
yum install metricbeat
Now edit the metricbeat configuration file at /etc/metricbeat/metricbeat.yml
The file has two main sections: The modules section that specifes which modules will be active and what data Metricbeat will gather, and the output sections that specifies where Metricbeat should send its findings – Elasticsearch or Logstash.
The configuration file I used for this post can be found here (I only used system and MongoDB modules). You should, of course, change the host names according to your setup. This is a minimal file that contains only what I needed, but there are many more options.
In the same directory where the configuration file is, you can find a file named metricbeat.full.yml that contains all the possible modules and output options, so you can see in it examples for other modules, for logging options and for Logstash output.
You can find more information on configuring MetricBeat here.
This handles the data Metricbeat gathers and where it sends it. Now we want to take care of its visualization in Kibana. Metricbeat comes with pre-built example templates that we should upload to Kibana before we can see any dashboards. There are two options for doing it.
Metricbeat is shipped with a template file that is /etc/metricbeat/metricbeat.template.json
It will automatically load the template file at startup if those options are defined in the metricbeat.yml config file in the Elasticsearch output section:
template.enabled: true template.name: "metricbeat" template.path: "/etc/metricbeat/metricbeat.template.json" # Overwrite if template already exist template.overwrite: true
You can also manually upload the latest template directly from Elasticsearch servers using the import_dashboards script that is located under the Metricbeat binary directory. The “es” parameter is the url of Elasticsearch where the templates should be loaded:
/usr/share/metricbeat/scripts/import_dashboards -es http://192.168.56.101:9200
Now let’s go and see how it looks like in Kibana at http://<hostname>:5601
If this is the first time you enter Kibana, it will ask you to specify a default index pattern. you can enter “metricbeat-*”. If you already have a different default index pattern then you can find “metricbeat-*” under management -> index patterns:
If you click on it you can see a very long list of fields, that corresponds to the collected metrics. You can later add those metrics to your custom dashboards:
Metricbeats comes with basic dashboards that are good only for demonstration. If you want to see its true power you should customize those dashboards.
Just for example, we will test the system module which is the default and MongoDB module, but there are many more available.
In order to access the Metricbeat dashboards, go to dashboards -> open and you will see a list of available dashboards to the left:
When you click one of the above links, the dashboard opens. For example, this is the CPU dashboard:
Now let’s try MongoDB. My MongoDB is quite idle so we need to create some action in order to see something happening in the dashboard. I used this small bash script to create a 30 Million lines csv file:
#!/bin/bash MAXLINES=$1 COUNTER=1 echo $MAXLINES while [ "$COUNTER" -le "$MAXLINES" ] do NUMBER=$RANDOM echo $COUNTER,$NUMBER,"Line number" $COUNTER >> sampledata.csv let "COUNTER+=1" done
Then I loaded it into MongoDB using mongoimport:
mongoimport --host mongo1:27000 --db test --collection sampledata --fields line,value,message --type csv --file /root/sampledata.csv
Let’s look at the Mongo dashboard:
You can see the operation counters rising and the WiredTiger cache filling as the database imports the data. However this is not very useful and many important parameters are missing (for example replication lag). You should edit the dashboard or create your own to add those extra metrics to make it really useful.
By default, auto refresh is off. To activate it you have to choose auto-refresh from the upper menu of the dashboard and set the desired interval:
To view historical data, choose the men item just next to auto refresh and choose the time frame you want to see:
Metricbeat is a versatile framework that can collect metric data from many sources, and the list of sources continuously grows thank to independent, Third party modules. It is also easy to configure and operate and with some Kibana skills you can create great looking dashboards.
It can do some monitoring but I do not think it can compete with specialized monitoring systems like Nagios or Ganglia. A better use case will be as a tool for convenient gathering of metric data from various machine sources and storing them in one place, in order to later run analytics on this data.