What is centralized logging?

Logging helps in proper troubleshooting and diagnosis. It also provides a degree of protection since even if someone gains access to the servers, they will be unable to erase their trail because the logs are stored elsewhere.

 

Centralized logging allows collecting logs from multiple machines or applications and aggregating them in a single place. Instead of needing to login to each of your machines to examine the logs, you can view them all in one spot. It makes searching through all of the logs easier, and some systems even give a dashboard to visualize the logs.

 

EFK Stack

The EFK stack consists of

• Elasticsearch

• Fluentd

• Kibana

The three components together provide a centralised log management system. Let's have a look at what each component accomplishes.

 

Elasticsearch is a scalable search engine that is used to store all the logs. It can handle large volumes of data and the input data may be unstructured. It is based on the Apache Lucene search engine.

 

Fluentd collects logs from various sources and forwards it to Elasticsearch where it is indexed. Fluentd has a lot of available plugins which help to parse the logs. Rules can be written to specify the sources and logs are recognized by tags, which are then utilized for routing.

 

Kibana aids in the visualization and navigation of our Elasticsearch data. Making a visualization in Kibana is as simple as it gets and provides temporal and location-based visualization, as well as ML-powered anomaly detection.

 

The EFK stack is open source, heavily customizable and scalable to a large extent. You can deploy it on your server or use services like  Amazon Elasticsearch Service.