What is ELK Stack?
To understand ELK first we need to know what is “Observability of an application or its deployment status” – From an operational perspective I need to know stability, availability, and usability status of an application. This is particularly important from a service level perspective where downtime and errors need to be quickly handled and understood by knowing its causes and fixing issues – these could be there during development in a CI/CD environment or production
We use ELK stack and collectively using 3 different open source tools – Elasticsearch, Logstash and Kibana to solve the observability problem.
For effective observability we need to collect logs, identify metrics and track events or traces of errors which could be relevant for problem identifications and resolutions if any.
System logs could be generated from Windows or Linux system logs, database logs such as MySQL and application and server logs from Apache, Tomcat etc.. In cloud native environments log analysis and log monitoring could be an important initiative for DevOps.
Logstash enables as a collector, processor and forwarder of logs – and very helpful in case of multiple servers on multiple hosts and troubleshooting for errors because of a consolidated view and search capability
Setup and Installation
Kibana – https://www.elastic.co/guide/en/kibana/current/install.html
Elasticsearch – https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html
Logstash – https://www.elastic.co/downloads/logstash
Configuration
Configure all servers logs whether its tomcat, MySQL or other servers such as application logs, or other tools such as redis, kafka etc.. (refer https://www.elastic.co/guide/en/logstash/current/config-examples.html)
Benefits and Usage
- Integrate logs from multiple micro services for advanced analytics
- Error count and drill down
- Analytics for Specific api and charts counter etc…
- Collecting and viewing metrics