Mustafa Uysal

I'm traveling light, it's au revoir…

Centralized Logging with Graylog on Docker

Logs are the answer that when something goes wrong. When you work on an enterprise scale, you need a centralized logging mechanism. (You can’t jump one server to other and tail that streams)

For the central log management, you need something like Graylog, logstash, ELK… the list goes on.

Setting up Graylog Server

Life was tough before docker, thankfully we can setup Graylog server through docker, follow the instructions on docker page.

I prefer to use docker-compose, and something like this should work.


version: '2'
services:
# MongoDB: https://hub.docker.com/_/mongo/
mongodb:
image: mongo:3
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
environment:
– http.host=0.0.0.0
– transport.host=localhost
– network.host=0.0.0.0
# Disable X-Pack security: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/security-settings.html#general-security-settings
– xpack.security.enabled=false
– "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:2.4.0-1
environment:
# CHANGE ME!
– GRAYLOG_PASSWORD_SECRET=somepasswordpepper
# Password: admin
– GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
– GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
links:
– mongodb:mongo
– elasticsearch
depends_on:
– mongodb
– elasticsearch
ports:
# Graylog web interface and REST API
– 9000:9000
# Syslog TCP
– 514:514
# Syslog UDP
– 514:514/udp
# GELF TCP
– 12201:12201
# GELF UDP
– 12201:12201/udp

In most cases, it just works out of the box.

Protip: Graylog use MongoDB for settings, configuration, etc. and it holds log data on the Elasticsearch. So, be careful when setting RAM usage for the JVM. 

Sending the logs

There are many ways to send logs. We use monolog from the sending custom logs or WordPress level logs.

For the syslog;

(probably rsyslog pre-installed),

  • Create new file under the “/etc/rsyslog.d” directory – 90-graylog2.conf
  • *.* @SERVER_IP_ADDRESS:PORT;RSYSLOG_SyslogProtocol23Format
  • restart the service > “service rsyslog restart”
  • voila!

Conclusion

We are holding more than 1.5 billion logs on the single machine, that logs data about 1.1 TB and the logs come from WordPress, custom logs, syslog, HAProxy across the ~20 servers.

ProtipIf the logs are not that critical and you don’t have high IO rate. You might avoid using SSD disks. That will reduce the cost, and you can hold a lot of indices on the same machine.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.