Centralized Logging with Graylog on Docker

Logs are the answer that when something goes wrong. When you work on an enterprise scale, you need a centralized logging mechanism. (You can’t jump one server to other and tail that streams)

For the central log management, you need something like Graylog, logstash, ELK… the list goes on.

Setting up Graylog Server

Life was tough before docker, thankfully we can setup Graylog server through docker, follow the instructions on docker page.

I prefer to use docker-compose, and something like this should work.


version: '2'
services:
# MongoDB: https://hub.docker.com/_/mongo/
mongodb:
image: mongo:3
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
environment:
http.host=0.0.0.0
transport.host=localhost
network.host=0.0.0.0
# Disable X-Pack security: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/security-settings.html#general-security-settings
xpack.security.enabled=false
"ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:2.4.0-1
environment:
# CHANGE ME!
GRAYLOG_PASSWORD_SECRET=somepasswordpepper
# Password: admin
GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
links:
mongodb:mongo
elasticsearch
depends_on:
mongodb
elasticsearch
ports:
# Graylog web interface and REST API
9000:9000
# Syslog TCP
514:514
# Syslog UDP
514:514/udp
# GELF TCP
12201:12201
# GELF UDP
12201:12201/udp

In most cases, it just works out of the box.

Protip: Graylog use MongoDB for settings, configuration, etc. and it holds log data on the Elasticsearch. So, be careful when setting RAM usage for the JVM. 

Sending the logs

There are many ways to send logs. We use monolog from the sending custom logs or WordPress level logs.

For the syslog;

(probably rsyslog pre-installed),

  • Create new file under the “/etc/rsyslog.d” directory – 90-graylog2.conf
  • *.* @SERVER_IP_ADDRESS:PORT;RSYSLOG_SyslogProtocol23Format
  • restart the service > “service rsyslog restart”
  • voila!

Conclusion

We are holding more than 1.5 billion logs on the single machine, that logs data about 1.1 TB and the logs come from WordPress, custom logs, syslog, HAProxy across the ~20 servers.

ProtipIf the logs are not that critical and you don’t have high IO rate. You might avoid using SSD disks. That will reduce the cost, and you can hold a lot of indices on the same machine.

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: