Import lumberjack events manually with stdin

A typical install of logstash-forwarder (lumberjack) is configured to watch a set of files in specific locations and often playing with that file is impossible. However, you might need to load a file into it that it doesn’t typically monitor.

In another situation, you may need to load historic logfiles into LSF. This can be problematic as LSF keeps track of its position in a given file and will often recognise the file as one it has already processed and won’t  reimport events it considers as “old”.

So here is a quick way of getting events in without interrupting your log shipping.

1. Create a new config file somewhere where the user you run LSF as can read it e.g. /etc/logstash-forwarder/temp.conf
2. Add a bare-bones config with your remote server and a single stdin input: { “network”: { “servers”: [ “10.0.0.10:5043” ], “ssl certificate”: “/etc/logstashforwarder/ssl/logstashforwarder.crt”, “ssl ca”: “/etc/logstashforwarder/ssl/ca.crt”, “ssl key”: “/etc/logstashforwarder/ssl/logstashforwarder.key”, “timeout”: 15 }, “files”: [ { “paths”: [ “-” ], “fields”: { “type”: “nginx” } } ] }
3. cat your logfile into a *new *instance of LSF with the config above like so:cat /var/log/nginx/temp/server.access | /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/temp.conf -spool-size 100 -log-to-syslog
4. You can watch syslog to see if your events are being shipped by tailing syslog.
5. ???
6. Profit.

You can shut down the temp instance once the flood of events dies down.

Cheers!