Recently I came across a complex Logstash configuration in which I needed to use multiple times the same input plugins types (jdbc) that will read from different sources. In addition I needed to use the same output plugin type (elastic search) multiple times, writing to different destinations (different indexes on the same elastic search instance). And last, I wanted to split up the configuration in multiple smaller fragments for maintenance.

The first thing I did was reading the manual (duh) and saw the option of specifying a directory with a wildcard to logstash:

logstash -f /some/path/*.conf

In each of those files, I configured a complete pipeline (input, filter, output). Unfortunately, this didn’t work for the second pipeline. The first was run, the second not. As it seems, Logstash is simply concatenating all the files in lexicographical order (my first was called dev-jdbc.conf, the second test-jdbc.conf). I came across this post, which confirmed my experiences. The option suggested there was to use multiple Logstash instances.

I hate that.

So I tried to cut the files down into several start and end files per section (input, filter, output), giving me this:

000-intput-start.conf
.....
199-input-end.conf
200-filter-start.conf
...
399-filter-end.conf
400-output-start.conf
...
599-output-end.conf

Between those sections, I added de ‘normal’ content:

001-dev-jdbc.conf
002-test-jdbc.conf
401-dev-elastic.conf
402-test-elastic.conf

Resulting in this ordered list:

000-intput-start.conf
001-dev-jdbc.conf
002-test-jdbc.conf
199-input-end.conf
200-filter-start.conf
399-filter-end.conf
400-output-start.conf
401-dev-elastic.conf
402-test-elastic.conf
599-output-end.conf

Of course, I needed to use ‘type’ in the inputs to tag the sources and IF’s in the outputs to select the right ones.

Unfortunately, when I started Logstash, assuming that all files will be concatenated in order, Logstash complained right at the first file that is was invalid. It needs to be a complete valid file. I didn’t get it to work with simply

input {

So then I made a custom *nix startup script to concatenate all the files in order into once single config file:

rm $LS_CONF_DIR/complete.conf
cat $(ls $LS_CONF_DIR/*.conf) > $LS_CONF_DIR/complete.conf
echo "Reading Logstash config from "$LS_CONF_DIR"/complete.conf"
exec $LS_EXEC -f ${LS_CONF_DIR}/complete.conf

And this works like a charm! Now Logstash is reading from several sources and writing to several targets, and I still am able to seperate different configs in small fragments.