Recently I came across a complex Logstash configuration in which I needed to use multiple times the same input plugins types (jdbc) that will read from different sources. In addition I needed to use the same output plugin type (elastic search) multiple times, writing to different destinations (different indexes on the same elastic search instance). And last, I wanted to split up the configuration in multiple smaller fragments for maintenance.
The first thing I did was reading the manual (duh) and saw the option of specifying a directory with a wildcard to logstash:
logstash -f /some/path/*.conf
In each of those files, I configured a complete pipeline (input, filter, output). Unfortunately, this didn’t work for the second pipeline. The first was run, the second not. As it seems, Logstash is simply concatenating all the files in lexicographical order (my first was called dev-jdbc.conf, the second test-jdbc.conf). I came across this post, which confirmed my experiences. The option suggested there was to use multiple Logstash instances.
I hate that.
So I tried to cut the files down into several start and end files per section (input, filter, output), giving me this:
000-intput-start.conf ..... 199-input-end.conf 200-filter-start.conf ... 399-filter-end.conf 400-output-start.conf ... 599-output-end.conf
Between those sections, I added de ‘normal’ content:
001-dev-jdbc.conf 002-test-jdbc.conf 401-dev-elastic.conf 402-test-elastic.conf
Resulting in this ordered list:
000-intput-start.conf 001-dev-jdbc.conf 002-test-jdbc.conf 199-input-end.conf 200-filter-start.conf 399-filter-end.conf 400-output-start.conf 401-dev-elastic.conf 402-test-elastic.conf 599-output-end.conf
Of course, I needed to use ‘type’ in the inputs to tag the sources and IF’s in the outputs to select the right ones.
Unfortunately, when I started Logstash, assuming that all files will be concatenated in order, Logstash complained right at the first file that is was invalid. It needs to be a complete valid file. I didn’t get it to work with simply
input {
So then I made a custom *nix startup script to concatenate all the files in order into once single config file:
rm $LS_CONF_DIR/complete.conf
cat $(ls $LS_CONF_DIR/*.conf) > $LS_CONF_DIR/complete.conf
echo "Reading Logstash config from "$LS_CONF_DIR"/complete.conf"
exec $LS_EXEC -f ${LS_CONF_DIR}/complete.conf
And this works like a charm! Now Logstash is reading from several sources and writing to several targets, and I still am able to seperate different configs in small fragments.