Content packs not auto loading steams about graylog-docker

[tobias-g]

Yep looks like that was it 👍

Started elastic search and followed the logs unitl I saw

[2018-08-07T14:37:25,519][INFO ][o.e.c.m.MetaDataCreateIndexService] [XZjuckQ] [.triggered_watches] creating index, cause [auto(bulk api)], templates [triggered_watches], shards [1]/[1], mappings [triggered_watch]

And started graylog after that, streams and inputs all accounted for and present after that.

Also as far as exposing the data directory in ES using AWS ES anyway the above was just a local example.

Thanks for the help here :)

[mariussturm]

Hi,
in general that should work. Do you see any error in the server logs related to content pack loading?

[tobias-g]

See the following when I look for content pack related stuff in graylog logs.

2018-08-07 13:52:26,037 ERROR: org.graylog2.bundles.BundleImporter - Error while creating entities in content pack. Starting rollback.
java.lang.IllegalStateException: No default index set configured. This is a bug!
    at com.google.common.base.Preconditions.checkState(Preconditions.java:459) ~[graylog.jar:?]
    at org.graylog2.indexer.indexset.MongoIndexSetService.getDefault(MongoIndexSetService.java:105) ~[graylog.jar:?]
    at org.graylog2.indexer.MongoIndexSetRegistry.getDefault(MongoIndexSetRegistry.java:145) ~[graylog.jar:?]
    at org.graylog2.bundles.BundleImporter.createStream(BundleImporter.java:573) ~[graylog.jar:?]
    at org.graylog2.bundles.BundleImporter.createStreams(BundleImporter.java:557) ~[graylog.jar:?]
    at org.graylog2.bundles.BundleImporter.runImport(BundleImporter.java:189) [graylog.jar:?]
    at org.graylog2.bundles.BundleService.applyConfigurationBundle(BundleService.java:112) [graylog.jar:?]
    at org.graylog2.periodical.ContentPackLoaderPeriodical.doRun(ContentPackLoaderPeriodical.java:196) [graylog.jar:?]
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2018-08-07 13:52:26,085 INFO : org.graylog2.inputs.InputStateListener - Input [Syslog TCP/5b69a4194cedfd0001f3c24d] is now STARTING
2018-08-07 13:52:26,067 ERROR: org.graylog2.periodical.ContentPackLoaderPeriodical - Uncaught exception in periodical
java.lang.RuntimeException: java.lang.IllegalStateException: No default index set configured. This is a bug!
    at org.graylog2.bundles.BundleImporter.runImport(BundleImporter.java:200) ~[graylog.jar:?]
    at org.graylog2.bundles.BundleService.applyConfigurationBundle(BundleService.java:112) ~[graylog.jar:?]
    at org.graylog2.periodical.ContentPackLoaderPeriodical.doRun(ContentPackLoaderPeriodical.java:196) ~[graylog.jar:?]
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.IllegalStateException: No default index set configured. This is a bug!
    at com.google.common.base.Preconditions.checkState(Preconditions.java:459) ~[graylog.jar:?]
    at org.graylog2.indexer.indexset.MongoIndexSetService.getDefault(MongoIndexSetService.java:105) ~[graylog.jar:?]
    at org.graylog2.indexer.MongoIndexSetRegistry.getDefault(MongoIndexSetRegistry.java:145) ~[graylog.jar:?]
    at org.graylog2.bundles.BundleImporter.createStream(BundleImporter.java:573) ~[graylog.jar:?]
    at org.graylog2.bundles.BundleImporter.createStreams(BundleImporter.java:557) ~[graylog.jar:?]
    at org.graylog2.bundles.BundleImporter.runImport(BundleImporter.java:189) ~[graylog.jar:?]
    ... 4 more

Not sure what to make of it to be honest, hopefully makes more sense to someone else.

[mariussturm]

The default index set in Elasticsearch that is created during first start-up is not there. Maybe your ES cluster is in an error state or not accessible at all?

[tobias-g]

Running the following locally gets me this scenario:

docker run --name mongo -d mongo:3

docker run --name elasticsearch \
    -e "http.host=0.0.0.0" \
    -e "xpack.security.enabled=false" \
    -d docker.elastic.co/elasticsearch/elasticsearch:5.6.2

docker create --link mongo --link elasticsearch \
    -p 9000:9000 -p 12201:12201 -p 514:514 -p 5555:5555 \
    --name graylog \
    -e GRAYLOG_CONTENT_PACKS_LOADER_ENABLED=true \
    -e GRAYLOG_CONTENT_PACKS_AUTO_LOAD=input.json,streams.json \
    -e GRAYLOG_CONTENT_PACKS_DIR=data/contentpacks \
    -e GRAYLOG_WEB_ENDPOINT_URI="http://127.0.0.1:9000/api" \
    graylog/graylog:2.4

docker cp /tmp/contentpacks/. graylog:/usr/share/graylog/data/contentpacks/

docker start graylog

Elastic search doesn't seem to be in an error state as far as I can tell.

Not used elastic search too much so is this a case of needing to wait a bit for elastic search to start and settle itself or do I need to set something up in elastic search before I start graylog to get the default index set?

GrayLog does eventually start up and work as expected so it's either fixing this issue itself of waiting until that index is there?

[mariussturm]

That's possible, there shout be a log message about it later in the file. Usually you expose the data directory of the ES container anyways to keep state of the system. If you let it create once the import of the content pack should work then.