2017. augusztus 14., hétfő

Graylog, Elasticsearch, Maximum number of fields, graylog_deflector

Just because I'm not a Linux guy
A few days ago, I setup our shiny new Graylog server.
I successfully added something like 10+ windows servers to collect the event log entries from.
As the second part of the task I wanted to add linux servers also.
I added the first one, it looks like the collector running and the messages are arriving, but no message shown on the web console. Weird.
Digging a bit deeper I found thousands of this message in the indexer fail:
{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [graylog_0] has been exceeded"}
This means we exceeded the maximum field number (I think due to the event types in the Windows).
I read through some forum posts about it. Tried to change the settings in the elasticsearch.yml file.
It didn't help. The result of my actions was a inoperable elasticsearch.
Finally I deleted the whole thing (elasticsearch) together with the indexes, and reinstalled it.
The result:
A working elasticsearch instance. The 1000 field limit kept, and in addition I got a new error. It said something like this: The graylog_deflector is an index and not an alias.
Googling around, I found the problem, but not the solution. Then I was start to think instead of googling. What I've learned:

  1.  From one of the log files I learned, that the elasticsearch configuration isn't done through the config file but through the web API with JSON objects.
  2. curl is your friend
  3. The Graylog creates a graylog_deflector index when it can't find the graylog_deflector alias. What it unable to use. You can't do anything with it from the Graylog, so you screwed.

The solution based above:

  1. In the Graylog web UI go to the System/Indices>Indices. Select the Default index set
  2. In the Maintanance select the Rotate active write index. It will create a graylog_0 index (but it will not work)
  3. Go to the console and stop the graylog:
    sudo service graylog-server stop
  4. Handle the 1000 field problem:
    curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
      "index.mapping.total_fields.limit" : "5000"
  5. Stop the graylog_deflector index:
    curl -XPOST 'localhost:9200/graylog_deflector/_close?pretty'
  6. Delete the graylog_deflector index:
    curl -XDELETE 'localhost:9200/graylog_deflector?pretty'
  7. Add the graylog_deflector as alias to the newly created graylog_0 index:
    curl -XPOST 'localhost:9200/_aliases?pretty' -H 'Content-Type: application/json' -d'
        "actions" : [
            { "add" : { "index" : "graylog_0", "alias" : "graylog_deflector" } }
  8. Restart graylog:
    sudo service graylog-server start
  9. Now the graylog starts the correct reindexing process it can even take days to finish, but you can see your collected messages in the meantime.

Nincsenek megjegyzések:

Megjegyzés küldése