A few days ago, I setup our shiny new Graylog server.
I successfully added something like 10+ windows servers to collect the event log entries from.
As the second part of the task I wanted to add linux servers also.
I added the first one, it looks like the collector running and the messages are arriving, but no message shown on the web console. Weird.
Digging a bit deeper I found thousands of this message in the indexer fail:
{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [graylog_0] has been exceeded"}
This means we exceeded the maximum field number (I think due to the event types in the Windows).
I read through some forum posts about it. Tried to change the settings in the elasticsearch.yml file.
It didn't help. The result of my actions was a inoperable elasticsearch.
Finally I deleted the whole thing (elasticsearch) together with the indexes, and reinstalled it.
The result:
A working elasticsearch instance. The 1000 field limit kept, and in addition I got a new error. It said something like this: The graylog_deflector is an index and not an alias.
Googling around, I found the problem, but not the solution. Then I was start to think instead of googling. What I've learned:
- From one of the log files I learned, that the elasticsearch configuration isn't done through the config file but through the web API with JSON objects.
- curl is your friend
- The Graylog creates a graylog_deflector index when it can't find the graylog_deflector alias. What it unable to use. You can't do anything with it from the Graylog, so you screwed.
The solution based above:
- In the Graylog web UI go to the System/Indices>Indices. Select the Default index set
- In the Maintanance select the Rotate active write index. It will create a graylog_0 index (but it will not work)
- Go to the console and stop the graylog:
sudo service graylog-server stop - Handle the 1000 field problem:
curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
"index.mapping.total_fields.limit" : "5000"
}' - Stop the graylog_deflector index:
curl -XPOST 'localhost:9200/graylog_deflector/_close?pretty' - Delete the graylog_deflector index:
curl -XDELETE 'localhost:9200/graylog_deflector?pretty' - Add the graylog_deflector as alias to the newly created graylog_0 index:
curl -XPOST 'localhost:9200/_aliases?pretty' -H 'Content-Type: application/json' -d'
{
"actions" : [
{ "add" : { "index" : "graylog_0", "alias" : "graylog_deflector" } }
]
}' - Restart graylog:
sudo service graylog-server start - Now the graylog starts the correct reindexing process it can even take days to finish, but you can see your collected messages in the meantime.
Nincsenek megjegyzések:
Megjegyzés küldése