2017. augusztus 14., hétfő

Graylog, Elasticsearch, Maximum number of fields, graylog_deflector

Just because I'm not a Linux guy
A few days ago, I setup our shiny new Graylog server.
I successfully added something like 10+ windows servers to collect the event log entries from.
As the second part of the task I wanted to add linux servers also.
I added the first one, it looks like the collector running and the messages are arriving, but no message shown on the web console. Weird.
Digging a bit deeper I found thousands of this message in the indexer fail:
{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [graylog_0] has been exceeded"}
This means we exceeded the maximum field number (I think due to the event types in the Windows).
I read through some forum posts about it. Tried to change the settings in the elasticsearch.yml file.
It didn't help. The result of my actions was a inoperable elasticsearch.
Finally I deleted the whole thing (elasticsearch) together with the indexes, and reinstalled it.
The result:
A working elasticsearch instance. The 1000 field limit kept, and in addition I got a new error. It said something like this: The graylog_deflector is an index and not an alias.
Googling around, I found the problem, but not the solution. Then I was start to think instead of googling. What I've learned:

  1.  From one of the log files I learned, that the elasticsearch configuration isn't done through the config file but through the web API with JSON objects.
  2. curl is your friend
  3. The Graylog creates a graylog_deflector index when it can't find the graylog_deflector alias. What it unable to use. You can't do anything with it from the Graylog, so you screwed.


The solution based above:

  1. In the Graylog web UI go to the System/Indices>Indices. Select the Default index set
  2. In the Maintanance select the Rotate active write index. It will create a graylog_0 index (but it will not work)
  3. Go to the console and stop the graylog:
    sudo service graylog-server stop
  4. Handle the 1000 field problem:
    curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
      "index.mapping.total_fields.limit" : "5000"
    }'
  5. Stop the graylog_deflector index:
    curl -XPOST 'localhost:9200/graylog_deflector/_close?pretty'
  6. Delete the graylog_deflector index:
    curl -XDELETE 'localhost:9200/graylog_deflector?pretty'
  7. Add the graylog_deflector as alias to the newly created graylog_0 index:
    curl -XPOST 'localhost:9200/_aliases?pretty' -H 'Content-Type: application/json' -d'
    {
        "actions" : [
            { "add" : { "index" : "graylog_0", "alias" : "graylog_deflector" } }
        ]
    }'
  8. Restart graylog:
    sudo service graylog-server start
  9. Now the graylog starts the correct reindexing process it can even take days to finish, but you can see your collected messages in the meantime.



2017. július 31., hétfő

Pen Plotter 3. - Working

It is finally working.
Built the whole thing, assembled, then redesigned most of the pen holder.
The problem was the following:
  • The SG90 microservo I intended to use was not strong enough
  • The line I wanted to use to pull up the Z carriage was a bad design decision
  • The arduino was not able to provide enough current to the stalled microservo, so it rebooted when the pen pulled up.

In the new design:

  • The SG90 replaced with a more powerful MG995
  • The servo moving the carriage directly
  • Added a 6V regulator for the servo, what gets its power from the beefy 12V PSU and not the arduino.


The electronics on the head still messy as I don't designed a proper PCB yet (but it will only happen when I finish the milling and the laser engraving head design):



And finally here is the first video of the working unit:


2017. július 27., csütörtök

Pen Plotter 2. - Measuring Proxxon

I've a Proxxon Micromot drill. I'm planning to attach it to the plotter as replacement to the pen, to be able to do some light cuts.
For this I don't want to use it's original power supply unit. As the speed control is located in the power supply I wanted to know it's properties:

  • How the motor driven? Pure PWM or filtered to DC?
  • What is the PWM frequency?

To measure this without disassembling, not looked like an easy task. The reason is that Proxxon uses a proprietary connector, what is not available on the market. So I measured the diameter of the pins and looks like with two 2.3mm drill bit I made it:


The measurements:
Without load:


At minimum speed:


At maximum speed:


It is a bit noisy to my taste and figured out that it runs at 100Hz and not really filtered.
Based on this I can design the new electronics into the plotter.

2017. július 5., szerda

Pen Plotter 1.

I'm not documented this project deeply, step-by-step. Now I'm just writing where am I today with the build. Mainly pictures.
More than three month ago I had a conversation with my son. He told me that he want to build a "drawing machine" essentially a pen plotter together with me. I started to design one. After a few conversation he told me that he wanted to build it from Lego.
I was already too deep in the designing already, so I didn't want to stop. In addition I was planning to add exchangeable head to be able to use it as a light router and a laser cutter/engraver.
I think the design mainly finished:



I left out a few things from the design, like drag chain, screws, spacers, timing belts.
During the design process I already printed some of the parts:


As you can see, there are some assembled parts also on the picture, like the Y axis drive (round white thing in the center) and Z axis carriage (the green thing).
How the Y axis timing belt mount into the holder:


Lots of holes, the base plate:


X axis assembled:


continue...

2017. május 27., szombat

StepTest 1.

During the Cyclone mechanical build I hate when I have to wind various axes by hand from one end to the other. I was thinking. What if I build a simple circuit for the stepper motors to do this for me. No MCUs, just a stepper driver, a variable oscillator with a speed control potentiometer, a direction control switch, and the necessary power supply.
I wanted to keep it as simple as possible, so I just picked up two schematics from the net as base.
The DRV8825 stepper module minimal config from pololu:



And a 50% duty 555 config:


I used 680nF in the place of the C1 and a 1K resistor in series with a 10K potentiometer in the place of R2. Here are the result.
Slowest:


Fastest:



This allows me to change the frequency between 74 and 704Hz.

In addition to the two circuits above I added a simple 78l05 regulator, so the whole circuit runs from a single 12V supply.

Here is the build on breadboard:


During the test I realized, that the motor is quite noisy. So I think to use the microstepping and in parallel increase the frequency by reducing the value of the C1.
Next: proper build.

2017. május 26., péntek

Cyclone PCB Factory 5.

Oh, noooo.
I just tried to attach my Dremel to the Cyclone:


This doesn't work.
Just found out that I've a Dremel 300 and the Z axis is designed for the Dremel 3000.
There are no ready design available for the Dremel 300. Shit.
I've a few options:
Design a Z axis for my Proxxon.
Buy something from eBay/Aliexpress (775 DC motor, CNC spindle), they have ready Z axis design.
Buy a Dremel 3000.
After a half day thinking, evaluating the options, decided to buy a Dremel 3000. It needs the least effort now. I was able to find one for ~$40.
I ordered it in the morning. It landed in the FoxPost's box on the same evening (It was fast):


2017. május 25., csütörtök

Cyclone PCB Factory 4.

Started to correct the things from the previous post. Went to two local shops (had no more time), to find M3 grub screws for the steppers. No success.
What if I try to make a few at home.
Here is the setup:


And the result:


I think, not bad for the first try.