Grok parse fail via kibano logs of dockermonitoringloggingalerting_elastalert

[riemers]

Pretty much all is working, works like a charm. Ony in kibana is see the most entry's coming from dockermonitoringloggingalerting_elastalert. It doesn't say much only it does include the tag "_grokparsefailure" it creates that names because my folder is that btw. Since i see that kibana and elastalert are being build with docker compose (would prefer to not have that before it though, but no problem)

Is there any way to debug this? To see what the grok is going on :) i did ran that seperate config check of logstash but that says all fine too.

Also any best practices about adding more "hosts" to this setup? Thanks, and good work on all this.

[uschtwill]

Hey @riemers, the logstash config validation just checks, if the config is legitimate, not if it makes sense. ;)

To debug this, the question really is: what is in the message field of those elastalert logs that are coming in?

`_grokparsefailure is the default tag, that logs get tagged with, if none of the other filters matched. Check the documentation on it here.

Regardless of everything else, if you just want to not get that many logs from elastalert, you can define a filter in the logstash that drops those logs instead of sending them to Elasticsearch (or adjust the loglevel of elastalert itself of course). Just check what the defining characteristics of those unnecessary logs are.

About "more hosts": Not really. But it should be pretty straight forward, throw a filebeat, node-exporter and cadvisor on every host you want to monitor. Configure the Prometheus config, maybe tweak a filter here and there, done. The hard part will be to figure out the networking and security (shutting down ports), but that really isn't specific to this suite. Probably best to use some framework for that (e.g. Docker Swarm, Kubernetes and such). Actually I am currently dabbling with Kubernetes and like it a lot, so I might port the whole thing at some point.

[riemers]

Ah i think i understand the grokparse issues then, if no match then it becomes that. Basicly i need to add more filters.

Made the dashboard more container group aware, it was missing the items on the graps and i use xfs rather then ext on the free and used (shows a bit weird if you have multiple filesystems)

Working on a docker compose for an extra host. If i have an example one could use that to add it to the setup.

[uschtwill]

Sure thing, I am more than happy to include such an extra docker-compose.yaml! Would you mind adding it via PR, when you feel it's ready?

[riemers]

Sure, its just the stripped version. You still need to add the ip, testing that setup right now. Will update you when its ok.

[riemers]

I have this so far, but i don't see any logs coming in, i assume i have to set the listening on the main config also to something public but since its in a network i don't know if it can reach eachother. I openened the ports for logstash too but have not had a lot of luck as of yet. Also using the latest cadvisor since the last one had a bug. I think it is mainstream now anyways.

`version: '2'

services:

#########################################################

LOGGING

#########################################################

Runs on your node(s) and forwards all logs to Logstash.

filebeat:
container_name: filebeat
image: prima/filebeat
volumes:
- ./filebeat/filebeat.yml:/filebeat.yml
restart: always
labels:
container_group: monitoring
logging:
driver: gelf
options:
gelf-address: udp://realip:12201
labels: container_group

#########################################################

MONITORING

#########################################################

Runs on your node(s) and forwards node(host) metrics to Prometheus.

nodeexporter:
container_name: nodeexporter
image: prom/node-exporter
expose:
- 9100
restart: always
labels:
container_group: monitoring
logging:
driver: gelf
options:
gelf-address: udp://realip:12201
labels: container_group

#########################################################

Runs on your node(s) and forwards container metrics to Prometheus.

cadvisor:
container_name: cadvisor
image: google/cadvisor:v0.24.1
ports:
- 8080:8080
expose:
- 8080
volumes:
- "/:/rootfs:ro"
- "/var/run:/var/run:rw"
- "/sys:/sys:ro"
- "/var/lib/docker/:/var/lib/docker:ro"
restart: always
labels:
container_group: monitoring
logging:
driver: gelf
options:
gelf-address: udp://realip:12201
labels: container_group

#########################################################`

[uschtwill]

@riemers: Yeah, you would have to foward the ports logstash uses to the host like so:

    ports: 
      - "5044:5044"
      - "12201:120201"

leading to a logstash manifest like this:

# Aggregates logs and forwards them to Elasticsearch.
  logstash:
    container_name: logstash
    image: logstash
    expose:
      - 5044
      - 12201/udp
    ports: 
      - "5044:5044"
      - "12201:120201"
    volumes:
      - ./logstash/config:/config
      - ./logstash/patterns:/opt/logstash/extra_patterns
    restart: always
    labels:
      container_group: monitoring
    networks:
      default:
        ipv4_address: 192.16.0.38
    logging:
      driver: gelf
      options:
        gelf-address: udp://192.16.0.38:12201
        labels: container_group
    command: logstash -f /config --auto-reload

Of course then you have to figure out your networking and routing, which may or may not be straight forward, depending on the complexity of your setup. But having done that the other hosts should be able to ship node and container logs to logstash (using the host IP, not the logstash IP to define where to ship the logs).

Vice versa, so Prometheus can scrape the metrics from cAdvisor and nodeexporter, the other hosts would have to have those ports being forwarded as well. In the manifests you posted above, you're doing so for cAdvisor with port 8080, but nodeexporter is missing it for port 9100.

[riemers]

I'll work it out a bit further and once i have something i'll post it back or make a pull request. A bit slow these days, just became father again. Bit crazy now :)

[uschtwill]

Alright, I'll just leave this open until then.

And hey, congratulations! All the best... and many calm nights! :)

[riemers]

Just a note, it seems like some images like logstash/elasticsearch went to 5.x it seems. the auto reload function didn't work anymore and the kibana dashboard needed a newer version to work with elasticsearch. Maybe it was just me but i had to change some versions here and there to get it back up and running. Managed to get a host monitored so far with my configuration, adjusted the dashboard for it now trying to get logs from it too.

P.s. above example should have been " - 12201:12201/udp" then logshipping works for me too. The /udp part was crucial ;)

p.p.s. elastalert support for elasticsearch 5.x is not there yet, see Yelp/elastalert#790

[uschtwill]

Thanks for pointing this out, @riemers! I just added tags to fix the images for the ELK stack containers. The idea is to keep things working until ELK 5.0 has been around for a while and the different images have been updated accordingly, then we can upgrade the whole stack at once and iron out little things like dashboards, configuration etc.

Let me open a new issue for this, so we don't forget: #6

And regarding Logstash? Do you mean you got it to work with multiple hosts?

[riemers]

Another one for the docs, if you use elastalert and you have your own CA for certificates it is not as simple as replacing ./alertmanager/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt in the docker file. Instead since it runs python, you have to replace the python request pem. Which for me ended up with ./elastalert/certs/ca-certificates.crt:/usr/lib/python2.7/site-packages/requests-2.12.1-py2.7.egg/requests/cacert.pem something to take note if someone has their own CA too.

Still working on the rest, i managed to get the other hosts to attach to the docker setup. Just thinkering on how to make it more "secure" (looked at the FYI issue reported by another guy with ssl support and such but is only for loghost)

p.s. as a sidenote in the docs. If you docker-compose stop 'container' you can start it up with "docker-compose up 'container'" and the output will be on your screen instead of the gelf adapter, which makes it easier to debug issues.

[uschtwill]

Hey @riemers,

that ca-certificate-for-alertmanager-issue seems very specific. How about we don't put it in the README for that reason? I think people who really do have the same issue will be able to find it here as well, or we can refer them here if they don't find it by themselves.

Goof job on making them connect! Did you make any progress on securing them yet? What do you think about #10?

And for your p.s.: You're right, maybe this is a better default behaviour, @pierreozoux seems to have indicated the same. I am going to make some changes to make logging to the screen the default and provide info on how to turn it off.

[riemers]

Fine with me. Just trying to make sure people don't have to do the same hoop. Op 13 december 2016 om 20:01:53, Wilhelm Uschtrin ([email protected]) schreef:

…

Hey @riemers <https://github.com/riemers>, that ca-certificate-for-alertmanager-issue seems very specific. How about we don't put it in the README for that reason? I think people who really do have the same issue will be able to find it here as well, or we can refer them here if they don't find it by themselves. Goof job on making them connect! Did you make any progress on securing them yet? What do you think about #10 <#10> ? And for your p.s.: You're right, maybe this is a better default behaviour, @pierreozoux <https://github.com/pierreozoux> seems to have indicated the same. I am going to make some changes to make logging to the screen the default and provide info on how to turn it off. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApoOWciMIDsAj7goxc80WnwPIfWaGERks5rHuuhgaJpZM4KW8S2> .

[uschtwill]

Alright then.

Regarding the logging I just got tripped up badly:

With docker-compose 1.8.x it was possible to just log to both logstash and stdout by omitting -d from docker-compose up. But with 1.9.x this doesn't work anymore, every container just prints a single line of WARNING: no logs are available with the 'gelf' log driver with docker-compose not returning but staying in the foreground instead.

The idea was to use that feature of 1.8.x (that people apparently considered a bug) to by default log to stdout as well, just like you described. Sadly, this won't work anymore.

My next-best idea is to just include a hint in the README on how to selectively "toogle" between logging to logstash and stdout for individual containers.

[riemers]

That is bad indeed, i used that up without -d too. Still works on my side too. Perhaps you could add extra entry's in the docker compose and call them "alertmanager-logs" instead of "alertmanager" so you can still do "docker-compose up alertmanager-logs" don't know if that is something that can be done, just thinking out loud here.

[uschtwill]

Hmm, I am not sure I understand what you mean. :)

[riemers]

well docker compose consist of multiple containers, perhaps you can make a version with default logging to stdout and another for gelf. That way you can just swap them in. You could even just stop/start individually. Just don't know if 2 docker compose files in the same dir can work or perhaps just rename them when you want to use them (if that makes sense)

[uschtwill]

Ahhh, now I get it. Yeah, that could work I suppose, something like docker-compose -f docker-compose.kibana.yml and docker-compose -f docker-compose.stdout.yml. On the other hand I feel like this convolutes things a bit, even if wrapped in a setup.sh, so you could use it with sh setup.sh kibana and sh setup.sh stdout ...

So far I've added a paragraph to the readme to clarify things.

[uschtwill]

Closing this, since the original issue has long been addressed and a new issue has been opened.