tuto:monitoring:graylog_to_crowdsec

Différences

Ci-dessous, les différences entre deux révisions de la page.

Lien vers cette vue comparative

Les deux révisions précédentes Révision précédente
Prochaine révision 		Révision précédente
tuto:monitoring:graylog_to_crowdsec [05/03/2021 18:35]
dani 		tuto:monitoring:graylog_to_crowdsec [05/03/2021 19:19] (Version actuelle)
rv [Background]
Ligne 3: Ligne 3:
 ===== Background ===== ===== Background =====
  
-Crowdsec's architecture allows running several agents, each parsing the local logs on the server it's running, and sending events to a local API. While this approach works and is flexible, it might not be the most efficient. In my case, all my server are already sending their logs to a Graylog instance. Running one crowdsec agent on all of those VM would be a waist :+Crowdsec's architecture allows running several agents, each parsing the local logs on the server it's running, and sending events to a local API. While this approach works and is flexible, it might not be the most efficient. In my case, all my servers are already sending their logs to a Graylog instance. Running one crowdsec agent on all of those VM would be a waste :
   * I really like the idea behind the Journal (systemd-journald), it's very conveniant. But it has a major drawback : it's slow has hell ! Better have everything on SSD, or reading the journal will slows everything down. As I already have journalbeat collecting logs from the Journal, I prefer not adding another Journal reader, which will slows things down even further   * I really like the idea behind the Journal (systemd-journald), it's very conveniant. But it has a major drawback : it's slow has hell ! Better have everything on SSD, or reading the journal will slows everything down. As I already have journalbeat collecting logs from the Journal, I prefer not adding another Journal reader, which will slows things down even further
   * Let's assume we have 40 VM on which we'd like crowdsec agent running. This means something like 40x80MB = 3.2GB of RAM, just for crowdsec   * Let's assume we have 40 VM on which we'd like crowdsec agent running. This means something like 40x80MB = 3.2GB of RAM, just for crowdsec
Ligne 12: Ligne 12:
 ===== Send logs from Graylog to ? ===== ===== Send logs from Graylog to ? =====
  
-As I already have all my logs in Graylog, it'd be better to send this stream of logs to a single crowdsec installation. But, for now, crowdsec doesn't have network logs input, it can only reads files and the Journal (I've opened a [[https://github.com/crowdsecurity/crowdsec/issues/621|ticket]] for this). So, the idea is to somehow forward the logs I want fro Graylog to a small daemon, which would write logs for crowdsec to consume. +As I already have all my logs in Graylog, it'd be better to send this stream of logs to a single crowdsec installation. But, for now, crowdsec doesn't have network logs input, it can only reads files and the Journal (I've opened a [[https://github.com/crowdsecurity/crowdsec/issues/621|ticket]] for this). So, the idea is to somehow forward the logs I want from Graylog to a small daemon, which would write logs for crowdsec to consume. 
  
 +Here's the global flow
 +
 +{{ :tuto:monitoring:graylog_to_crowdsec.png |}}
 ===== g2cs ===== ===== g2cs =====
  
Ligne 36: Ligne 39:
 <note tip>You can choose a directory on a tmpfs filesystem to improve performance, as you do not need those logs to be persistent</note> <note tip>You can choose a directory on a tmpfs filesystem to improve performance, as you do not need those logs to be persistent</note>
  
 +<note>We could probably do the same thing with rsyslog, but it's configuration is arcane to me, so, it was easier to write the small g2cs daemon instead</note>
 ===== Configure crowdsec ===== ===== Configure crowdsec =====
 Now that we have our g2cs daemon running, you can configure crowdsec acquisition to read these files. Something like Now that we have our g2cs daemon running, you can configure crowdsec acquisition to read these files. Something like
Ligne 93: Ligne 97:
  
 You should now see logs flowing from Graylog, to crowdsec. I'm using this on a small graylog setup, ingesting about 400msg/sec, out of which ~200msg/sec are parsed by my single crowdsec install. I just have to install the bouncers where I want to react to all the agressive IP collected on all my servers You should now see logs flowing from Graylog, to crowdsec. I'm using this on a small graylog setup, ingesting about 400msg/sec, out of which ~200msg/sec are parsed by my single crowdsec install. I just have to install the bouncers where I want to react to all the agressive IP collected on all my servers
 +
 +===== Run crowdsec with less privileges =====
 +Bonus point : as crowdsec only access logs from the g2cs daemon, you can run both as a less privileged user, instead of root. First, create an unprivileged user
 +
 +<code bash>
 +useradd -r -s /sbin/nologin g2cs
 +</code>
 +
 +Now adapt the systemd unit for crowdsec, eg in /etc/systemd/system/crowdsec.service.d/user.conf
 +
 +<code ini>
 +Service]
 +User=g2cs
 +Group=g2cs
 +</code>
 +
 +And create a systemd unit for g2cs itself, /etc/systemd/system/g2cs.service
 +
 +<code ini>
 +[Unit]
 +Description=Graylog to Crowdsec syslog daemon
 +After=syslog.target
 +
 +[Service]
 +Type=simple
 +ExecStart=/usr/local/bin/g2cs --port=514 --logdir=/run/g2cs/logs
 +User=g2cs
 +Group=g2cs
 +Restart=always
 +PrivateTmp=yes
 +PrivateDevices=yes
 +ProtectSystem=full
 +ProtectHome=yes
 +NoNewPrivileges=yes
 +SyslogIdentifier=g2cs
 +
 +# Allow binding on privileged ports
 +CapabilityBoundingSet=CAP_NET_BIND_SERVICE
 +AmbientCapabilities=CAP_NET_BIND_SERVICE
 +
 +
 +[Install]
 +WantedBy=multi-user.target
 +</code>
  • tuto/monitoring/graylog_to_crowdsec.1614965728.txt.gz
  • Dernière modification: 05/03/2021 18:35
  • de dani