Memory full

[Rhoenit]

I´m trying to use tpot in a VM but after 1-2 weeks the memory is full.
Does anyone have an idea whats wrong or how to clean up?

df-h

logstash

[t3chn0m4g3]

In /opt/tpot/bin you may find the clean.sh which can delete all logs from the honeypots.

In /etc/crontab look for the following lines and adjust the configs to your needs.

# Delete elasticsearch logstash indices older than 90 days
$myRANDOM_MINUTE $myDEL_HOUR * * *      root    curator --config /opt/tpot/etc/curator/curator.yml /opt/tpot/etc/curator/actions.yml

[Rhoenit]

does this help?

[t3chn0m4g3]

Does not seem to be too much of data in the logs.

[Rhoenit]

this tpot was fresh installed 3 days ago and already uses 89GB. The biggest logstash is 22mb.
Can you explain this and how to prevent?

[t3chn0m4g3]

Given the fact that as per your feedback the docker images, honeypot logs, elk data and system logs are not the culprit it is highly atypical and should look like this (install is running for about two week with very high volume of events):

[Rhoenit]

Now it is running for a week and I guess it is running out of space again this week. I already used clean.sh.
There aren´t a lot of events. Only 4 CiscoASA.
What is using so much space?

[t3chn0m4g3]

I cannot reproduce this. However you can try this and iteratively follow the folders using the most disk space. Once the folders and files are identified we have a clue to continue the investigation.

[Rhoenit]

I found this:

[t3chn0m4g3]

For some reason your daemon.log is unusually large. Please tail it for contents. The rest looks pretty much fine, even though syslog is quite large.

[Rhoenit]

[t3chn0m4g3]

Is it possible that for some reason the machine is constantly restarting the tpot.service (dps.sh -2). Based on the log messages it seems that some services are constantly restarting. At least this would explain the high volume of logs.

This installation is running since December of 2020, without any manual cleaning interaction and this is how it should look like:

[Rhoenit]

it only looks like ewsposter restarting all the time.

[t3chn0m4g3]

That is a known issue and @trixam is working on a fix.

[t3chn0m4g3]

Meanwhile you can adjust /etc/logrotate.d/rsyslog to your needs so that deamon.log will either be rotated earlier or deleted.