agroportal trafic analysis

[syphax-bouazzouni]

The 06/01/2022 we had this issue (see capture below) when accessing to http://agroportal.lirmm.fr/

The cause

This may because we are spammed by bot IPs (see capture below of the count of request send to Agroportal the 06/01/2022)

Logs

And there is what we can see an the access log

We can observe that they target this two routes :

• /feedback
• /login

[jonquet]

A noter l'existence d'un fichier robots.txt:
http://agroportal.lirmm.fr/robots.txt
http://bioportal.lirmm.fr/robots.txt

@syphax-bouazzouni Pourrait tu regarder si ces fichiers sont differents de ceux que j'avais customisé il y a qq mois. Me semble que j'avais mis plus de bot

[jonquet]

Ce jour de STI-RX:
Je viens de mettre en place mod_security sur le apache de agroportal. Toutes les règles sont désactivée sauf la surveillance de feedback avec des critères un peu stricts.

La conf du module est sous /etc/httpd/modsecurity.d et /etc/httpd/conf.d/mod_security.conf
et pour /feedback une entrée est rajoutée dans /etc/httpd/conf.d/10-appliance.ontoportal.org_non-tls.conf

[jonquet]

Le 'bon ' (celui edité en fevrier 2021) fihier robots.txt est ici:
https://github.com/ontoportal-lirmm/bioportal_web_ui/blob/development/config/robots/appliance.txt

[syphax-bouazzouni]

Summary of the latest updates

STI-RX changes :

They have added mod_security and enabled only for the /feedback
The configuration are in /etc/httpd/conf.d/10-appliance.ontoportal.org_non-tls.conf

Did it work ?

it seem that no, prouve the following screenshoot of the /var/log/httpd/appliance.ontoportal.org_non-tls_access.log file for the 24/Jan/2022 (got using the following command grep "24/Jan/2022" appliance.ontoportal.org_non-tls_access.log | less )

What next ?

In the above screenshot of the access we remark that they are from robots and more specifically from the AhrefsBots.

So the next step is to disallow robots for some paths, like feedback

Update robot.txt file

• disable robots agents for some paths (like /feedback) ontoportal-lirmm/bioportal_web_ui#57

Did it work ?

yes it worked. Below the screenshoot of the /var/log/httpd/appliance.ontoportal.org_non-tls_access.log file for the 26/Jan/2022 and we can see that there is no more /feedback or AhrefsBots.

But still we see that the Googlebot is still there (which is not a problem of him self) and index unwanted paths like /ajax.
So we need to :

• update robot.txt to disalow /ajax ontoportal-lirmm/bioportal_web_ui#58

[jonquet]

Similar to the /ajax exclusion we can do an exclusion for /javacsripts which is also very frequent in the logs

[syphax-bouazzouni]

27/01/2022 update

A new bot to disalow DataForSeoBot

And add the disalow /javacsripts and /widgets/

• update robot.txt to disalow /javascripts , /widgets and the agent DataForSeoBot ontoportal-lirmm/bioportal_web_ui#59

[syphax-bouazzouni]

Issue resolved, there is no more bots calling /feedback or /login in the access logs