add purge_history.sh and purge_remote_media.sh scripts

[rubo77]

add purge_history.sh and purge_remote_media.sh scripts to contrib/scripts/ and link them in purge_history_api.rst and purge_remote_media_api.rst

Signed-off-by: Ruben Barkow [email protected]

replaces #1034

[richvdh]

please can you (a) remove the changes to the docs (b) put the scripts in their own directory with a readme

[4nd3r]

can you please make the scripts pass shellcheck?

$ shellcheck purge_history.sh 

In purge_history.sh line 91:
POSTDATA='{"delete_local_events":"true"}' # this will really delete local events, so the messages in the room really disappear unless they are restored by remote federation
         ^-- SC2089: Quotes/backslashes will be treated literally. Use an array.


In purge_history.sh line 94:
    echo "########################################### "$(date)" ################# "
                                                       ^-- SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                       ^-- SC2046: Quote this to prevent word splitting.


In purge_history.sh line 108:
      OUT=$(curl -s -d $POSTDATA -X POST $DOMAIN':8008/_matrix/client/r0/admin/purge_history/'$ROOM'/'$EVENT_ID'/?access_token='$TOKEN)
                       ^-- SC2090: Quotes/backslashes in this variable will not be respected.
                                                                                              ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                      ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                                ^-- SC2086: Double quote to prevent globbing and word splitting.


In purge_history.sh line 109:
      PURGE_ID=$(echo $OUT |grep purge_id|cut -d'"' -f4 )
                      ^-- SC2086: Double quote to prevent globbing and word splitting.


In purge_history.sh line 116:
          STATUS=$(curl -s GET $DOMAIN':8008/_matrix/client/r0/admin/purge_history_status/'$PURGE_ID'?access_token='$TOKEN |grep status|cut -d'"' -f4)
                                                                                           ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                    ^-- SC2086: Double quote to prevent globbing and word splitting.


In purge_history.sh line 119:
          SLEEP=$(($SLEEP + 1))
                   ^-- SC2004: $/${} is unnecessary on arithmetic variables.

$ shellcheck purge_remote_media.sh 

In purge_remote_media.sh line 50:
curl -v POST '$DOMAIN:8008/_matrix/client/r0/admin/purge_media_cache/?before_ts='$UNIX_TIMESTAMP'&access_token='$TOKEN
             ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.
                                                                                 ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                ^-- SC2086: Double quote to prevent globbing and word splitting.

[rubo77]

SC2089: Quotes/backslashes will be treated literally. Use an array.

using arrays everywhere would make the code reeaaally cluttered

[rubo77]

where do I find a documentation about how to send an Authorization header? So I can change that in my scripts?

[richvdh]

where do I find a documentation about how to send an Authorization header?

https://matrix.org/docs/spec/client_server/r0.4.0.html#using-access-tokens

fixed

[rubo77]

I refactored the authentification and added a database function

There are only two shellcheck warnings left:

$ shellcheck purge_history.sh 

In purge_history.sh line 87:
POSTDATA='{"delete_local_events":"true"}' # this will really delete local events, so the messages in the room really disappear unless they are restored by remote federation
         ^-- SC2089: Quotes/backslashes will be treated literally. Use an array.


In purge_history.sh line 107:
      OUT=$(curl --header "$AUTH" -s -d $POSTDATA POST "$API_URL/admin/purge_history/$ROOM/$EVENT_ID")
                                        ^-- SC2090: Quotes/backslashes in this variable will not be respected.

Those are irrelevant to security, because there are no external variables used.

[richvdh]

thanks!

[saesaeru]

Please make it work for sqlite3 for all rooms.

[rubo77]

For sqlite3 just uncomment line 66 and comment out line 68

[saesaeru]

How to make this script automatic for All ROOMS, because inserting room-ID to the script by hand is not comfortable.
It's better to creen all messages in all room by Cron, but it's not possible with this script too.

[rubo77]

you can select all the rooms that are not encrypted and loop over the result:

+# SELECT room_id FROM rooms WHERE room_id NOT IN (SELECT DISTINCT room_id FROM events WHERE type ='m.room.encrypted')

+# or

+# select all rooms with at least 100 members:

+# SELECT q.room_id FROM (select count(*) as numberofusers, room_id FROM current_state_events WHERE type ='m.room.member' GROUP BY room_id) AS q LEFT JOIN room_aliases a ON q.room_id=a.room_id WHERE q.numberofusers > 100 ORDER BY numberofusers desc

[saesaeru]

The script working and erasing messages from DB, but the messages not erasing from clients. Why?
How to erase messages from client???

[saesaeru]

you can select all the rooms that are not encrypted and loop over the result:

+# SELECT room_id FROM rooms WHERE room_id NOT IN (SELECT DISTINCT room_id FROM events WHERE type ='m.room.encrypted')

+# or

+# select all rooms with at least 100 members:

+# SELECT q.room_id FROM (select count(*) as numberofusers, room_id FROM current_state_events WHERE type ='m.room.member' GROUP BY room_id) AS q LEFT JOIN room_aliases a ON q.room_id=a.room_id WHERE q.numberofusers > 100 ORDER BY numberofusers desc

I so this in a scrip file, but how to get result of this SELECT to: ROOMS_ARRAY=( ????

[rubo77]

The script working and erasing messages from DB, but the messages not erasing from clients. Why?
How to erase messages from client???

that would be #1730
(still on my todo list)

maybe you can finish that one?

[saesaeru]

I'm glad to.. but don't know how :)
Please do it. if you can. without this function synapse/matrix is useless.
(no button "clear history" in chat room in riot.im :( too )