Retention Feature Questions / Issues

[bverkron]

Attempting to use the retention feature but having some issues and want to confirm how it's supposed to work.

My goal is to have it purge files that are older than 15 days (for example) from one of the ftp directories for a user. So far I've setup the following but it's not purging any files when I run it and I see no errors in the container logs. Perhaps I'm misunderstanding how it's designed to work?

Note that I originally had just /ftp-storage/cam-ftp/ for the path but thought maybe it was a recursion issue do went all the way down to the lowest level folder but no difference. I also made sure the user has delete privileges and can delete files when connect to the ftp.

I've manually triggered the action several times and I get the confirmation in the UI it's been run and can see in the docker container log it was triggered but nothing else happens. The files don't get deleted and there are no further messages I can find.

sftpgo         | {"level":"info","time":"2024-08-31T18:15:30.312","sender":"DataRetention","connection_id":"data_retention_reolink_cam_ftp","message":"retention check completed"}

:/media/ext_storage/ftp/cam-ftp/door_bell/2024/08/05$ ls -alh
total 266M
drwxr-xr-x  2 mcp mcp 4.0K Aug  5 23:09 .
drwxr-xr-x 29 mcp mcp 4.0K Aug 31 04:21 ..
-rw-r--r--  1 mcp mcp   77 Aug  5 18:28 Doorbell_00_20240805182327.txt
-rw-r--r--  1 mcp mcp    0 Aug  5 18:29 Doorbell_00_20240805182411.txt
-rw-r--r--  1 mcp mcp   77 Aug  5 18:29 Doorbell_00_20240805182412.txt
-rw-r--r--  1 mcp mcp    0 Aug  5 18:30 Doorbell_00_20240805182456.mp4
-rw-r--r--  1 mcp mcp  14M Aug  5 18:30 Doorbell_00_20240805182457.mp4
-rw-r--r--  1 mcp mcp  49K Aug  5 18:30 Doorbell_00_20240805182458.jpg
-rw-r--r--  1 mcp mcp    0 Aug  5 18:30 Doorbell_00_20240805182549.mp4
-rw-r--r--  1 mcp mcp  49K Aug  5 18:30 Doorbell_00_20240805182550.jpg
-rw-r--r--  1 mcp mcp  23M Aug  5 18:31 Doorbell_00_20240805182550.mp4
-rw-r--r--  1 mcp mcp    0 Aug  5 18:31 Doorbell_00_20240805182556.jpg
-rw-r--r--  1 mcp mcp  49K Aug  5 18:31 Doorbell_00_20240805182558.jpg
-rw-r--r--  1 mcp mcp  50K Aug  5 18:32 Doorbell_00_20240805182709.jpg
-rw-r--r--  1 mcp mcp  14M Aug  5 18:32 Doorbell_00_20240805182709.mp4
-rw-r--r--  1 mcp mcp    0 Aug  5 18:32 Doorbell_00_20240805182735.jpg
-rw-r--r--  1 mcp mcp    0 Aug  5 18:32 Doorbell_00_20240805182735.mp4
-rw-r--r--  1 mcp mcp  50K Aug  5 18:32 Doorbell_00_20240805182736.jpg
-rw-r--r--  1 mcp mcp  29M Aug  5 18:33 Doorbell_00_20240805182736.mp4
-rw-r--r--  1 mcp mcp  51K Aug  5 18:33 Doorbell_00_20240805182809.jpg
-rw-r--r--  1 mcp mcp    0 Aug  5 19:06 Doorbell_00_20240805190127.mp4
-rw-r--r--  1 mcp mcp  49K Aug  5 19:06 Doorbell_00_20240805190129.jpg
-rw-r--r--  1 mcp mcp  19M Aug  5 19:07 Doorbell_00_20240805190129.mp4
-rw-r--r--  1 mcp mcp  46K Aug  5 19:20 Doorbell_00_20240805191459.jpg
-rw-r--r--  1 mcp mcp  15M Aug  5 19:20 Doorbell_00_20240805191459.mp4
-rw-r--r--  1 mcp mcp    0 Aug  5 19:22 Doorbell_00_20240805191725.jpg
-rw-r--r--  1 mcp mcp  49K Aug  5 19:22 Doorbell_00_20240805191727.jpg
-rw-r--r--  1 mcp mcp  12M Aug  5 19:22 Doorbell_00_20240805191727.mp4
-rw-r--r--  1 mcp mcp    0 Aug  5 19:30 Doorbell_00_20240805192527.mp4
-rw-r--r--  1 mcp mcp  49K Aug  5 19:30 Doorbell_00_20240805192528.jpg
-rw-r--r--  1 mcp mcp  18M Aug  5 19:31 Doorbell_00_20240805192528.mp4
-rw-r--r--  1 mcp mcp  14M Aug  5 19:31 Doorbell_00_20240805192606.mp4
-rw-r--r--  1 mcp mcp    0 Aug  5 19:31 Doorbell_00_20240805192647.mp4
-rw-r--r--  1 mcp mcp  49K Aug  5 19:31 Doorbell_00_20240805192648.jpg
-rw-r--r--  1 mcp mcp  11M Aug  5 19:32 Doorbell_00_20240805192648.mp4
-rw-r--r--  1 mcp mcp  48K Aug  5 19:35 Doorbell_00_20240805193009.jpg
-rw-r--r--  1 mcp mcp  15M Aug  5 19:35 Doorbell_00_20240805193009.mp4
-rw-r--r--  1 mcp mcp    0 Aug  5 20:01 Doorbell_00_20240805195650.mp4
-rw-r--r--  1 mcp mcp  15M Aug  5 20:02 Doorbell_00_20240805195651.mp4
-rw-r--r--  1 mcp mcp  49K Aug  5 20:02 Doorbell_00_20240805195652.jpg
-rw-r--r--  1 mcp mcp  12M Aug  5 20:25 Doorbell_00_20240805201936.mp4
-rw-r--r--  1 mcp mcp  48K Aug  5 20:24 Doorbell_00_20240805201937.jpg
-rw-r--r--  1 mcp mcp  48K Aug  5 20:33 Doorbell_00_20240805202843.jpg
-rw-r--r--  1 mcp mcp  12M Aug  5 20:34 Doorbell_00_20240805202843.mp4
-rw-r--r--  1 mcp mcp  58K Aug  5 20:54 Doorbell_00_20240805204932.jpg
-rw-r--r--  1 mcp mcp  15M Aug  5 20:55 Doorbell_00_20240805204932.mp4
-rw-r--r--  1 mcp mcp    0 Aug  5 21:39 Doorbell_00_20240805213432.jpg
-rw-r--r--  1 mcp mcp  62K Aug  5 21:39 Doorbell_00_20240805213434.jpg
-rw-r--r--  1 mcp mcp  13M Aug  5 21:39 Doorbell_00_20240805213434.mp4
-rw-r--r--  1 mcp mcp  61K Aug  5 21:49 Doorbell_00_20240805214357.jpg
-rw-r--r--  1 mcp mcp  15M Aug  5 21:49 Doorbell_00_20240805214357.mp4
-rw-r--r--  1 mcp mcp    0 Aug  5 21:56 Doorbell_00_20240805215132.mp4
-rw-r--r--  1 mcp mcp  62K Aug  5 21:56 Doorbell_00_20240805215134.jpg
-rw-r--r--  1 mcp mcp  14M Aug  5 21:57 Doorbell_00_20240805215134.mp4
-rw-r--r--  1 mcp mcp    0 Aug  5 22:58 Doorbell_00_20240805225324.txt
-rw-r--r--  1 mcp mcp   77 Aug  5 22:58 Doorbell_00_20240805225326.txt
-rw-r--r--  1 mcp mcp   77 Aug  5 23:09 Doorbell_00_20240805230419.txt

[nzx-jsantaana]

Is "/sftp-storage/cam-ftp/" the path that the user sees, or is that the path on the local filesystem that's backing the virtual folder? Actions run in the context of a user, so they generally need to work with the user's virtual path, not the "real" backing path. (External system commands are an exception, of course.)

The complication here is what happens if you run an action from a manual trigger, and I don't actually know how that's handled. It's triggered by an admin (who has no virtual folder mappings), not a user - but that would imply that "manual trigger" is basically useless for anything working with virtual folders, so I hope I'm missing something there.

[bverkron]

Actions run in the context of a user, so they generally need to work with the user's virtual path, not the "real" backing path.

This was the key to the issue, I believe. It's really not clear that's the case in the UI and the doc, while expansive, doesn't really explain the behaviour. It just says:

Data retention check. You can define per-folder retention policies.

Additional info and examples would be nice.

However, this feature seems to make it very easy to delete more than you want. It's a bit convoluted to tie to just to a user and their specific share. When I tried to set the path to simply / to catch all the sub directories in the FTP share (which is just for one user right now for dumping the cam footage) it deleted contents from a completely different WebDav share & user in a completely different location because that path seems to apply globally to anything that matches anywhere?

I think I made the necessary adjustments to the event manager to prevent that by tying it to a specific user but I really don't trust this feature to not wipe stuff unintentionally with a slight misconfiguration. It's really too vague as to how to limit it to just what I want.

I think I will skip this built-in retention and keep using my independent Ansible playbooks that run on a schedule to clean-up things in a very clear and specific way.

Thanks for your insights regardless.