Recordings visible in the WebUI, but not present in the configured directory #837
Comments
|
I think I might know what I did wrong, so it seems in the dev version the configuration has changed and the In docker compose I now changed my And in Viseron I added: Will need to make a few tests where something gets detected and recorded to see if the recordings finally end up where they should (for example if they will get moved after a shutdown). |
|
Retested with the new storage settings, something is still off, the log shows: Note the log entry: Now looking inside of the container ( Now, gracefully stopping the container, hoping that Nothing in the logs indicates it was going to move the recording anywhere? The recordings directory on the host shows the correct subdirectory structure, but has not Is this a bug or what am I missing? |
|
Sorry for the confusion! The In v3, the cameras recordings are stored as short If you are in need of having the events as mp4 files, just like v2 did, you can set the The logline and the folder creation is missleading since the file is never created, will fix that right away |
|
Thank you for getting back to me! Yes indeed I saw the segments on ramdisk, but assumed they were something internal, especially seeing in the logs how they get regularly deleted. I was indeed looking for one complete clip for the whole event, expecting it to end up in I now found it in the sources and have a question regarding this remark:
Does that mean, that after the event fragments get concatenated into one event mp4, they will not get cleaned up? If so, why? My naive assumption would be, that it is not necessary to keep the same data twice or is there a technical limitation? Thanks again for helping out, I'll test it shortly. My main issue at this point is the terrible detection quality of the Coral TPU regardless of the model used, but that is not a Viseron problem :) |
|
Did a short test with the suggested setting and the complete My assumption was, that Does this work differently or is my configuration not correct? |
Correct, the fragments are not cleaned up since they are what Viseron uses mainly. The Viseron UI does not utilize the MP4 recordings at all, it only displays HLS playlists using the m4s files. |
That should be working, let me investigate. Just a note on your configuration, the second tier pointing to |
OK, floating some ideas:
Yes, thanks for the hint, I noticed 😅 |
|
I have more logs and a somewhat strange behaviour, I should probably update my dev image to make sure I am running the latest code. So, after making some changes to the configuration (enabled more detection labels) I hit the restart button in the UI, which resulted in the following backtrace: the startup after that looked fine, so I assume all is good, but when I returned to check the events I saw that nothing new has been recorded, although I was outside to test detection. I checked the logs and saw the following errors: Not sure what happened there and why the files were missing. I did not restart the whole container, I only restarted Viseron via the UI. The encoding error is also quite strange, I do not use anything except UTF-8. |
The segments are stored for as long as they are wanted. I do see the need for storing the MP4 files, and you are not the first to have similar requirements. storage:
recorder:
tiers:
- path: /config/tier1
continuous:
max_size:
mb: 50
events:
max_size:
mb: 100
event_clips:
max_size:
mb: 1 # Setting a low value would essentially mean they get moved instantly
move_on_shutdown: true
- path: /config/tier2 # Store only event clips in this tier
event_clips:
max_size:
gb: 100
- path: /config/tier3
continuous:
max_size:
mb: 50
events:
max_size:
mb: 100I will have a look at how complicated that would be. v3 has been in beta for ages now and i would like to get it released soon and i can work on polishing it further later on. |
Yes i noticed that as well,
The missing files most likely a side effect of the unclean shutdown.
Indeed, never seen that before. Can you enable debug logging and see if the error keeps appearing? logger:
default_level: debug |
I guess it is :) So, my understanding was that an "event" is something that triggers a recording when it occurs, so essentially in the end you get a recording of the event. Now, seeing your config proposal I think I did not get that right, it seems there are "events" and "event clips" which is not the same? Ah, I think I know what you mean, an "event clip" is the complete mp4? I am not sure if these names are not too confusing... From the user perspective I consider the gazillion of segments something internal to how the application works, focus of those is to show up and be playable in the UI. They should get rotated according to the configured rules to ensure the system does not run out of space, but otherwise I do not think that anyone or anything except for the application itself is doing anything with those files? The complete mp4 files, as you called them "event clips" are actually what a user would want to grab and copy out of the Viseron storage and save them permanently, the assumption here is, that these files are needed outside of Viseron. Let me float yet another idea :) So, as long as I am within Viseron (i.e. using the UI) I do not really care what format the files are in, the UI plays everything nicely and I can look through the events to check what happened. The format only becomes interesting as soon as I see something caught on camera, that I need to store for myself permanently. This is actually the use case for having an mp4 clip of the event in the first place. So, perhaps a more intuitive way would be to offer some sort of an export button in the UI, allowing to convert selected events to "event clips" and have them saved in a dedicated storage tier. This way there is no automatic duplication as there is no real need for bulk-saving the mp4 clips - they are only needed when the user saw an event that needs to be saved permanently and since reviewing the events is anyway a manual step, hitting a button in the UI to generate an mp4 clip should be totally fine from a user's perspective. What do you think?
I totally get that, as a developer I also do not like when something does not come to an end and just drags on and on, better to get out a stable release with lesser features and then follow up by smaller releases, than polishing a scope-creeped beta release for years :>
Unfortunately I was not able to reproduce it, i.e. the exceptions during the shutdown are there, but the exception at start did not occur anymore. Perhaps some UTF-8 characters got truncated when saving due to the shutdown exceptions and produced illegal UTF-8, which in turn caused an exception on startup? Just guessing.. |
I think placing the segments in the
Yes that is correct.
Yes having a way to selectively export seems to be a good resolution. |
Hmm, as a user who is new to Viseron, "recordings" was the name where I expected to find the actual coherent mp4 recordings. One has to actually dive a bit deeper to understand the difference between "recordings", "clips" and "segments", actually if short segments are stored in "recordings" and if we also conclude, that those short segments are something internal to Viseron, then why isn't the directory where they are stored called accordingly? If you have to keep the names as they are now, then indeed having My later argument was, that if we can get the export feature, then having
Oh it's on the list? I did not see it mentioned anywhere in #619 which is already merged?
Awesome! If there is anything to test, please ping me. I am currently setting things up and waiting for the remaining cameras to arrive, so doing a lot of tests and experiments at the moment. |
Yeah a lot of people seem to have the same expectations as you have.
Sorry my bad, its this PR: #716 |
Indeed, I'd be interested to read and understand how you see these things, because the separation of "Events" and "Recordings", where both are playable videos of the same occurrence is not quite clear to me 😅
No problem, as a user I'm happy that you are interested in feedback and are considering to tune a thing or two. I've been running an open source project too back in the day, so I know how much work it actually is :)
Ah, this must be the point "Allow export of timespan"? |
|
I am still battling the "clips" issue, so my config has: So my expectation was that it would now create those I see the event in the UI, but there are no mp4 files corresponding to this event. A few mp4 files with earlier timestamps have been created, but last I see is from 14:00 while I am looking for an event after 15:08 I assume it is available in the segments format since I can play it in the UI, but dozens of files named as Also, when I go to "Recordings" in the UI, I only see one video from today, although I have dozens of events which I expected to end up as a "recording", I guess this brings me back to the earlier question on what the difference between "Events" and "Recordings" is and what it is good for? I do not see any errors in the log, so I really have no clue why no mp4 "clips" have been created for these events, but I'd really like to download them. Any hints on how to figure out what segments belong to which event, without having to go through everything manually? |
Yes my plan is to allow export of individual events or by specifying a time range. |
|
Could you paste your config? I tried (with your filename_pattern just to be sure) and i get clips created properly. abc@458814191c5f:/workspaces/viseron/config/tier3/recordings/viseron_vscode_camera/2024-11-25$ ls -al
total 5576
drwxr-xr-x 2 abc abc 4096 Nov 25 22:05 .
drwxr-xr-x 9 abc abc 4096 Nov 25 21:55 ..
-rw-r--r-- 1 abc abc 1977969 Nov 25 21:55 21:53:44.mp4
-rw-r--r-- 1 abc abc 1887373 Nov 25 22:01 22:01:04.mp4
-rw-r--r-- 1 abc abc 1831587 Nov 25 22:05 east_2024-11-25-22:05:04.mp4Note that they are stored in UTC so depending on your timezone they might not appear where you think they are. |
OK, sorry, no need to check the config - this was exactly it, the recordings were made, but time was off, and since I did not see any recordings for after 3pm. I thought that they were missing. Which brings me to another interesting question: how can I, as a user, map the time I see in the UI to the timestamp of the filename (of course knowing this nuance one could "manually" convert the desired time to UTC and figure out what the clip name might be)? I would expect them to be the same, since I am interested in a recording of a particular event and the time when it happened is important. Is the issue with my pattern From a software engineering point of view I understand, that using UTC eliminates all sorts of problems (especially the daylight saving time jump horrors), but when exporting the clip I would expect it to be saved in the same time format that is being displayed in the UI, otherwise it's just confusing... Would you be open to the thought of tuning this (or perhaps adding an option for this) at some point? |
|
I agree, it should definitely be in the servers timezone. Will fix for the next beta release |
|
Just released It contains some fixes for the |
|
Cool, thank you for the update, will pull it right away. Going back to the question of "Events" vs "Recordings" - I was hoping you could explain the idea between these two? I understand, that for 24/7 recordings it becomes unrelated to events as it's just a continuous recording no matter what, but I do not think that I have that enabled and I'm still having a hard time figuring out what the difference between an "event" recording and a "recording" recording is? EDIT: need to revisit the statement about missing recordings, may have been a user error :) |
|
I noticed one more issue which I have been following previously and which is still there in the latest dev version: after midnight I check the events, naturally it shows only the ones from the current (new) day. I go to the calendar icon, so that I can select the previous date (i.e. before midnight), but I can't click the previous date - I guess, because it thinks, that there are no events there. I know however, that there have been various events during the day and they have been visible before midnight. Has anyone noticed this? @roflcoopter please let me know if I should rather open a new issue about this particular topic. EDIT: today is the first time that I saw this work - not sure why it did not work previously, but today I was able to select the previous day after midnight. |
|
Tested the feature which should move the recordings after shutdown from ramdisk to the next tier. According to the log this seems to work now: |
Hi,
I am not quite sure if this is similar to #797 but I simply can't find my recordings, although they are shown in the web interface.
I am using the dev version, the relevant docker compose config:
In my viseron config (I know it's default) I specified the recordings path anyway:
On the host the directories
recordings/camera_1/2024-11-18/get created, but there are not files there.I do not see any errors in the logs:
However, the file:
/recordings/camera_1/2024-11-18/east_2024-11-18-14:08:47.mp4is nowhere to be found. I tried attaching to a shell inside the docker container and the recordings directory is there as empty as on the host:I tried searching for the file in the container root, also in the host system root and I do not see it anywhere. Yet
http://nvr:8888/#/recordings/camera_1/2024-11-18shows quite a number of files which can be played back in the browser.So... where are they located? :)
The text was updated successfully, but these errors were encountered: