-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BigBlueButton 2.3 - invalid recordings #81
Comments
Do you have the RECORDINGS_METRICS_READ_FROM_DISK optimization turned on or do you get recordings data from BBB's API? |
Turned on. |
Have you managed to figure this out? |
in my case I see published records == unprocessed records
|
one more thing: |
I have the same problem with @amg-web. unprocessed record number never drops even though everything is already published. I am using bbb 2.3.13 and exporter 0.6.0 |
|
|
Does the unprocessed metric have the value 4?
…On Fri, Sep 10, 2021, at 4:06 PM, semzor wrote:
> 1. Do you also have the RECORDINGS_METRICS_READ_FROM_DISK <https://bigbluebutton-exporter.greenstatic.dev/exporter-user-guide/#optimizations> enabled?
> 2. How many files do you have under `/var/bigbluebutton/recording/status/sanity`?
1. Yes it is enabled in my config.
2. I have 4 files under `/var/bigbluebutton/recording/status/sanity` right now.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub <#81 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADPHVUUDFS4MWCCZVMRNXQ3UBIGGTANCNFSM4W37VLWQ>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
sanity files is not deleted in BBB v.2.3.x |
I see, do you maybe know where the unprocessed files are being stored now? We need to fix this. |
sanity files was removed by /usr/local/bigbluebutton/core/scripts/rap-process-worker.rb |
Thanks for the detailed report, I'll try to look over it over the weekend. |
Hello, Then i would guess that modifying collector.py:345 Current versions of BigBlueButton use this "recorded" status in order to determine that a recording is pending. I wanted to update the code in my server but did not find how to change code inside the docker... Saludos! |
I wasn't able to dedicate any time over the last two weekends. :/ But the suggested fix seems trivial enough to give it a try right away. I created a preview release, docker tag v0.7.1-preview1. Give it a try and report back :) |
I have tried and it seems a fix for unpublished stat. However, the processing stat is still incorrect in my instance (now using version 2.5) because of the three failed cases I mentioned earlier, should error metrics to be added and clear the processing metrics? |
I tried several times using v0.7.1-preview1 to see if the metrics goes above zero. But it did not. At first i thought that maybe it was because i did not have enough simultaneous recordings on the tested server. But then i gave up. It was always Zero. I could not determine if the documentation is wrong and .done files are never created in "recording/status/recorded" folder. Maybe they are created and deleted too fast. get all files in recording/status/sanity in an array |
Oh, I haven't noticed that it has never been increased. After inspection, it seems recorded done files are removed after being watched, created relevant raw folder and enqueued for next process, that's why it seems no increase, because it is nearly processed and removed instantly. A recap of what I found and thought before:
All the above are based on status level. However as mentioned in their docs, I only need to care about /var/bigbluebutton/published /var/bigbluebutton/unpublished /var/bigbluebutton/recording/raw when moving recordings from another server. This will however become unable to republish the recordings and can only rebuild the recordings if relevant raw files exist. And besides removing the done files mentioned above, rebuilding recording also removed relevant entries in published, unpublished and deleted folder and enqueue the meeting id directly to sanity stage and published.done = published + unpublished + deleted. So, I guess maybe monitoring the output folder instead of the status folders will be better: PUBLISHED= while deleted may not be correct and/or may not exist with different definitions (e.g. soft deleted), the same also goes to unpublished case, but I guess this should be enough. And for Processing and Unprocessed, the definitions of the default workflow stages are as follows: Since "Recorded" state changed too fast, cannot use recorded status as unprocessed base, sanity.done = processed.done + processed.fail, so maybe unprocessed should be :
may also minus published status count but that is skipping flow, we may also skip handling that. And for processing, maybe: And may add one or more error metrics, since sanity, process and publish phrase may have fail results. |
@balbertho Hi.
This solution is light and will work on new servers. However i think it is not accurate to assume that processed.done file has to exist for a recording to be pending (unprocessed) because the bbb-record logic does not. For instance, as far as i understand, if there is a sanity.done file and the recording is published it will not be processed again even if the processed.done file is there. In fact, some people do clean that folder on purpose. On the other hand the approach i propose, i believe is exact in all scenarios. The only disadvantage i see is that if there are too many files, it will be slower. A possible solution would be to implement both methods and allow to switch them via configuration since (i think) most cases will be covered by @balbertho 's method? @greenstatic any thoughts? |
Hello again.
is the best solution. @greenstatic Could you please include this in the next version / preview? |
Potentially resolves #81. This second fix implements: #81 (comment)
Thanks @balbertho and @pabloapico for your investigation. I've created a build v0.7.1-preview2 with the unprocessed count ls /var/bigbluebutton/recording/status/sanity | wc -l MINUS ls /var/bigbluebutton/recording/status/processed | wc -l recommended fix. Please report back with your findings. |
Thanks @greenstatic for changing the code and @pabloapico for supporting me. However, I think I have made misleading statements before. Since now it seems that "unprocessed" metric now is equal to "processing" count without all fail status at least for time I am monitoring, and it sounds a little bit weird. Not only processed.done and sanity.done files, in fact all status files can be deleted safely except recorded.done files which are monitored and deleted by bbb-rap-starter.service. And then what it does is to create a raw folder for that recording and enqueue the archive task to Redis database, then delete the recorded.done file, which is usually less than 1 second work. And for other latter sanity, process, publish etc. are done by bbb-rap-resque-worker.service, which enqueue for next task flow to Redis database after previous task flow script completed. That's why the main folders are only /var/bigbluebutton/published /var/bigbluebutton/unpublished /var/bigbluebutton/recording/raw as mentioned before. By default, /var/bigbluebutton/recording/raw will be cleaned up also but once cleaned up, the corresponding recording can never be rebuilt again. /var/bigbluebutton/unpublished is not cleaned up automatically and can be used as the base for republish. So, the real "unprocessed" metrics seems to be that the recordings raw folder exist but no corresponding (archived.done files or archived.norecord files) and sanity.done files and not in process status folder (which are the one processing or failed). But this will need to keep track of a list of recordings not yet processed. Here is a log for inspection to 2 critical bbb-record services discussed above with more than 1 meeting ending at nearly the same time:
|
Could you try the latest release v0.7.0 (just released now) and see if the metric is correct? Please also mention which version of BBB you are running. |
Something seems messed up in combination with BBB 2.3.
Fresh install, shows X deleted recordings.
Unfortunately, for a fresh recording, there should be NO recordings?
Also, the imported recordings do not count up correctly, and there is one "unprocessed"...?
The text was updated successfully, but these errors were encountered: