Frigate+ Base Model 2024.0 Update

[blakeblackshear]

TLDR: The annotation tool was upgraded to speed up labeling and the base model was updated by using some algorithms to select a diverse set of user images to include in the training set. You must request a new model to get a model trained on the new base.

Annotator upgrades

Before I get into the base model update, there were also some upgrades to the annotation tool in Frigate+ for a faster keyboard workflow:

• The dialog to select a label no longer pops up after creating a box, and the last used label is assigned by default
• Pressing 's' will switch to the next label for the current box
• Pressing 'd' will mark the selected box difficult
• Pressing Tab cycles to the next bounding box (largest to smallest)
• Difficult boxes now have a yellow bounding box
• The order of bounding boxes has been optimized to avoid smaller ones getting stuck behind larger ones
• Smaller boxes can be created
• False positive boxes now have a strike through on the label
• Some other small bugfixes

Two common requests are zooming further and AI assisted labeling. These are coming in a future update.

Base Model 2024.0

Stats on new user submissions since launch

Image submissions have skyrocketed with no signs of slowing down since the initial invitation only launch. There have now been more than 10 times as many images submitted since the first base model was trained. In addition, the first base model was trained prior to false positive submissions in Frigate+, so there are an incredible number of images available for training.

Selecting new images

The first thing I had to figure out was which images should be selected to add to the training set. The typical guidance is to focus on selecting a diverse sampling of training examples.

To do this, I used an embeddings model to generate embeddings on all the images. This allows me to plot those images on a scatter plot. In the following image, every image is a unique dot. The pink dots represent the images used for the first base model.

From there, I used a k-means clustering algorithm to group the images into clusters of visually similar images. This image shows the results of the clustering. Many of the colors appear similar in color, but there are almost 10k unique clusters.

Then I selected one image from each cluster that did not already contain an image in the base training set. This image has a dark blue dot for each image used for the first base model.

This image adds a light blue dot for all of the images I selected to add to the base training set for the second base model. You can see it avoided selecting more images in areas that are already heavily represented.

I went through a similar process for sourcing false positive submissions.

Once I had the images I wanted to include, I had to review every single image to ensure the bounding boxes were accurate and there wasn't anything missing to ensure the base training set isn't polluted.

Performance metrics

Quantifying the improvement isn't exactly a straightforward process. The training set is changing, which means the goal posts are moving as well. A very popular metric used to evaluate performance is the Mean Average Precision (mAP). The specification is the standard for evaluating performance in the COCO competition. I calculated the mAP scores for the initial base model and the 2024.0 model for the most recent validation set which is a diverse sampling of current user submissions.

The mAP for 2024.0 is ~15% higher. I don't think its correct to say that means the model is 15% better than it was before, but this is a standard way to evaluate model performance. I may develop better evaluation methods in the future.

Caveats

While all the metrics from this base model indicate that it is an across the board improvement, I do expect that the differences will result in some new false positives. I personally experienced this when running my first fine tuned model from the new base. You may need a second training to address them if that happens. I believe this will occur less and less frequently over time as there are false positives submitted from multiple base model versions.

More to come

• I have already started working on the next base model
• I think there are ways to further improve the way false positives are incorporated into the training set by focusing on false positives that are persistent after several trainings
• At some threshold it will be possible to start grouping users with similar scenes into more targeted datasets
• A framework for expanding the label set is high on my priority list

[ccutrer]

IIRC the first base model had ~15k images on it. How many does 2024.0 have? And do you think that is even a relevant metric?

[blakeblackshear]

I added about 5000. I do think it's a relevant metric for now.

[IsaacInsoll]

Thank you so much for all the work you do!

roughly how many model releases do you think you will do this year? Just a ballpark figure would be great so I can “budget” my tokens accordingly as Australia has a bad exchange rate right now so I’m wanting to make my token allocation last the whole year 😀

[blakeblackshear]

I'm planning to target a quarterly release at this point.

[crankin91]

Sorry to hijack, but curious to ask another Aussie, do you get false positives on your wheelie bin? I've had to place object masks over mine because they get picked up ask cars, people, dogs, you name it.

[pyrodex]

Love this!

Three questions on the models though and Frigate+. I recently renamed my cameras and causing them to be duplicated but in more readable as shown in the example below.

1. If I retrain now, did a recent retrain BEFORE the rename, should I wait and submit more examples of both false and positive to get my numbers back up to retrain and get the new base model or are the names of my cameras and the images submitted for them irrelevant in the model I get back from the retrain?
2. I retrained around 1/9 so would my most recent model HAVE this already or was it after 1/9?
3. Does the difficult flag do anything yet? I see it mentioned in the news here but not sure so I don't use it often but not sure if we should be doing it and when is the best practice.

Thanks for all the hard work!

[kiloptero]

sorry to post this but.... this mean if we pay for frigate+ ( i will be so happy to do that) we can use a better model for the objects? maybe its a stupid question

[pyrodex]

sorry to post this but.... this mean if we pay for frigate+ ( i will be so happy to do that) we can use a better model for the objects? maybe its a stupid question

Paying allows you to train your own models. So far the 50$ is worth it in my opinion. I've worked hard to submit false and true positives with annotations for a few months and it has paid off. I was even able to tune back some of the more complex filters for objects I had and still got less false positives.

[blakeblackshear]

If I retrain now, did a recent retrain BEFORE the rename, should I wait and submit more examples of both false and positive to get my numbers back up to retrain and get the new base model or are the names of my cameras and the images submitted for them irrelevant in the model I get back from the retrain?

Names of the cameras are irrelevant for training.

I retrained around 1/9 so would my most recent model HAVE this already or was it after 1/9

I just updated today, so it would be on 2023.0 base. You can see the base version in the models table now.

Does the difficult flag do anything yet? I see it mentioned in the news here but not sure so I don't use it often but not sure if we should be doing it and when is the best practice.

I have been recommending that you use it in situations where objects are truly difficult to identify such as a car barely visible through a bush. I am not using it in training yet, but I believe it will be useful in the future.

[kitchung]

Can we please have a tag for USPS just like we have for FedEx, UPS and Amazon, @blakeblackshear ?

[NickM-27]

This has been asked many times and is coming along with other labels in the future, as was referenced at the end of this post.

[kiloptero]

thanks nicolas... any change in the frigate more than the api key in the config of the addon? Andres Roepke Del Solar. Ingeniero Civil Industrial Ingeniero Electronico Mobile 88196932

…

On Sat, Jan 27, 2024 at 21:16 Nicolas Mowen ***@***.***> wrote: This has been asked many times and is coming along with other labels in the future, as was referenced at the end of this post. — Reply to this email directly, view it on GitHub <#9466 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIOTKWXRDISKBQ2LJZUTULLYQWKGVAVCNFSM6AAAAABCNR52C2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DENRYGI2TE> . You are receiving this because you commented.Message ID: ***@***.*** com>

[NickM-27]

https://deploy-preview-6262--frigate-docs.netlify.app/integrations/plus#use-models

[kiloptero]

yes, i read that...

model:
path: plus://e63b7345cc83a84ed79dedfc99c16616

the e63.... will be the name of the model? and what about the others? need to use more path?

[dimatx]

'e63...' will be replaced by a similar looking alphanumeric string that is the reference to the model that you'll see in your frigate+ portal once you have trained a model using your images.

[nrpetonr]

good to see frigate constantly improving. what is the timeplan for openvino?

[blakeblackshear]

It's hard to say because I don't quite know what it will take to get there. It could be soon if it's really simple, or it could be a while if it requires changes on the Frigate side. I know that's not really a useful answer.

[BNWEIN]

Great work as always! I’ll be training a new model later today.

A question;

If I am not mistaken the number of positives and false positives we submit is important when it comes to training, if you submit say 100 false positives and 100 positives, then only 20 if your false positives will be used during training. On that basis it would be really helpful if the portal showed us the number of positives and the number of false positives we have submitted. Will that be coming and if so, any idea when?

[blakeblackshear]

It's slightly more complicated than that. Technically, only 20% of training examples can be images without any labels. So a false positive where a cat was identified as a person wouldn't count against that 20%. A false positive where a chair was identified as a person and there were no other labeled objects in the image would count against that limit.

I do have statistics like that in my backlog. Not sure when that will be my top priority as I also have OpenVino support and expanded labels at the top of my list.

[gacekk]

Maybe a stupid question. Do I need to train any pictures myself and if so, how many? I can't see any models in models section

[NickM-27]

https://deploy-preview-6262--frigate-docs.netlify.app/plus/

In order to request your first model, you will need to have annotated and verified at least 10 images. Each subsequent model request will require that 10 additional images are verified. However, this is the bare minimum. For the best results, you should provide at least 100 verified images per camera. Keep in mind that varying conditions should be included. You will want images from cloudy days, sunny days, dawn, dusk, and night.

[gacekk]

Thank you. And is that 100 images per camera per object type? To be honest I would be fine with having several: car, person, and couple animal types such as cat, dog, fox, roe, pheasant

[NickM-27]

100 images per camera is what it says

[tetele]

Two common requests are zooming further and AI assisted labeling.

I have another suggestion: could you use Frigate+ to trigger a detection on a snapshot using the locally installed model and hardware? Usually snapshots sent to Frigate+ have a single detected object (or false positive), but I'd like to use the model to detect everything and then just tweak the bounding boxes/labels/etc.

I guess a local API (accessible from the workstation you use to access Frigate+) could be called using a web protocol to send the snapshot back from Frigate+ to the local Frigate instance.

[NickM-27]

but I'd like to use the model to detect everything and then just tweak the bounding boxes/labels/etc.

that is what is meant by AI assisted labeling

[pyrodex]

Two common requests are zooming further and AI assisted labeling.

I have another suggestion: could you use Frigate+ to trigger a detection on a snapshot using the locally installed model and hardware? Usually snapshots sent to Frigate+ have a single detected object (or false positive), but I'd like to use the model to detect everything and then just tweak the bounding boxes/labels/etc.

I guess a local API (accessible from the workstation you use to access Frigate+) could be called using a web protocol to send the snapshot back from Frigate+ to the local Frigate instance.

I would like to see a feature along these lines to "inject" a picture from a camera that maybe Frigate missed. Like if we do 24/7 recording or motion but the object detection failed it would be nice to take a high resolution screenshot of that still from the camera and inject it into either Frigate or Frigate+ as like an ad-hoc process to then get it trained into the models.

[NickM-27]

There is a pinned feature request for that as well #8038

[tetele]

The new keyboard shortcuts are honestly insanely good and welcome. I think they just cut the time I spent labelling images to about a quarter ❤️

Thank you so much for them!

[NathanVanCleave]

Yes! They're so much faster and make me want to keep annotating when I realize how many I'm going through.

[canadarob]

Thanks for all your work! I updated yesterday morning as I also happened to have 700 annotated images ready. I won't 100% know the improvements are all the model but so far there has been less false positives. And a HUGE improvement in face recognition for my use case

[mattthewclayton]

Two relatively random questions, but it seems like the correct thread to ask them on:

1. Do you track which users submissions are included in the base model and training data? I assume one day it might be useful to share the dataset more widely or to join it with other datasets, I’m not sure if our current agreement prohibits this happening or not, and I personally don’t care, but it would be nice to know if my face and licence plate are going to be out in the wild one day.

2. I sometimes forget to lock my car, I manually check the camera to see if it’s locked because it folds in the wing mirrors when it’s locked, so there is a visible cue in the images. I’d love to use frigate and HA to tell me if my car is left unlocked. Do you foresee this level of customisation of the labels ever to be possible, or do you think it’s quite unlikely? (I understand that training a model for this specific purpose would be quite easy for a single user, but making Frigate and the tooling around it capable of this level of customisation is the real challenge.)

And as always great work, Frigate is by far the most useful and stable piece of home automation (paid or otherwise) that I use.

[blakeblackshear]

1. This is very clearly outside the bounds of what I am permitted to use the images for. I am exclusively permitted to use them to provide services back to Frigate+ subscribers. That can't change without a notice about a change and everyone has the ability to request all their data be purged at any time.
2. I think its possible and would be done as a downstream classification model similar to how others have implemented license plate and/or face recognition. Classification models seem to be a bit simpler since they are more targeted.

[NathanVanCleave]

I sometimes forget to lock my car

I added contact sensors to my car door locks and bought a spare FOB to wire up with ESPHome to automatically lock the cars. Having older vehicles made this cheap and easy, but might not be as feasible on newer vehicles.

[nulkathos]

A question about false positive submission.
Now automatically, when in wrong detection and sending to frigate+ , a negative label is added but, why not adding the possibility to enforce (if there are any other object that something will be detected as wrong ) a negative label?

[blakeblackshear]

When submitting a false positive from Frigate, it includes additional metadata such as the exact region passed to object detection, the confidence and the model that was being used when it happened. Without that metadata, I can't effectively reproduce the false positive.

[davefx]

I'm still missing the feature to report false negatives (that is, easily-submit images for which no objects had been detected).
For example, I'd like to upload images (specially night-vision images) from recordings from several cameras for which no person detection has happened. As we know, night-vision detection is something what doesn't work very well with the COCO model.

I think that probably most of the current images in the database only include images with correct or wrong object detections (either detected by the default COCO model included in Frigate by default, or by the different Frigate+ models), as uploading images from recordings at this moment is a pain in the neck.

[NickM-27]

It will definitely be easier in 0.14 but I've also found it's not that bad currently. In chrome there is a feature when you right click on a video to save that frame. Just do that a handful of times and then upload to frigate+

Definitely worked well at the beginning to give some specific examples to my frigate+ model.

[xavan83]

Do you plan to "offer" a possibility to test the Base Model 2024 during a few hours/days, before taking a subscription?

[blakeblackshear]

I don't. There isn't a way to do that without giving it away for free. I have no way to disable it after a few hours/days.

[ykreal]

Considering the constraints you've outlined, I'd like to explore an alternative approach to evaluate the effectiveness of your models. Would it be possible for users to submit some images to be used as a testing dataset? Following this submission, could you provide us with the predict results for evaluation purposes?

By enabling users to assess the performance of the models based on their own datasets, we believe it would offer valuable insights into the suitability of your models for our specific use cases. This approach would also allow us to make more informed decisions regarding potential subscriptions.

[blakeblackshear]

At some point I would like to offer this @ykreal

[YBonline]

• The dialog to select a label no longer pops up after creating a box, and the last used label is assigned by default

This change has led to me making a few mistakes, I'm sure I'll get used to it, but I'm sure others are as careless as me lol

Can I request perhaps that next to the add button, just put the label list choice right there, it can retain the default/last choice still without a prompt, but make it easier to change first before drawing the label so we don't forget?

Also I just trained my first model on 2024.0 and my perpetual false positives that I've been struggling with seem way down, at least in the first couple hours! I could never get the old model to realize an umbrella without a person is not a person, and there were quite a few chairs that the old model would insist someone was sitting in, no matter what, but 3 hours in, it finally figured that out! Great work, great improvements!

[blakeblackshear]

I have some ideas for ways to address this when I make the next round of enhancements.

[YBonline]

Hmm, I was excited when I switched yesterday to the new model and stopped seeing the continuous false positives with the umbrellas and chairs that are sometimes occupied but always indicate a person, but apparently I've just trained out a lot of them completely. Now I'm having false negatives everywhere that apparently I'm missing tons of events (for the last day).

I upped my threshold to 90% under the previous model because I had so many false positives, and all 99% of the true positives seemed to hit 90% anyways, so maybe I need to try to drop it, but I do seem to be struggling with this. For the moment I switched back to my previous model as my next week has quite a busy schedule, but I figured I'd mention it here before trying to create yet another model and trying to get more training in.

I guess it might also be I don't have enough training data. I got my last standalone DVR integrated with my Frigate installation and now have 38 cameras (and resource usage seems to be good, motion detection issues better then they have ever been). I only have 823 verified images, which I guess means I'm very, very short of the recommended minimum 3,800 trained images for my installation (some of them are very tough, there are a number of cameras that are in areas that are supposed to get 2-3 people a week walking by them, unless something is wrong (hence the camera).

If you feel like taking a look to see if I'm doing something poorly or wrong, I'd appreciate it, but I won't complain any more at least until I hit the recommended minimums!

[borgqueenx]

I read about people reporting here having way less false positives, but i wonder how this is for "stubborn" spots. I have hanging chair outdoor that moves a bit in the wind and frigate keeps thinking its a person. I submitted like 150 images teaching the model it is not a person, retrained, use new model, and now often it reports the chair as person with higher then 90% confidence.
So in my experience submitting purely false positives is not working at all.
I will soon try to sit in the chair, stand in front of it, in day and darkness, and then hopefully submitting true positives will make the model learn what the difference between a person and chair without a person is.
I also have a coatrack that shows the same problem. At least here i can understand why it sees it as person due to the coats. I am collecting true positives in front of the rack because here i also collected 100+ false positives and themodel did not learn from it unfortunately.

[NickM-27]

Most of my false positives went away quickly, I had a couple very stubborn spots being: railing on the backyard deck and an assortment of rocks around a tree. After submitting ~200 examples of false positives and ~100 or so examples of true positives in the area the false positives have not occurred at all for multiple months

[ccutrer]

I also have a coatrack that shows the same problem.

My coat rack is one of my most stubborn spots. Before Frigate+ I basically masked out the entire hallway, missing all true positives, because it was so bad. I still mask out the wall, and don't miss events anymore, but I still get fairly regular false positives that extend from the wall down to the floor. It's definitely improving with time. But yes, 100 identical false positives won't help the model much. I try to submit as many true positives as I can in that area that are slightly lower confidence, but I also go and re-arrange the coats whenever I get a persistent false positive, and try to submit one or two from each configuration (or more if lighting changes). I'm hopeful with my next training, based on 2024.0 (I've got a backlog of already submitted images I still need to annotate before I kick off another training), I'll be able to raise my minimum confidence threshold and my false positives will almost completely disappear.

[Saduff]

I'm trying to get started with Frigate+. Found the following in the docs:

The baseline model isn't directly available after subscribing. This may change in the future, but for now you will need to submit a model request with the minimum number of images.

But how do I submit images of packages for example if the default model doesn't track them? Or do I just have to submit images of objects that are tracked (e.g. car, person) and only after getting my first model I can start submitting images of packages for a future model?

[blakeblackshear]

Also from the docs:

This model will support all label types available even if you do not submit any examples for those labels.

https://docs.frigate.video/plus/first_model

[NickM-27]

In most cases a package will be included in a person event as the person delivers the package or the delivery truck leaves.

That said, there's no requirement that you need an example of every label submitted. It's more important to get his diversity in the weather and times of day for the submissions like the docs suggest. Then after you get your first plus model it will be easier to submit those new labels

[LandoGemini]

I have a set of 7 cameras and I was planning on moving the angle of two of them, will that negatively affect the models I've trained?

[blakeblackshear]

It could. It depends on how different things look. I doubt you will see a major change though.

[turboc1208]

Might be nice if we could get a count of the different labels we have used and from which cameras. That way we can send an appropriate balance of object types for each camera.

I've got 9 cameras and I'm finally up to about 50 on each. Most are dogs and the same cars parked outside my house and in the driveway. Do you really want that many repetitive pictures? I'm getting them at different times of the day, and in different weather conditions ( it rained last night ).

Also, in the future do we need to submit 100 from each camera or just the one we are trying to improve the performance on?

[NickM-27]

You can view the frigate+ roadmap here, statistics for uploads is listed as later https://github.com/orgs/frigate-nvr/projects/2

Also, in the future do we need to submit 100 from each camera or just the one we are trying to improve the performance on?

100 is recommended as the starting point. In general once you have your first model you only need to submit images for cameras that have false positives or false negatives, but you may find better results submitting from a mix of cameras where objects score lower.

[dimatx]

I'm finding that the model does not do nearly as well at night for cars. Using Dahua TM5442AS cameras with IR. Pointed at a driveway. Don't seem be to able to land on a sweet spot threshold that works well to avoid false positives during the day while still identifying cars at night. Wondering if others see the same deficiency.

[dimatx]

Hey @NickM-27 - Been using:

      threshold: 0.90
      min_score: 0.50

And still not getting the car detected, here are two more new examples. Wondering if you have further thoughts.

[NickM-27]

you would want to run the video through frigate again and record the debug live view so we can see what frigate is seeing https://docs.frigate.video/development/contributing#2-modify-your-local-config-file-for-testing

[NickM-27]

Some of these angles are difficult as you are basically asking frigate to detect 2 bright spots as a car. They are detected though so you may just need to decrease min_initialized

[dimatx]

Do you think increasing the FPS (from 5) has the potential to help here, or probably not.

[NickM-27]

Yeah it could help

[Pick-Nick10]

What metric are you using to decide that cars are not detected well (just want to understand the full picture). it might be good to see some examples, my main camera on the front of the house that sees cars is a color night camera but it doesn't have any issues detecting cars at night

Hey @NickM-27 how are you getting multiple objects detected in a single snapshot? I've only ever seen one object labeled in any of my captures at any given time

[NickM-27]

this is the debug live view, not a snapshot