Use Frigate mask definitions #79

ragenhe · 2021-08-25T21:53:30Z

Great project!! Thank you for all the work.

It would be great if we could copy / paste motion masks from our frigate setup.

Or maybe there is something already similar.

My problem is Face Detection is running against my TV as a second person.

phammichael · 2021-08-26T00:04:30Z

I have the same problem with my TV and have been trying to figure out a solution.

jakowenko · 2021-08-26T14:12:49Z

Hey there! Thanks for checking out my project. I can definitely add in support for the motion masks. Are you thinking that anytime the bounding box of the face intersects the motion mask that the result wouldn't be saved at all?

ragenhe · 2021-08-26T14:20:39Z

Yes. Or maybe not even identified / processed.

Basically just ignore the masked area.

jakowenko · 2021-08-26T14:47:43Z

To know where the face is the image has to be processed first, but if it falls within the bounding box, I won't save the result at all. So it will never show up on the UI.

ragenhe · 2021-08-26T15:00:25Z

Awesome! Thank you!!!

phammichael · 2021-08-26T15:49:36Z

To know where the face is the image has to be processed first, but if it falls within the bounding box, I won't save the result at all. So it will never show up on the UI.

As an alternative solution, can the image be cropped before it's sent to Facebox? For example, crop the area of the image where the TV is, and then send the image to Facebox to process the image and detect any faces.

jakowenko · 2021-08-26T20:58:10Z

To know where the face is the image has to be processed first, but if it falls within the bounding box, I won't save the result at all. So it will never show up on the UI.

As an alternative solution, can the image be cropped before it's sent to Facebox? For example, crop the area of the image where the TV is, and then send the image to Facebox to process the image and detect any faces.

I like your idea better. I can add a black mask to the image then process that and any results should be ones that fall outside the mask.

jakowenko · 2021-08-27T05:28:14Z

Hey @ragenhe and @phammichael, I just pushed a new beta build to include support for this feature.

You can now add a masks property to your config which includes an array of masks each with a camera and mask definition (this should be the same format as Frigate). Once an event comes in the API will determine if there is a mask defined for that camera and if so apply a black mask to that area. Below is a sample of what it would look like.

masks:
  - camera: back-door
    mask: 0,0,500,0,500,500,250,800,0,500
  - camera: back-door
    mask: 100,30,250,0,500,1500,250,800,10,500
  - camera: garage
    mask: 800,0,1200,0,1200,800,800,800

Currently I keep the black mask in the final image since it makes it easier to debug that it's working. But if you are happy with this I can make it so only the masked image is passed to the detectors, but the unmasked one is saved for the matches page.

Here's a sample image I used to test the masking and an example of how it would look.

Let me know what you think!

ragenhe · 2021-08-27T13:16:46Z

Smoke test has been successful.
I'll be back after more testing.
Thank you!!!

ragenhe · 2021-08-27T13:31:43Z

I believe (assume) the masked area is being ignored as a person on my TV was not identified as a second person.
However, on the matches page, for "unknown" persons, there is no mask visible.
This probably as intended, lol, I'm just making sure I report back what I'm seeing.

jakowenko · 2021-08-27T13:56:28Z

The mask should show on the image, but I just thought of something that may be causing it to not be on it.

The Frigate mask is defined for the resolution of those cameras, but in Double Take I pull in the images from the API at a different resolution to keep processing time down. By default the Frigate image height is 500px, which means the mask defined at the camera resolution is different. I'll need to think of a way to resize that mask if possible, does what I'm saying make sense?

I was able to get the mask to show up by changing the default Frigate image height in Double Take to the same height of my cameras, in this case 1920x1080. The mask then showed up on the matches page image.

frigate:
  image:
    height: 1080

ragenhe · 2021-08-27T13:59:04Z

Makes perfect sense. IMO I don't think it is the end of the world if we don't see the mask.

jakowenko · 2021-08-27T14:03:57Z

Makes perfect sense. IMO I don't think it is the end of the world if we don't see the mask.

If you don't see the mask right now though it means it's not working.

I think what I'll end up doing is let the user set the size of the image when defining the masks. That way I can resize the mask to be proportional to the resized image from the API. Maybe something like this.

masks:
  - camera: back-door
    mask: 0,0,500,0,500,500,250,800,0,500
    size: 1920x1080
  - camera: back-door
    mask: 100,30,250,0,500,1500,250,800,10,500
    size: 1920x1080
  - camera: garage
    mask: 800,0,1200,0,1200,800,800,800
    size: 1920x1080

But in the meantime if you update frigate.image.height to be the height of your camera images, then you should start seeing the mask hopefully on images on the matches page.

ragenhe · 2021-08-27T14:18:52Z

Perfect. I'll make the change and report back asap.

ragenhe · 2021-08-27T14:24:18Z

With the image size being defined, I see the mask and all appears to be working as intended.

Thanks!!

jakowenko · 2021-08-27T14:39:49Z

Awesome, glad to hear! I'll work on an update to dynamically adjust the mask size. I was thinking of letting the user configure if they want the final image to be saved without the mask too. So in the future it's easier to debug.

Something like this maybe.

masks:
  - camera: back-door
    mask: 0,0,500,0,500,500,250,800,0,500
    size: 1920x1080
    visible: true # show mask on final image
  - camera: back-door
    mask: 100,30,250,0,500,1500,250,800,10,500
    size: 1920x1080
    visible: false # hide mask on final image

Something else to consider too is that the MQTT snapshot image size is configured on the Frigate side. So for the masking to work on the Double Take side for MQTT images then the size would have to be updated in Frigate.

cameras:
  front-door:
    mqtt:
      timestamp: False
      bounding_box: False
      crop: False
      height: 1080

jakowenko · 2021-08-30T03:47:54Z

I pushed a new beta build to dynamically adjust the mask if a size is passed. I also slightly changed the format of the mask definitions. They look like this now. If you want to see the mask on the matches page, then set visible: true, if not they won't be shown on the final image.

masks:
  living-room:
    size: 1920x1080
    mask:
      - 1864,649,1884,494,1687,473,1681,614
      - 590,467,766,283,1146,264,1176,805,809,834
    visible: true

ragenhe · 2021-08-31T18:04:48Z

This is working like a charm @jakowenko!!

jakowenko · 2021-08-31T20:10:29Z

Awesome! I'm glad to hear!!!

I added some intelligence to it as well. For example when processing the snapshot image if there is a mask defined for that camera, then it won't crop it. If no mask is defined than it crops the image using the Frigate API. Similar logic for the Frigate MQTT images, but the cropping for that is defined on the Frigate config side.

# [0.11.0](v0.10.2...v0.11.0) (2021-09-21) ### Bug Fixes * add auth middleware to filters route ([d25c1fd](d25c1fd)) * add auth token to saveURLs function ([#70](#70)) ([167758f](167758f)) * add support for multiple training uploads ([#77](#77)) ([142b3f7](142b3f7)) * better error handling when training fails to prevent stuck loading bar ([07dfd25](07dfd25)) * better handling of new filters ([3fffa6e](3fffa6e)) * button alignment on train toolbar ([c9b38ce](c9b38ce)) * catch errors from recognize/test ([51629d3](51629d3)) * catch get-orientation errors ([d1ca17d](d1ca17d)) * catch if paginated page has no results and return to page 1 ([713be92](713be92)) * catch invalid config on save before writing file ([#94](#94)) ([e4503e8](e4503e8)) * catch when time format is null ([#98](#98)) ([ed872dd](ed872dd)) * check for first number in status to determine if file is trained or untrained ([125f96e](125f96e)) * clean /tmp files after processing and remove all on restarts [#76](#76) ([a5c761f](a5c761f)) * clean mqtt /tmp files after processing [#76](#76) ([b6478ce](b6478ce)) * clear disabled array ([dd84f85](dd84f85)) * define default value for error ([5f27099](5f27099)) * don't reprocess if no detectors configured ([157e7c7](157e7c7)) * loop through compreface plugin results on ui ([83cea24](83cea24)) * **mqtt:** don't publish message if client isn't connected ([5901ebe](5901ebe)) * pass auth token on /cameras route to recognize endpoint ([2e940bf](2e940bf)) * pass upcoming filename to start function to use in response ([32f0bae](32f0bae)) * prevent pagination changes if api is loading ([c767aae](c767aae)) * sanpshot mask check ([56f3ea6](56f3ea6)) * update respond middleware to use res.customStatusCode instead of native res.statusCode ([caf6a89](caf6a89)) * use HTTPSuccess for test controller response ([9bcf8fa](9bcf8fa)) ### Features * ability to adjust thumbnail quality/size and page limit ([e5207fa](e5207fa)) * ability to include base64 encoded string in API results and MQTT messages ([#52](#52)) ([233d56a](233d56a)) * ability to increase auth token expiration ([#78](#78), [#80](#80)) ([d68d39e](d68d39e)) * ability to mask images before processing [#79](#79) ([decb245](decb245)) * ability to reprocess images from the matches page ([#84](#84)) ([809d5f3](809d5f3)) * ability to resize source images with query string ([c2ea600](c2ea600)) * add a delay expressed in seconds between each detection loop ([#83](#83)) ([23dc29e](23dc29e)) * add createdAt, updatedAt to detector detail and tooltip ([#100](#100)) ([03c83f5](03c83f5)) * allow customizing frigate labels ([#95](#95)) ([5eb100a](5eb100a)) * apple-touch-startup-image and theme color ([d8106bb](d8106bb)) * configure detector timeouts ([f654dec](f654dec)) * copy yaml config with defaults ([052ab4b](052ab4b)) * enable or disable frigate mqtt topic snapshot processing ([#83](#83)) ([3bf2bea](3bf2bea)) * filter training results when dropdown is used ([#89](#89)) ([81232aa](81232aa)) * include reasons why image was a miss on matches page ([#90](#90)) ([f5e220b](f5e220b)) * include version on config page with ability to copy ([029bfea](029bfea)) * log level support ([#84](#84)) ([5f91b83](5f91b83)) * pagination and filtering on all matches + refactoring ([af30071](af30071)) * publish errors to mqtt topic ([#52](#52)) ([01a2d6c](01a2d6c)) * redact secrets and keys from logs ([0f3ef02](0f3ef02)) * sockets for live reloading ([50fef76](50fef76)) * support for compreface mask plugin ([#85](#85)) ([7951524](7951524)) * support for multiple frigate urls and topics ([4ead9f7](4ead9f7)) * **train:** pagination ([f749437](f749437))

# [1.0.0](v0.10.2...v1.0.0) (2021-09-21) ### Bug Fixes * add auth middleware to filters route ([d25c1fd](d25c1fd)) * add auth token to saveURLs function ([#70](#70)) ([167758f](167758f)) * add support for multiple training uploads ([#77](#77)) ([142b3f7](142b3f7)) * better error handling when training fails to prevent stuck loading bar ([07dfd25](07dfd25)) * better handling of new filters ([3fffa6e](3fffa6e)) * button alignment on train toolbar ([c9b38ce](c9b38ce)) * catch errors from recognize/test ([51629d3](51629d3)) * catch get-orientation errors ([d1ca17d](d1ca17d)) * catch if paginated page has no results and return to page 1 ([713be92](713be92)) * catch invalid config on save before writing file ([#94](#94)) ([e4503e8](e4503e8)) * catch when time format is null ([#98](#98)) ([ed872dd](ed872dd)) * check for first number in status to determine if file is trained or untrained ([125f96e](125f96e)) * clean /tmp files after processing and remove all on restarts [#76](#76) ([a5c761f](a5c761f)) * clean mqtt /tmp files after processing [#76](#76) ([b6478ce](b6478ce)) * clear disabled array ([dd84f85](dd84f85)) * define default value for error ([5f27099](5f27099)) * don't reprocess if no detectors configured ([157e7c7](157e7c7)) * loop through compreface plugin results on ui ([83cea24](83cea24)) * **mqtt:** don't publish message if client isn't connected ([5901ebe](5901ebe)) * pass auth token on /cameras route to recognize endpoint ([2e940bf](2e940bf)) * pass upcoming filename to start function to use in response ([32f0bae](32f0bae)) * prevent pagination changes if api is loading ([c767aae](c767aae)) * sanpshot mask check ([56f3ea6](56f3ea6)) * update respond middleware to use res.customStatusCode instead of native res.statusCode ([caf6a89](caf6a89)) * use HTTPSuccess for test controller response ([9bcf8fa](9bcf8fa)) ### chore * release ([3dddd40](3dddd40)) ### Features * ability to adjust thumbnail quality/size and page limit ([e5207fa](e5207fa)) * ability to include base64 encoded string in API results and MQTT messages ([#52](#52)) ([233d56a](233d56a)) * ability to increase auth token expiration ([#78](#78), [#80](#80)) ([d68d39e](d68d39e)) * ability to mask images before processing [#79](#79) ([decb245](decb245)) * ability to reprocess images from the matches page ([#84](#84)) ([809d5f3](809d5f3)) * ability to resize source images with query string ([c2ea600](c2ea600)) * add a delay expressed in seconds between each detection loop ([#83](#83)) ([23dc29e](23dc29e)) * add createdAt, updatedAt to detector detail and tooltip ([#100](#100)) ([03c83f5](03c83f5)) * allow customizing frigate labels ([#95](#95)) ([5eb100a](5eb100a)) * apple-touch-startup-image and theme color ([d8106bb](d8106bb)) * configure detector timeouts ([f654dec](f654dec)) * copy yaml config with defaults ([052ab4b](052ab4b)) * enable or disable frigate mqtt topic snapshot processing ([#83](#83)) ([3bf2bea](3bf2bea)) * filter training results when dropdown is used ([#89](#89)) ([81232aa](81232aa)) * include reasons why image was a miss on matches page ([#90](#90)) ([f5e220b](f5e220b)) * include version on config page with ability to copy ([029bfea](029bfea)) * log level support ([#84](#84)) ([5f91b83](5f91b83)) * pagination and filtering on all matches + refactoring ([af30071](af30071)) * publish errors to mqtt topic ([#52](#52)) ([01a2d6c](01a2d6c)) * redact secrets and keys from logs ([0f3ef02](0f3ef02)) * sockets for live reloading ([50fef76](50fef76)) * support for compreface mask plugin ([#85](#85)) ([7951524](7951524)) * support for multiple frigate urls and topics ([4ead9f7](4ead9f7)) * **train:** pagination ([f749437](f749437)) ### BREAKING CHANGES * v1.0.0

# [1.0.0](v0.10.2...v1.0.0) (2021-09-21) ### Bug Fixes * add auth middleware to filters route ([d25c1fd](d25c1fd)) * add auth token to saveURLs function ([#70](#70)) ([167758f](167758f)) * add support for multiple training uploads ([#77](#77)) ([142b3f7](142b3f7)) * better error handling when training fails to prevent stuck loading bar ([07dfd25](07dfd25)) * better handling of new filters ([3fffa6e](3fffa6e)) * button alignment on train toolbar ([c9b38ce](c9b38ce)) * catch errors from recognize/test ([51629d3](51629d3)) * catch get-orientation errors ([d1ca17d](d1ca17d)) * catch if paginated page has no results and return to page 1 ([713be92](713be92)) * catch invalid config on save before writing file ([#94](#94)) ([e4503e8](e4503e8)) * catch when time format is null ([#98](#98)) ([ed872dd](ed872dd)) * check for first number in status to determine if file is trained or untrained ([125f96e](125f96e)) * clean /tmp files after processing and remove all on restarts [#76](#76) ([a5c761f](a5c761f)) * clean mqtt /tmp files after processing [#76](#76) ([b6478ce](b6478ce)) * clear disabled array ([dd84f85](dd84f85)) * define default value for error ([5f27099](5f27099)) * don't reprocess if no detectors configured ([157e7c7](157e7c7)) * loop through compreface plugin results on ui ([83cea24](83cea24)) * **mqtt:** don't publish message if client isn't connected ([5901ebe](5901ebe)) * pass auth token on /cameras route to recognize endpoint ([2e940bf](2e940bf)) * pass upcoming filename to start function to use in response ([32f0bae](32f0bae)) * prevent pagination changes if api is loading ([c767aae](c767aae)) * sanpshot mask check ([56f3ea6](56f3ea6)) * update respond middleware to use res.customStatusCode instead of native res.statusCode ([caf6a89](caf6a89)) * use HTTPSuccess for test controller response ([9bcf8fa](9bcf8fa)) ### chore * release ([824db79](824db79)) ### Features * ability to adjust thumbnail quality/size and page limit ([e5207fa](e5207fa)) * ability to include base64 encoded string in API results and MQTT messages ([#52](#52)) ([233d56a](233d56a)) * ability to increase auth token expiration ([#78](#78), [#80](#80)) ([d68d39e](d68d39e)) * ability to mask images before processing [#79](#79) ([decb245](decb245)) * ability to reprocess images from the matches page ([#84](#84)) ([809d5f3](809d5f3)) * ability to resize source images with query string ([c2ea600](c2ea600)) * add a delay expressed in seconds between each detection loop ([#83](#83)) ([23dc29e](23dc29e)) * add createdAt, updatedAt to detector detail and tooltip ([#100](#100)) ([03c83f5](03c83f5)) * allow customizing frigate labels ([#95](#95)) ([5eb100a](5eb100a)) * apple-touch-startup-image and theme color ([d8106bb](d8106bb)) * configure detector timeouts ([f654dec](f654dec)) * copy yaml config with defaults ([052ab4b](052ab4b)) * enable or disable frigate mqtt topic snapshot processing ([#83](#83)) ([3bf2bea](3bf2bea)) * filter training results when dropdown is used ([#89](#89)) ([81232aa](81232aa)) * include reasons why image was a miss on matches page ([#90](#90)) ([f5e220b](f5e220b)) * include version on config page with ability to copy ([029bfea](029bfea)) * log level support ([#84](#84)) ([5f91b83](5f91b83)) * pagination and filtering on all matches + refactoring ([af30071](af30071)) * publish errors to mqtt topic ([#52](#52)) ([01a2d6c](01a2d6c)) * redact secrets and keys from logs ([0f3ef02](0f3ef02)) * sockets for live reloading ([50fef76](50fef76)) * support for compreface mask plugin ([#85](#85)) ([7951524](7951524)) * support for multiple frigate urls and topics ([4ead9f7](4ead9f7)) * **train:** pagination ([f749437](f749437)) ### BREAKING CHANGES * v1.0.0

jakowenko · 2021-09-21T18:57:31Z

Closing as this was merged into release 1.0.0

jakowenko added the enhancement New feature or request label Aug 26, 2021

jakowenko added a commit that referenced this issue Aug 27, 2021

feat: ability to mask images before processing #79

decb245

ozett mentioned this issue Sep 7, 2021

False-Positives: How do i limit? Zones? #91

Closed

jakowenko closed this as completed Sep 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Frigate mask definitions #79

Use Frigate mask definitions #79

ragenhe commented Aug 25, 2021

phammichael commented Aug 26, 2021

jakowenko commented Aug 26, 2021

ragenhe commented Aug 26, 2021

jakowenko commented Aug 26, 2021

ragenhe commented Aug 26, 2021

phammichael commented Aug 26, 2021

jakowenko commented Aug 26, 2021

jakowenko commented Aug 27, 2021

ragenhe commented Aug 27, 2021

ragenhe commented Aug 27, 2021

jakowenko commented Aug 27, 2021

ragenhe commented Aug 27, 2021

jakowenko commented Aug 27, 2021 •

edited

Loading

ragenhe commented Aug 27, 2021

ragenhe commented Aug 27, 2021

jakowenko commented Aug 27, 2021 •

edited

Loading

jakowenko commented Aug 30, 2021 •

edited

Loading

ragenhe commented Aug 31, 2021

jakowenko commented Aug 31, 2021 •

edited

Loading

jakowenko commented Sep 21, 2021

Use Frigate mask definitions #79

Use Frigate mask definitions #79

Comments

ragenhe commented Aug 25, 2021

phammichael commented Aug 26, 2021

jakowenko commented Aug 26, 2021

ragenhe commented Aug 26, 2021

jakowenko commented Aug 26, 2021

ragenhe commented Aug 26, 2021

phammichael commented Aug 26, 2021

jakowenko commented Aug 26, 2021

jakowenko commented Aug 27, 2021

ragenhe commented Aug 27, 2021

ragenhe commented Aug 27, 2021

jakowenko commented Aug 27, 2021

ragenhe commented Aug 27, 2021

jakowenko commented Aug 27, 2021 • edited Loading

ragenhe commented Aug 27, 2021

ragenhe commented Aug 27, 2021

jakowenko commented Aug 27, 2021 • edited Loading

jakowenko commented Aug 30, 2021 • edited Loading

ragenhe commented Aug 31, 2021

jakowenko commented Aug 31, 2021 • edited Loading

jakowenko commented Sep 21, 2021

jakowenko commented Aug 27, 2021 •

edited

Loading

jakowenko commented Aug 27, 2021 •

edited

Loading

jakowenko commented Aug 30, 2021 •

edited

Loading

jakowenko commented Aug 31, 2021 •

edited

Loading