[RFC] Trigger Clips and Snapshots via API

[herostrat]

Idea:

Allow triggering of clips and snapshots via external command.

Implementation:

Implement API endpoints for:

• Webinterface
  • In camera view two buttons
• HTTP
  • /api/<camera_name>/create-snapshot
  • /api/<camera_name>/create-clip
• MQTT
  • frigate/<camera_name>/snapshots/create
  • frigate/<camera_name>/clips/create

The configuration of the created object is according to the user config and therefore the same as triggered via internal detection.

Background/Usecases:

• motion/PIR/light barrier sensors in home automation as trigger condition
• timer based trigger condition
• ...

@blakeblackshear is this acceptable or according to the projects vision?
I am interested in implementing this myself, so it is no FR

@ALL Additional ideas and suggestions?

---

Relevant Issues/Discussions:
#1040
#926
#345
#1008

[blakeblackshear]

I am not completely convinced a few other enhancements wouldn't obviate the need for this feature. What if the 24/7 recordings could be set to only retain segments that had motion? Also, what if 24/7 recordings could be turned on and off like detection, clips, and snapshots? I don't see any use cases in the linked tickets that wouldn't be covered by these two enhancements.

If this is implemented, my gut says this should be done by manually saving a frame as a snapshot and putting an event into the event queue for processing. I think it will need to have a predetermined length, and the properties on the event table will need to be reconciled to accommodate a manual recording. It should fit fairly easily into the existing event handling and clip creation.

[herostrat]

I am not sure I understand correctly and/or made it clear enough what my idea is.

Basically I want to use the event queue, but add additional triggers for it.

My understanding is that currently the only way to add something to.the event queue is when frigate has detected something.

Please correct me if I am wrong, I am most certaintly not too familiar with the codebase.

How would you use external sensors to generate clips oe snapshots and save them?

[blakeblackshear]

I understand the idea and agree with the implementation of using the event queue. That's basically what I said.

I just don't see the need to use external sensors to trigger clips or snapshots if Frigate is working properly. I would rather ensure Frigate has the features to capture anything of interest than rely on external systems to turn things on and off.

[herostrat]

Ah I see, sorry for the confusion. English is not my mother tongue.

I agree for the most part.
In my mind there are a few usecases where frigate is working and detecting properly, but also could use this feature.
Sometimes there is just nothing to detect in the video feed but I want a clip nonetheless.

Thanks for your feedback!

[Whytey]

Additional use case - video doorbell. Doorbell is pressed so trigger a clip/recording. Frigate might not have detected motion (maybe the local kids are being tricky and hiding themselves??) but you would like to have a clip so you can see what was going on in the background??

[blakeblackshear]

If they were hiding themselves and frigate doesn't detect any motion, wouldn't you just be recording a static image?

[Whytey]

I was thinking more along the lines of them being in the distance but too small to be within the detection threshold. Corner cases admittedly.

[blakeblackshear]

And those far away objects are related to your doorbell ringing? I'm trying to understand if this is a hypothetical scenario or if you have a use case I haven't considered.

[Whytey]

Purely hypothetical.

I like the idea of externally triggering Frigate... Though purely due to having had poor experiences of false positives with other NVR solutions.

[stephengolub]

I like the idea of triggering a recording with "custom events" and was actually looking into the code to see how complicated it would be. I ran out of free time a while back, so maybe I'll revisit it.

Here's where I think just relying on motion/object detection could fail: Say you only have person detection set up, and a delivery driver comes up with a load of packages or big jacket or something and the model just doesn't quite recognize them. Then you miss the recording.

Having the "custom event" (in this case a "ding dong" or "ring" event) that you can send a payload to Frigate and say:

{
  "camera": "front_door",
  "event": "ring"
}

I can also imagine triggering perimeter recording if my door bell rings, or almost any other type of thing.

As an MVP, I'd just accept event and camera and use the default clip recordings. Maybe in the future add the ability to specify length, etc.

[blakeblackshear]

If 24/7 recordings had the ability to only retain segments where motion was detected, wouldn't you cover the edge cases where the model didn't detect for some reason?

Also seems like a switch to turn off 24/7 recordings would address the use case of turning on recording based on an event.

[stephengolub]

My thinking is that it would tag a point in time by leveraging the existing concept of "events".

By adding a custom "ding" event or whatever, I will see those separated out in the UI and database and could even have a "last_ding" camera in home assistant.

Another use case I would use was "garage" event. So if someone opens or closes my garage, I would run a snapshot of the driveway camera as well and have the specific point in time tagged.

[blakeblackshear]

Those seem like worthwhile use cases. This should fit reasonably well into the the existing architecture as I mentioned above.

[tam-wh]

Ideally, I would use object detection in frigate but I don't have a coral and I rather not max up my CPU for object detection. Since I have a PIR sensor in home assistant, it would be great to use that to trigger an event in frigate and record a clip. (E.g. If someone enters my room when I'm away)

Ability to control 24/7 recording with Mqtt/hass is important as well for certain use case (e.g. using hass automation to stop recording when I'm home and start recording when I'm away)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[blakeblackshear]

@herostrat please don't spend any time on this. Upcoming changes will make manually triggered events much easier to implement, so any work here would be a waste of time.

[vhbramos]

I have a few use cases that could leverage this feature:

1. Audio Detection: My Amcrest cameras can detect audio and it would be great to manually trigger a clip recording in Frigate. There is no motion, but the camera audio can provide important insights about ongoing events.
   Example: Detection of fire/smoke alarms or any other audio alarms. ( Wize cameras have a specific event to alert users of Fire/Smoke alarms )

2. Tripwire: This can be a Frigate core feature in the future, but in the meantime I could use an external sensor ( some cameras also have it embedded ) to trigger a recording and a specific alert if motion is detected from a specific area to a specific area. This would be an additional alert, since I would still want Frigate to monitor a larger area for objects ( like people and cars )

3. Other AI detection: I am looking to deploy a couple of cameras with more advanced AI capabilities, like detecting parked cars and objects left in specific areas. I can get motion events for those situations in Frigate today but those extra labels would help me get better notifications

[zaPure]

I agree that this would be a useful feature to trigger recording from MQTT events, e.g. my alarm or electrical fence triggers or garden beams go off. I would like the camera to start recording so that I can get a sense of what triggered it. As I have my cameras are set up only to record on people detection.

[oramirite]

I'd like to chine in on the need for this as well. I regards to Frigate working properly obfuscating the need for this feaure: I feel it's anything but. Frigate itself, when working correctly, can benefit from it's own detection to trigger events manually.

I have an overhead street camera that's at a perfect angle for person detection. Frigate detects people on my stoop and even near my mailbox perfectly. My doorbell camera points out at a busy street however, and unless I can get some kind of z-depth detection it's impossible to mask things out in a way that will only detect people on my stoop. The overhead camera is perfect for drawing zones. But the doorbell camera is the only one that can capture faces. Plus, it captures audio. There is simply no way for that camera on it's own to properly filter out noise though. It really needs the benefit of other sensors to retain events properly.

It would be great to rely on the overhead camera's detection, and be able to roll my own logic for capturing the same clips on my doorbell camera. This way, the media library of my Doorbell would only ever contain events where people were accurately detected on my stoop.

If I enable detection right now, I'm sure the doorbell will capture that same clip of the person on my stoop... along with at least 200+ other people that day who have simply passed by. The doorbell camera angle is just REALLY bad for actual detection, but works perfect for faces and general viewing when the clips are captured. The difference is less than a pixel from the doorbell angle whereas the overhead angle makes it simple.

While I appreciate that Frigate wants to cover and and all use-cases through detection, I feel like there's always going to be edge cases where complimenting the detection with an additional sensor will work perfectly.

[blakeblackshear]

You should create a zone for your stoop and add that zone to required zones for your doorbell camera. Then you will only get events where the person stepped onto your stoop.

Either way, this feature is coming. No need to continue to advocate for it.

[oramirite]

That's great news @blakeblackshear, thanks so much!! And thanks for such a quick reply before.

I can't see the ground due to this doorbell camera's angle, so there's nowhere to draw that zone you mentioned. The overhead camera I mentioned works flawlessly with zones so using that as a trigger will be perfect. It's no fault of Frigate at all - my house is elevated from the street, so the doorbell camera only sees humans from the waist up. Detection works great... it's just not possible to differentiate someone walking past my house from them being at the door.

Excited for this addition. Thank you so much for creating what might be the best NVR solution out there!!

[Notrial]

Has this feature been implemented in the recent version updates?

[chrisella]

If I understand, this issue solved the case I'm after which is my 433Mhz doorbell is picked by my my 433 receiver on ESPHome, which in turn runs an automation in HomeAssistant, I then want that automation to tell Frigate to save a recording (/create an event) of the last X seconds + next Y seconds and save that... am I right that this helps fill that last piece of the puzzle ? If so what is the current state of this, if not yet ready then is there an alternative way I could achieve this people are using ?

[NickM-27]

It's in draft PR, will he added in a future version. And yes it should solve what you're describing.

[chrisella]

Great, ok thanks I'll just be patient and wait then.

[bagobones]

Playing around with double-take the ability to force a snapshot / event when someone presses the door bell and is likely looking at it would be very nice for face detection / custom events based on specific people.

I am currently getting the walk up to the doorbell but since the event starts when the person is far away, face detection is very poor.

[raintonr]

I am currently getting the walk up to the doorbell but since the event starts when the person is far away, face detection is very poor.

Sounds like the detection target you have configured is 'person' but you really want a target of 'face'. Then you would also like the target 'face' to be further classified to the actual individual.

Seems like a new feature request and nothing to do with triggering via API.

[scaytrase]

Seems like a new feature request and nothing to do with triggering via API.

as far as I understand @bagobones he doesn't asks for built-in face recognition here, but for a way to make a clip\snapshot with trigger API for doing so (maybe in existing external software we don't know). so looks like relevant use case - trigger clip\snapshot by doorbell press.

[bagobones]

All I am asking for is to trigger an "event" and then integrations like double-take can add sub labels for faces

[roger-]

Adding a related request: ability to trigger a full object detection over the entire frame (perhaps respecting configured masks).

[blakeblackshear]

ability to trigger a full object detection over the entire frame

This might seem like it is related, but it's very different and it wouldn't make sense to do this at the same time as this feature as it would be implemented in completely different parts of the code base. I would open a different request.

[NickM-27]

ability to trigger a full object detection over the entire frame

This might seem like it is related, but it's very different and it wouldn't make sense to do this at the same time as this feature as it would be implemented in completely different parts of the code base. I would open a different request.

That's my bad, misunderstood the other issue request as a duplicate

[Guisin]

Hello, is there any forecast for this implementation?

[chrislauyp]

@blakeblackshear @herostrat @stephengolub
Hi there,
Any update in the customer external event feature?
Thanks

[blakeblackshear]

It might be a part of 0.13

[hawkeye217]

It might be a part of 0.13

Would love to see this as part of 0.13, and I'd be happy to help test Nick's PR (or whatever implementation is decided upon) when committed. Dahua cameras do pretty decent AI detection and with the Dahua integration, it's pretty trivial to set up some Home Assistant automations to grab the event stream (which includes event start/stop and bounding box attributes) and then pass it on to create manual events in Frigate.

[bagobones]

Would be nice if the API / feature had several ways to operate

1. Momentary event.. Record for a default time.. Maybe just depend on the existing pre_capture and post_capture event values or a new value custom_capture period. Very simple webhook to record a short clip.
2. Momentary event with the recording time sent in the event. IE record for 30s is sent with the trigger.
3. Two event triggers with an event ID of some kind? Requires sending both a start and stop event, should probably have a custom_max_caputure value to stop if the stop event is never seen.

[NickM-27]

@bagobones all of that is already supported in this implementation

[MrSiO]

@bagobones all of that is already supported in this implementation

Hi Nick.

Can you identify in the documentation where this is explained?
Eighter by MQTT or HTTP API, I cannot find how to manually trigger start/stop recording on a specified camera.

(I'm running latest Beta)

Thx

[NickM-27]

Are you looking at beta or normal docs? https://deploy-preview-6262--frigate-docs.netlify.app/integrations/api#post-apieventscamera_namelabelcreate

[bagobones]

Do be clear the fictionally of the api does NOT trigger recording it triggers events.

As per the big warning in the beta doc if you are using motion recording you can end up with empty / deleted events.

"Recording retention config still applies to manual events, if frigate is configured with mode: motion then the manual event will only keep recording segments when motion occurred."

[MrSiO]

Are you looking at beta or normal docs? https://deploy-preview-6262--frigate-docs.netlify.app/integrations/api#post-apieventscamera_namelabelcreate

I was looking at Normal docs.
Didn't realize there was a separate doc for Beta. It's bookmarked now.

Thx! Will implement asap!

[kirilnedev]

Maybe I missed that it was implemented, but one more use case... Currently there are plenty of inexpensive camera's which have surprisingly good Human detection + most of them have some Output alarm... this may be quite useful as it may offload the Human detection to a camera consuming 1W which already has the required hardware...

[oramirite]

Maybe I missed that it was implemented, but one more use case... Currently there are plenty of inexpensive camera's which have surprisingly good Human detection + most of them have some Output alarm... this may be quite useful as it may offload the Human detection to a camera consuming 1W which already has the required hardware...

This was added :)

#3184 (comment)

[kirilnedev]

Thanks a lot for the link, @oramirite ! :) Great to hear