Object detection in surveillance video (2020)

Hey Python people,

There was a thread about object detection quite a while back, but I haven't seen a mention of it since.

My situation is that, like many Indigo users, I have a Unifi Protect video setup and I'm not at all satisfied with its motion notifications. I experimented with Security Spy for person detection and that seemed to work OK but seems inelegant to me.

I started playing around with OpenCV, and it only took a couple evenings to get a useable Python service that watches an RTSP feed from Protect, runs it through YOLOv3-tiny, and logs when recognized objects are seen. It's taking around 75mS/frame without GPU on my laptop, and i'm recognizing at 1 frame per second. It wouldn't be unreasonable to run 5 cameras concurrently without further optimization, and I've ordered a Coral accelerator (https://coral.ai/products/accelerator/) to see how much impact that makes.

My question is: Is anybody else messing around with this sort of thing? Does anyone see value in making a new person-detector (cat-detector / toaster-detector / ...) plugin, or extending the existing security camera plugin?

I'm also looking for suggestions on a more surveillance-specific neural net that's reasonably performant. I'm not going to invest in training my own, but I'm guessing the Coco dataset isn't ideal for this application.

Any other thoughts, comments, ideas?

(FWIW, the installation wasn't that bad. I only needed NumPy and OpenCV-Python, and I might be able to factor out the NumPy requirement if I had to. The Yolo stuff is public domain, so I could bundle it with a hypothetical plugin)

Thanks

Yep. I run a bunch of cameras thru Blue Iris, which works great. Their motion detection is "ok", but it easily gets tripped up on moving leaves, branches casting shadows as the sun moves across the sky, branches casting shadows when the wind blows, etc. So I needed a way to detect actual issues vs shadows and foliage.

So when Yolo first came out, what I did was throw together a simple python script that, when triggered, goes and asks Blue Iris for a series of recent frames, and then downscales them and runs them thru Yolo, identifying if there's anything of interest. I only check for a subset of the classified items items after having it discover "sheep" in my driveway. (Although, you never know...). I also had it double check (x,y) of the found target against a polygon of interest within the image, so that, for example, I could have it just check for humans approaching the front door, and ignore them if they are on the sidewalk.

I set the script to run 5 seconds after Blue Iris sends Indigo a motion alert (accomplished thru the Blue Iris Plugin), so that it could look back over the previous 5 seconds and find things. If it did, then it grabs a few frames, throws them together as an animated gif, and sends it via Pushover to my phone, so I get an alert with animated gif preview of what happened a few seconds ago. Here is a primitive early version with a terrible camera that was stuck on the wall looking thru the garage window.

Your approach is much better, being real-time, but at the time I did this, the performance just wasn't there. Plus, I didn't want to monitor the rtsp streams in real time, unless it was on the same machine as Blue Iris (too much network traffic). Nowadays, I have a beefier machine, and I'm sure the algorithms have improved a lot. Maybe the classifiers are so good nowadays that it could discern a delivery guy from the pizza person. FWIW, Blue Iris also has added some sort of AI as well, but it's cloud-based and I'm sure far more limited than what we could do ourselves, so I haven't even looked at it.

One challenge I had was that all the good code was python3, so I had to have a separate python virtualenv set up so that it didn't conflict with Indigo's stuff. I run the script from the command line in a trigger (which also lets me pass parameters, like which camera, as command line arguments). And then there's the models, and how to train them. I think I tried a bunch and ended up with mobilenet caffe model, but I can't recall. Whatever it is, it's a starting point for cars and people. The stuff I threw together is not production code; it's just toothpicks and lint, and barely works, so it really wouldn't be useful as a basis for a plugin, except as a proof of concept. Let me know if it's of interest (but sounds like you've done the hard part already).

I need the real-time feature to implement the cat watcher: alert when they get outside, and make sure they roam around in the right places, and don't eat any plants!

You guys need to get on this!

Search the forums for DeepQuest DeepStack or DeepState AI.

Forum member GlennNZ made an awesome plugin for BlueIris, then he figured out how to run a dockerized, home-served, object recognizing AI agent with it and Indigo.

Hi all

Thanks for DeepQuest AI feedback!

Info is here:
https://github.com/Ghawken/DeepQuestAI

And some more on the the deepquest thread in BlueIris - my system continues to work well (so far 649 gig of image data processed...)

Should finalise and release in plugin store - but other events...etc..etc...

Glenn


Sent from my iPad using Tapatalk