As computer vision and machine learning get more powerful, businesses are using live video streaming analytics to add value to their operations — from assessing flaws on production lines, to conducting dynamic crowd analysis, or more prosaically, digital doorbell firms sending alerts for package delivery.

Analysing video footage in real time is challenging, however; not least because it requires keeping up with video frame rates of up to 60 frames per second. Compute is not cheap – particularly with streams being run through neural networks for analysis – and businesses can rack up costly bills fast.

A new tool open sourced by Microsoft, Project Rocket, aims to help tackle that problem by rethinking how live video streams are analysed.

microsoft project rocketProject Rocket: A Video Streaming Analytics Filter

Project Rocket, open sourced this week by Microsoft, aims to tackle the problem by helping users reserve resource-intensive operations for when they’re really necessary. While computer vision and deep neural networks (DNNs) are becoming hugely more powerful, they still need the individual frames to work on: a costly, time-consuming bottleneck.

Project Rocket lets users build a video pipeline that includes a cascade of DNNs in which a decoded frame is first passed through a relatively inexpensive “light” DNN like ResNet-18 or Tiny YOLO with a “heavy” DNN such as ResNet-152 or YOLOv3 invoked only when required for more robust analysis. The platform can be plugged into any TensorFlow or Darknet DNN model.

Users can also augment the above pipeline with a simpler motion filter based on OpenCV background subtraction, as shown in the figure below.

diagram

Cascaded pipelines let users filter out frames with limited relevant information and be “judicious” about invoking resource-intensive operations, then ship the outputs of the video analytics, such as the number of relevant objects in an object-counting application, to a database for after-the-fact review.

In a blog by principle researcher Ganesh Ananthanarayanan and team, Microsot’s Ananthanarayanan said the platform has been tested on smart city applications, including via a partnership with the city of Bellevue, Washington.

This was used in the city for aggregate car and bicycle counts provided by a system built on the framework that let city policy makers accurately assess the value of adding a bike lane to its downtown area.

The Rocket platform is written in .NET Core, which is compatible on Windows, as well as Linux, comes with instructions to create Docker containers, and –nof course – comes with code to help invoke customised machine learning models in Microsoft’s Azure cloud. For enterprises making use of video analytics and who haven’t coded together something passingly similar already, the toolkit looks well worth a look. Project Rocket’s code is here.

See also: Microsoft Open Sources Homomorphic Encryption Library “SEAL”