Motion CCTV on Linux

Motion is by far the oldest, but probably still the best CCTV package for Linux. I started using it around 2004 and have been using it ever since. While there are a couple of newer and shinier packages like iSpy, they are not a like for like match and they are not on par with motion in terms of flexibility. This is quite important if you are trying to do something weird an wonderful with your camera observations.

Use Cases

I have yet to find a piece of software that will work well for my use cases. While they are a bit off the beaten path, they are not that strange. So not having any of them supported out of the box is quite annoying.

  • Pictures from the wilderness. Reliably transfer pictures to a central site over an unreliable connection. In my case that is a VPN over a GPRS/EDGE mobile "broadband". Quotes needed by the way. Queue pictures as needed.
  • Trigger an alert on prolonged inactivity
  • Simple IPC to perform an action like "turn on sprinklers" based on a motion trigger.

Queued picture transfer

When Do We Need Queueing

99% of IP cameras out there will fail in a slow uplink scenario. In fact, this is one place where the inability and incompetence are standardised. All standards for image transfer in ONVIF are based on synchronous streaming. The standard profile specifies H264 and rtsp. Even in the case of TCP transport this is guaranteed to break if you have, let's say, 160Kbit. Trying to fit the standard 3Mbit base profile stream generated by a camera always results in the DVR dropping the connection. The optional MJPEG profile is not any better as you will capture synchronously, so your capture rate is limited by the uplink. If an event generates 40 images, but you had the bandwidth to transfer 3 between the start and the finish, you will record only three.

Last, but not least, if your connection is metered, you will finish off your allowance in a day or two. The answer here is supposed to be "perform motion detection at the camera". If we put aside the picture loss due to bandwidth constraints for a second, this can be done. In theory. In practice, the current level of standardisation in the detection of motion at the camera is somewhee around "my summer boyscout project". The relevant ONVIF S section contains nothing. You have to guess based on circulated WSDL files and they have partially nothing in them as well. What is clear though is that image push upon detection is not standardised. Standard provides only a trigger callback. So, you get a trigger and you now have to go and poll the camera. If you try this over a VPN to a remote site with GPRS/EDGE connection the event will be half the way through by the time you are started capturing. While most cameras have an upload facility as an option it expects that the network isnfast enough. So that does not work either.

It looks like we are sort of (excuse my engineering language), screwed. Enter motion + Linux. They succeed where the off the shelf fails and the standard has gone off the deep end. As a matter of fact we can implement most of it in shell scripts. Using python (as I do now) is unnecessary - I did it for fun and to refresh my python before switching projects.

Implementing Picture Queue (with drop)

We do not really need a proper Queue as such. It would have been nice to have it, but we can in fact survive without it by using tmpfs filesystem to store pictures while in-flight. So all we need to do on a modern debian box running Linux is to add this line to /etc/fstab:
none            /var/tmp       tmpfs    defaults        0       0

as well as a safety net (if it gets full) to /etc/cron.d/motion
4,24,44 * * * *       motion    find /var/tmp/motion -type f -mmin +90 -exec rm {} \; >/dev/null 2>&1 
33 * * * *       motion    find /var/tmp/motion/* -type d -mmin +120 -exec rmdir {} \; >/dev/null 2>&1 

The actual parameters of the safety net such as run frequency, file and directory age before deletion can be adjusted depending on the frame capture rate and individual frame size. Individual frames read by HD cameras can be quite large (in the 300KBytes range) necessitating a more frequent pruning action. Next we create a a simple start/stop scripts for our file moving routine. Here is our control script
#!/usr/bin/env python

"""Root Mover object for files generated by motion
and other similar utilities
"""

import socket
import sys
import optparse

class Activate(object):
    """Move files from A to B"""
    def __init__(self, control, message):
        self._control = control
        self._message = message
        self._socket = None

    def send(self):
        """Run the actual mover"""
        _socket = socket.socket(
            socket.AF_UNIX, socket.SOCK_DGRAM
        )
        _socket.connect(self._control)
        _socket.setblocking(0)
        _socket.send(self._message)

def main():
    '''Main routine: create an instance of the Activate and run it.'''

    usage = "usage: %prog [options]"
    oparser = optparse.OptionParser(usage = '\n'.join(usage))
    oparser.add_option(
        '-m', '--message',
        help = 'message',
        dest = 'message'
    )
    oparser.add_option(
        '-c', '--control',
        help = 'control socket',
        dest = 'control'
    )
    options, arguments = oparser.parse_args()

    if (
        options.message == None or 
        options.control == None
        ):    
        sys.exit(1)

    activate = Activate(options.control, options.message)
    
    activate.send()
    
if __name__ == '__main__':
    sys.exit(main())

We will call it start.py and put it into /usr/local/bin

We add it to the motion.conf file:
on_event_start /usr/local/bin/start.py -m START -c /var/run/motion/move-control
on_event_end /usr/local/bin/start.py -m STOP -c /var/run/motion/move-control

Next we need to create an actual "mover" - something that will take files out of /var/lib/motion and and move them to a location of our choosing (a remote NAS) while responding to START and STOP signals. This one is slightly more complicated:
#!/usr/bin/env python

"""Root Mover object for files generated by motion
and other similar utilities
"""

import socket
import sys
import os
import optparse
import select
import re
import time
from functools import partial

COPYING = 1
SLEEPING = 2

class Mover(object):
    """Move files from A to B"""
    def __init__(self, source, dest, control):
        self._source = source
        self._dest = dest
        self._control = control
        self._socket = None
        self._state = SLEEPING

    def run(self):
        """Run the actual mover"""
        try:
            os.unlink(self._control)
        except (OSError, IOError):
            pass
        self._socket = socket.socket(
            socket.AF_UNIX, socket.SOCK_DGRAM
        )
        self._socket.bind(self._control)
        self._socket.setblocking(0)

        regexp = re.compile("START")

        while True:
            inputs = [ self._socket ]
            outputs = []
            if self._state == SLEEPING:
                select.select(inputs, outputs, inputs)
            else:
                time.sleep(1)
            try: 
                data = self._socket.recv(2048)
                if regexp.match(data):
                    self._state = COPYING
                else:
                    self._state = SLEEPING
            except socket.error:
                pass

            os.chdir(self._source)
            for root, dirs, files in os.walk('.'):
                for dirname in dirs:
                    try:
                        os.mkdir(os.path.join(self._dest,root,dirname))
                    except (OSError, IOError):
                        print >> sys.stderr, "Cannot create target dir" + os.path.join(
                                self._dest, root, dirname
                            )
                for filename in files:
                    try:
                        target = open(os.path.join(
                            self._dest, root, filename
                            ),
                            "w"
                        )
                    except (OSError, IOError):
                        print >> sys.stderr, "{0} {1}".format(
                                os.strerror(ex.errno),
                                os.path.join(
                                    self._dest, root, filename
                                )
                            )
                    if time.time() - os.stat(
                                os.path.join(self._source, root, filename)
                                ).st_mtime > 1:
                        with open(os.path.join(self._source, root, filename), 'rb') as ofo:
                            for chunk in iter(partial(ofo.read, 32768), ''):
                                target.write(chunk)
                        try:
                            os.unlink(os.path.join(self._source, root, filename))
                        except (OSError, IOError):
                            print >> sys.stderr, "{0} {1}".format(
                                    os.strerror(ex.errno),
                                    os.path.join(
                                        self._source, root, filename
                                    )
                                )
                for dirname in dirs:
                    try:
                        os.rmdir(os.path.join(self._source, root, dirname))
                    except (OSError, IOError) as ex:
                        print >> sys.stderr, "{0} {1}".format(
                                os.strerror(ex.errno),
                                os.path.join(
                                    self._source, root, dirname
                                )
                            )

def main():
    '''Main routine: create a python Mover and run it.'''

    usage = "usage: %prog [options]"
    oparser = optparse.OptionParser(usage = '\n'.join(usage))
    oparser.add_option(
        '-s', '--source',
        help = 'source directory',
        dest = 'source'
    )
    oparser.add_option(
        '-d', '--dest',
        help = 'destination directory',
        dest = 'dest'
    )
    oparser.add_option(
        '-c', '--control',
        help = 'control socket',
        dest = 'control'
    )
    options, arguments = oparser.parse_args()

    if (
        options.source == None or 
        options.dest == None or 
        options.control == None
        ):    
        sys.exit(1)

    Run = Mover(options.source, options.dest, options.control)
    
    Run.run()
    

if __name__ == '__main__':
    sys.exit(main())

So... what does it do? It creates a listening control socket, waits for a START sent on it, then copies files until all files are copied, repeatedly reading the source directory until it gets a STOP. So all you need to do is to point it to a remote NAS and you are away. I do it via autofs, but is just spit and polish - the remote NAS can be permanently mounted. The only caveat for VPNs ove unreliable links and Linux is that you MUST have persistent client IP addresses. Linux really hates (and usually cannot recover cleanly) if the connection is reestablished with a different IP while a remote filesystem is mounted via NFS or SMB.

Is it difficult?

Hell no, in fact, if I was doing it for real for commercial needs I would have implemented a proper queueing system, not hacked one out of tmpfs, python and a remote NAS after dinner. It is trivial to do that on modern IP cameras - most of them have 256 or 512MB RAM as they use an off the shelf SOC. They use only a fraction of that and can easily enqueue several minutes of video for an event and transfer it in a manner which is suitable for a remote site on a narrowband link. It may be easy, it is definitely a feature which a lot of users will find useful, but you will not see it in a commercial camera. Some DVRs can do it, but even there it is implemented in a manner which will make any security engineer or software developer's hair rise.

Alert on A Prolonged Inactivity

Motion itself does not have this functionality. It is trivial, however to implement it in python. It is just a simple extension to our move script - every time we get a STOP we set a timer, every time we get a START we reset a timer. If the timer ever fires we have our event. ~ 7 lines of trivial python (I am not going to even quote it - just have a look at the SIGALRM example in the python documentation).

Is the feature needed? Ask that question yourselves if you have CCTV installed at one of your elderly relatives. Is it available in off the shelf equipment. Hell no.

Simple IPC to Perform Actions

Well, what can be simpler than running a script. It definitely beats having to run a fully blown SOAP stack and wait for a web callback as prescribed by ONVIF. While both of them are nowhere near what would be really needed in a proper large scale system (you really need to integrate a proper publish/subscribe for this to scale), at the very least scripting is something which is easy to do. Definitely easier than SOAP.

-- AntonIvanov - 11 Mar 2017
Topic revision: r2 - 12 Mar 2017, AntonIvanov

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback