Smarter shop surveillance

 

Of course, extracting such information from a couple of video streams is more difficult than it might seem. First, you’ll have to know what you’re looking for. How do you actually see that someone is interested in the merchandise?

Imagine a fashion shop. Someone comes in, walks around and stops near a rack with blouses. The person will browse through the items, occasionally pick one from the rack and inspect it. Experienced shop assistants will know that now it’s time to contact the customer to ask if she (assuming the shopper is female) likes what she sees.

Mirela Popa focused her PhD research at the faculty of Electrical Engineering, Mathematics and Computer Science on translating this shopkeeper’s logic into motion analysis and decision software. With a master’s degree in Algorithms and Data structures (University of Transsylvania in Brasov, 2005) and a postmaster’s in software technology (TU Eindhoven, 2006 – 2008), she has the right credentials.

She decided that not one, but two different cameras would give her the information she needed. She explains in her thesis, “The fish-eye camera serves for human detection and tracking, conveying global information about the shopper’s behaviour. For a more refined analysis, the high-definition camera is used for recognizing a customer’s actions. Furthermore, the most detailed information about a shopper’s behaviour can be obtained from a web camera which enables the analysis of the displayed facial expressions.”

The real difficult part is then to integrate the information from these systems by means of a reasoning module into fused behaviour recognition. Such a module would need to automatically determine if someone is just entering the shop to ask some information, to browse through the racks habitually but without purpose, or intending to make an actual purchase.

Popa left retrieving relevant information from the video feeds to existing technology. On top of that, she based the integrating logic on the same statistical algorithms that are used in speech recognition. Such algorithms limit the practically endless number of possibilities on the basis of grammatical rules. For example, if you hear “see you,” you expect “soon” or “tomorrow” to follow but not “car” or “chicken.”

She reasoned there was a similar “grammar” in shopping behaviour, making some actions far more probable than others. This enabled her to use a probabilistic approach instead of a more rigid deterministic one, where one action is always followed by the next.

With automatic recognition rates of trajectories through the shop (a shopping laboratory actually), shopping actions, facial expressions and behavioural types around 90%, Dr. Popa delivered a working proof-of-concept of smart shop surveillance.

She suggests the system may be improved by analysing gestures and speech on top of the observed behaviours. The challenge ahead is to keep track of someone in busy surroundings where the system may have to re-identify a subject. Plus, she points out, that facial analysis is a strong and spontaneous indicator of product appreciations. So, if in the near future, a web shop requests access to your webcam, you now know why.

→ Mirela Carmia Popa, Shopping Behavior Assessment, Automatic System for Behavioral Cues Analysis, 08 Jan 2013, PhD thesis supervisors Prof. Leon Rothkrantz and Prof. Catholijn Jonker.