Can a shelf know what is standing on it without cameras?
This is a project I had dreamed up a year before I got to build it. When the IoT course started I realized it was the perfect opportunity to actually try it, and fortunately my assigned group agreed. So we built a proof of concept: a shelf that keeps track of your items using nothing but weight. No cameras.
You scan the barcode with your phone before putting an item on the shelf. After that the shelf takes over. It keeps track of usage and lets you know if the milk has been out on the counter for too long, or if you are soon out of butter.
The hardware consist of four load cells with amplifiers, one in each corner of the shelf, are read by an ESP32. When something changes, the ESP32 sends the data over MQTT to a Raspberry Pi running Node-RED, which pushes it on to a Python API built with FastAPI. That is where all the logic lives. On top of that we built a ReactJS interface with a barcode scanner, an overview of the shelf contents and notifications, and we integrated a European food database for product information.
What's fascinating with this isn't the logic in itself, it's not that complex. But when calibrated correctly, and when weighing different calculations of the signal against each other, we managed to get quite accurate results. The most fascinating part was to see how accurate it really became: you can remove multiple items, use them so their weight changes, put them back in random order at new positions, and the shelf still keeps track of each one! That in combination with the open food database and barcode scanning makes this a surprisingly functional product, even in this early prototyping stage.
Dividing each sensor value by the total removes the weight and leaves only the pattern. A light and a heavy item in the same spot give the same normalized signature.
Each time something is lifted off or put back, the measured change is scored against the candidates: the items on the shelf when something is removed, the items currently out when something is returned. The score is based mainly on weight, supported by the sensor pattern, and the highest score wins. When several items are out, the returns are held pending until all are back and then assigned together, resolved based on match scores.
An item’s weight is shared between the four corner sensors. The closer a corner is to the item, the more of the weight it carries, so the sum gives the weight and the split gives the position.
Wanna see more of it?
The API and UI are available on my GitHub: @ellieskod and if you wanna talk more with me about this feel free to reach out on linkedin or email me!
