Augmented Reality Sandbox
A four-meter AR sandbox for Kopački Rit
A real-time augmented reality sandbox at Tikveš. Visitors shape the sand, and three Kinects and two projectors turn it into living terrain, with contour lines, flowing water and floods.
The Presentation and Education Centre Tikveš, in Kopački Rit Nature Park, wanted a hands-on way to explain how the wetland works: how water moves through it, how floods spread, and how the shape of the land decides the rest.
We built an augmented reality sandbox for it: an irregularly shaped table of real sand, about four meters long, watched by three Azure Kinect depth sensors and lit by two projectors.
What visitors do
Visitors dig, pile and smooth the sand, and the landscape updates under their hands in real time: elevation colors, contour lines, and water that runs downhill and pools in the hollows. They can flood the whole table and watch the water find its way through the terrain, the way the Danube does in the real Kopački Rit.
There's no text on the sandbox. A guide tells the story, and the sand becomes whatever the story needs: a riverbed, a flood plain, a marsh.
Why we built our own
Most AR sandboxes run on the same few open-source projects. They're a good starting point, but limited and slow, and not made for a large, irregular table with several sensors and projectors.
We wrote ours from scratch in Unreal Engine, with custom C++ and shaders. The terrain, water and visuals all run in real time, and every layer can be changed to fit what the museum wants to show.
Calibration was the hard part
Three sensors look at the sand and two projectors draw on it, and all of them have to agree on where every grain is.
Our first approach was the textbook one: solve the alignment mathematically, with transformation matrices. It didn't hold up. The real installation had more than angles to correct: lens distortion, uneven mounting, a sandbox that wasn't quite the shape of its plans.
What worked was a tool that let us drag the projected mesh into place by hand, like Liquify in Photoshop. Off-the-shelf projection mapping software wasn't an option, because it only solves the output side. A sandbox also needs its input, the Kinects, aligned to the same space. We ended up building a whole set of custom calibration, debugging and evaluation tools just to get there.
The centre in the press
The Presentation and Education Centre Tikveš opened in May 2022, and local media covered the opening:
- Unreal Engine
- C++
- Python