A novel multi-sensor hardware and software system to realistically capture 3D and 4D spatiotemporal audiovisual representations of objects, hands and hand–object interactions
Overview
From animating a character in a movie to analyzing an athlete’s performance, 4D motion capture is a technique that allows the detailed tracking and analysis of a subject’s dynamic movement in three dimensions (X,Y,Z) over time. Such technologies have widespread applications in enhancing the realism of digital characters, improving biomechanical performance of athletes and refining virtual environments, making it an invaluable tool in both scientific discovery and creative expression. Traditionally, 4D motion capture requires sophisticated arrays of cameras, sensors and lights that are synchronized via software. Yet, these systems are expensive ($200k to $2M), bulky, constrained to lab settings and can be hard to calibrate for proper synchronization between the hardware and software of the system. The disclosed technology develops a modular, inexpensive 4D motion capture system composed of multi-sensor hardware integrated with novel software algorithms to support automated 4D reconstruction. In this way, the platform is more accessible in cost and applications, allowing the fast, efficient analysis of 4D data.
Market Opportunity
The disclosed 4D motion capture technology has widespread applications in film and video game animation, sport science analysis, healthcare applications such as surgical planning, biomechanics research, product development, robotics and AI-development. In each of these fields, the present technology can provide bespoke services with 4D audiovisual capture to meet a customer’s need. Further, the data analyzed with this novel 4D capture system can be used to drive innovation. For example, generative AI systems have seen a rapid rise in the last few years, with a current market value >$40 billion. Systems like ChatGPT and StableDiffusion can generate high-quality text and images that are almost indistinguishable from those that are produced by humans. Yet, these AI systems lack “physical intelligence,” or the ability to interact with and shape environments in a human-like manner as well as datasets to be trained on. Using the novel 4D motion capture system that is disclosed, robust data can be generated to train AI models, build robots with transformative physical capabilities and deepen research of human motor cognition.
Innovation and Meaningful Advantages
The proposed technology contains a low-cost hardware system built from easily-sourced components integrated with novel software for efficient sensor calibration and synchronization as well as automated 4D reconstruction. Unlike other 4D capture systems that require extensive training due to their reliance on special hardware and proprietary control software, the disclosed technology is designed to be easily built, customized and operated by undergraduate and graduate students with basic training. Although low-cost (50X cheaper than traditional systems), the hardware technology has multiple sensors including RGB cameras, microphone array, depth sensors, LED lights and accelerometers for appropriate tracking. Because of this, the platform allows for high-fidelity capture of motion. Further, two sizes have been developed which include a table-sized system for capturing human hands, small objects and/or small robotic arms as well as a room-sized platform that can support full body capture, larger objects and robots to accommodate a wide variety of applications.
Collaboration Opportunity: We are interested in exploring research collaborations and licensing opportunities.
References
Principal Investigator