The Brief

Following are the three major developments we've made for this project: 1. Software for a system that analyzes athletes' 3D motion and absolute position within the stadium and estimates the original camera position based on old footage. 2. A system that projects this collected data in life-size inside the stadium through projection mapping, laser projection, and AR technology, and a system that synchronizes the athletes' footage to real-time venue footage. It required us to develop software that fulfills all necessary functions by drawing from open source libraries in order to achieve swift angular field of view adjustment, even when in undesirable locations. 3. A system that integrates all data and systems to enable switching between live television and event footage. We were required to control the camera's switcher system, the footage shown on the electrical screen, the projection mapping footage, laser projection, the CG data for the AR synchronization, and interlocking lighting, all in real time.

This system enabled us to generate precise 3D motion data and initial camera data from old footage. This technology can not only be applied to image creation, but also to sports analytics, archiving, and other uses as well. This project is not the first time we've attempted to create such a system. In the past, the quality of 3D motion data created from 2D footage depended on the skills of the animator, which was a problem. Therefore, for this project we used a special device in the shape of a mannequin to input the data. This enabled us to reduce the burden of inputting data, and also prevented unnatural human anatomy. We've also achieved seamless image synchronization by estimating the initial camera position, angle, and lens distortion from the original footage. The present laser projection software has only basic correction functions, so we developed a software that enables calibration functions which suit the size of the stadium. Regarding the design of the integrated system, we had to integrate various interfaces, such as measuring protocols and operating signals for broadcast equipment, because there were many intermixed broadcasting systems, self-made devices and software needed to fulfill the necessary synchronization precision due to the nature of sports.

Our system, which creates 3D motion data from old footage, is currently ready for release because we have finished fixing the bugs and refining the interface thanks to a large number of staff. We're now considering presenting papers at academic conferences and releasing an Open Source tool. This integrated playback system is not designed for versatility. It was customized to the specific requirements of this event, namely for live real-time broadcasting, which means that the number of installable devices and precision are the trade-off. For this reason, this system could not be used for something like a video that requires post-production work; however, linked to broadcasting equipment, the system could be utilized in future television broadcasting.