The Brief

For this project we had spectators at a National Stadium event take photos and immediately upload them. We then used a system we developed to run an image analysis of some of the photos and quickly create a 3D model of the stadium. The rapid processing enabled us to show the final result there at the same event. We also developed a new system for the web to analyze all uploaded images in a day and analyzed where in the stadium each image was taken. This allowed spectators to go online and use an SNS ID to see where they took their photos. There are services that use multiple images for 3D reconstruction of objects within those images, but you can only input around 100 images and processing takes at least an hour, so you cannot use these services as they are to process the thousands of images we wanted to input.

Using conventional technology as is was a big problem, but, sticking to the premise of a "3D reconstruction of the National Stadium," we lowered the computational workload through various constraints so we were able to release the 3D reconstruction during the event and find the locations by analyzing the thousands of photos from that day. Much research has already explored 3D reconstruction using multiple images and commercial services do exist. However, none of them could fulfill our needs. So we developed an entirely different image processing pipeline to release the 3D image at the event and post the model online at a later date. To release the 3D image at the event, we generated an elaborate model of the National Stadium by using an existing service in advance, and on the day of the event we analyzed the locations of some photos with another tool. Then we rapidly generated the 3D model with texture mapping, employing that day's photos along with the pre-rendered model. A problem with generating the model online was the analysis to "find the location of thousands of photos." Generally, the process of running an image analysis of correlations between photos and finding locations involves a computational workload that is proportional to the number of images to the fourth power. In other words, an analysis of 100 photos that takes minutes will take days to analyze thousands by the same process. So by randomly selecting a group of about 100 photos to find their locations and developing a new system to integrate the final results, we succeeded in analyzing thousands of images in one day.

By developing this system, we were able for the first time ever to use photos taken by spectators, create a 3D reconstruction of the venue and release the result right there. Although this system was customized for the National Stadium, we will continue developing the technology so we can use it for general purposes. Academic research often collects images online to use for 3D reconstruction of buildings, but this was the first test in which the event organizers asked for fan photos. Thus if we release these photos as a standard data set for academic research, it will have tremendous value, and we are considering that.