PanoTree: Automated Photospot Explorer in Virtual Reality Scenes

Tomohiro Hayase1, Braun Sacha2, Hikari Yanagawa1, Itsuki Orito3, Yuichi Hiroi1,
1Cluster Metaverse Lab 2École polytechnique 3Cluster, Inc
Paper Code
,

Exploration

Photospots

Abstract

Social VR platforms enable social, economic, and creative activities by allowing users to create and share their own virtual spaces. In social VR, photography within a VR scene is an important indicator of visitors’ activities. Although automatic identification of photo spots within a VR scene can facilitate the process of creating a VR scene and enhance the visitor experience, there are challenges in quantitatively evaluating photos taken in the VR scene and efficiently exploring the large VR scene. We propose PanoTree, an automated photo-spot explorer in VR scenes. To assess the aesthetics of images captured in VR scenes, a deep scoring network is trained on a large dataset of photos collected by a social VR platform to determine whether humans are likely to take similar photos. Furthermore, we propose a Hierarchical Optimistic Optimization (HOO)-based search algorithm to efficiently explore 3D VR spaces with the reward from the scoring network. Our user study shows that the scoring network achieves human-level performance in distinguishing randomly taken images from those taken by humans. In addition, we show applications using the explored photo spots, such as automatic thumbnail generation, support for VR world creation, and visitor flow planning within a VR scene.

Keywords: Reinforcement Learning, Vision Transformer, Tree Search, Social VR

panotree fig4

Fig. 4. Scoring network of the photo in VR space. The network learns whether the input image is human-captured or randomly captured and features the image likely to be captured by a human. The input images are labeled with 1 for human-captured and 0 for randomly captured. During the evaluation, for each input image, the network outputs a score indicating whether the image is likely to have been captured by a human.

panotree fig6

Fig. 6. (a) 3D Spatial division and tree structure of the PanoTree. Each scene is defined as a cuboid and is divided so that the longest edge is the most likely to be divided. (b) Sampled directional vectors \(\boldsymbol{\delta}_k\) (blue arrows) in \(N_\mathrm{dir}=15\). Each \(\boldsymbol{\delta}_k\) is a unit vector whose destination (red) is distributed over the unit sphere (green).

Video

BibTeX

@misc{hayase2024panotree,
      title={PanoTree: Autonomous Photo-Spot Explorer in Virtual Reality Scenes},
      author={Tomohiro Hayase and Braun Sacha and Hikari Yanagawa and Itsuki Orito and Yuichi Hiroi},
      year={2024},
      eprint={2405.17136},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}