从现实的虚拟现实环境Clouds

The advent of cost-effective head-mounted displays marked a new era in immersive virtual reality (VR) and sparked widespread applications in engineering, science and education. An integral component of any virtual reality application is the virtual environment. While some applications may have a completely imaginary virtual environment, others require the realistic recreation of a site or building. Relevant examples can be found in gaming, heritage site preservation and building information modelling. This article discusses how point cloud technologies were used to create a realistic virtual environment for use as immersive and interactive surveying labs.

Creating a virtual environment that is based on a physical site necessitates geometric information to a high level of detail. Technologies such as unmanned aerial systems (UAS) and terrestrial laser scanning (TLS) generate dense and accurate 3D point clouds. Each method offers advantages and disadvantages that complement each other. For example, UAS maps at an altitude with a downward looking view angle, while scanners are set at vantage locations limited by line of sight obstructions. Identifying data gaps and merging their point clouds can create points that are more complete. For the purposes of our research, we re-created part of the Penn State Wilkes-Barre campus. We conducted UAS flights with nadir imaging and collected TLS scans from various locations, mostly around the building, and merged the point clouds through a custom merging algorithm. Figure 1 shows an example of the data gaps in the sUAS and TLS datasets. For example, in this case the UAS point cloud has gaps around tree areas and cannot capture vertical walls of the building. On the other hand, the TLS dataset cannot capture the top of the building, which is captured by the UAS dataset, but can capture the building’s vertical walls. The TLS dataset was used as the reference and consisted of 180 million points. The sUAS point cloud had 70 million points, and 25 million points were kept to fill gaps in the TLS dataset.

Terrain and Structures Modelling

The terrain and any man-made structures can be modelled based on the merged point cloud. Using existing segmentation algorithms, it is relatively easy to separate the point cloud into ground and non-ground and create a terrain mesh from the ground points. While detail meshes of several million faces can be created, this introduces challenges to game engines (such as Unity and Unreal) that are often used to create virtual environments. For a pleasant virtual experience, it is important to always maintain a minimum of 60 frames per second (FPS), while other virtual reality hardware manufacturers often recommend at least 90 FPS. This means that detailed meshes of a few million faces need to be simplified to a few thousand without compromising the accuracy of the terrain. For example, in our implementation the initial terrain mesh had 4 million faces; however, when approaching complex areas (such as around the building), the FPS dropped to less than 60 FPS. We simplified the terrain and used 50,000 faces with a difference of 5cm in root mean square error (RMSE). Figure 2 (left) shows this simplified terrain. The terrain mesh was then converted to a Unity Terrain to take advantage of Unity’s occlusion culling rendering process, which renders objects that are only visible in the user’s field of view, thus ensuring that 90 FPS is consistently maintained.

它耗时为建模建筑物（例如建筑物和路缘）的几何建模。尽管存在某些自动形状检测算法，但它们检测到原始形状，例如盒子，球体，圆柱体等。尽管对于不需要高水平的建模细节的应用程序足以满足，但对于沉浸式虚拟环境而言，这是不可接受的。考虑到建筑物具有复杂的窗户和门结构，拱形的人行道，扶手，停车场条纹，沥青上的字母标记（例如“停止”）等等。在我们的案例中，我们从点云手工模拟了人造物体，这很费力。图2（右）显示了以3DS Max中的这些3D模型的示例。我们将建模的3D结构的准确性与建筑物拐角和停车场条纹的93个检查站进行了比较，该检查点在10-20厘米内显示了一项协议。未来改进的形状检测算法将增强从点云中提取结构的自动化，从而使其更容易。对于树木，采用了另一种方法。我们使用了一个随机生成器（来自Unity的资产存储），并将它们放在从合并点云中标识的相应位置。

Textures

纹理是虚拟环境的重要组成部分。创建和应用看起来像物理对应物的纹理会产生一种现实感。虽然点云提供了重要的几何信息，但从UAS和TLS拍摄的图片没有足够的分辨率来创建纹理。由于用户可以非常接近虚拟现实中的3D对象，因此他们将能够从空中图像中看到像素。游戏开发软件中的内置材料与其物理对应物相似，可以使用它们，这是一个简单的解决方案。沥青和草纹理就是这种情况。但是，某些物理对象具有独特且复杂的模式，因此需要从头开始开发纹理。例如，外部建筑墙通常具有独特的图案。为了创建此类纹理，我们使用了特写摄影，并创建了与Photoshop中物理相似的图案。重要的是，这些纹理伴随着正常地图，在虚拟现实中具有深度的幻想。 From these textures, we created materials that were then applied to the 3D objects. Figure 3 shows a top view of the virtual environment we created, while Figure 4 shows close-up examples of our textures. Of note are the custom textures in the buildings and the surrounding environment. For instance, in Figure 4 (top left), the windows were made reflective and in Figure 4 (bottom left), the parking lot sign was faithfully recreated. In addition, we added ambient sounds such as birds and the movement of tree branches with wind and rustling leaves. These add a sense of realism, which are important to create a sense of immersion.

Research Results

This article presents a workflow on how to utilize point cloud technologies to create realistic virtual environments. We plan to use such realistic environments in surveying labs in immersive and interactive virtual reality. Merging point clouds generated from different acquisition methods allows for the acquisition of complete and high-detail geometric information. 3D modelling and texturing are important steps to make the environment realistic, but these processes take time and effort. Virtual reality technology is rapidly advancing and is quickly finding its way into surveying and engineering, increasing the demand for replicating physical environments to virtual ones, and opening the door for new professional endeavours.

Acknowledgements

This project has been supported via grants from the following sources: Schreyer Institute for Teaching Excellence of the Pennsylvania State University; the Engineering Technology and Commonwealth Engineering of the Pennsylvania State University; Chancellor Endowments from Penn State Wilkes-Barre. Students John Chapman, Vincent F. Pavill IV, Eric Williams, Joe Fioti, Donovan Gaffney, Malcolm Sciandra, Courtney Snow and adjunct Brandi Brace are acknowledged for their involvement in this research.

参考

Bolkas，D.，Chiampi，J.，Chapman，J。，＆Pavill，V。F.（2020）。通过融合了SUA和TLS点云，创建虚拟现实环境。国际图像和数据融合杂志, 1-26.

Bolkas, D., Chiampi, J., Chapman, J., Fioti, J., & Pavill IV, V. F. (2020). Creating Immersive And Interactive Surveying Laboratories In Virtual Reality: A Differential Leveling Example.ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences, 5(5).