Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality

Weiquan Liu; Baiqi Lai; Cheng Wang; Xuesheng Bian; Wentao Yang; Yan Xia; Xiuhong Lin; Shang Hong Lai; Dongdong Weng; Jonathan Li

doi:10.1109/VRW50115.2020.00178

Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality

Weiquan Liu, Baiqi Lai, Cheng Wang^*, Xuesheng Bian, Wentao Yang, Yan Xia, Xiuhong Lin, Shang Hong Lai, Dongdong Weng, Jonathan Li

^*Corresponding author for this work

School of Optics and Photonics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

29 Citations (Scopus)

Abstract

Large-scale Light Detection and Ranging (LiDAR) point clouds provide basic 3D information support for Augmented Reality (AR) in outdoor environments. Especially, matching 2D images across to 3D LiDAR point clouds can establish the spatial relationship of 2D and 3D space, which is a solution for the virtual-real registration of AR. This paper first provides a precise 2D-3D patch-volume dataset, which contains paired matching 2D image patches and 3D LiDAR point cloud volumes, by using the Mobile Laser Scanning (MLS) data from the urban scene. Second, we propose an end-to-end network, Siam2D3D-Net, to jointly learn local feature representations for 2D image patches and 3D LiDAR point cloud volumes. Experimental results indicate the proposed Siam2D3D-Net can match and establish 2D-3D correspondences from the query 2D image to the 3D LiDAR point cloud reference map. Finally, an application is used to evaluate the possibility of the proposed virtual-real registration of AR in outdoor environments.

Original language	English
Title of host publication	Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	655-656
Number of pages	2
ISBN (Electronic)	9781728165325
DOIs	https://doi.org/10.1109/VRW50115.2020.00178
Publication status	Published - Mar 2020
Event	2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020 - Atlanta, United States Duration: 22 Mar 2020 → 26 Mar 2020

Publication series

Name	Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020

Conference

Conference	2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020
Country/Territory	United States
City	Atlanta
Period	22/03/20 → 26/03/20

Keywords

2D-3D feature representation
Augmented reality
Human-centered computing
Outdoor AR
Visualization
Visualization techniques
cross-domain data matching
virtual-real registration

Access to Document

10.1109/VRW50115.2020.00178

Cite this

Liu, W., Lai, B., Wang, C., Bian, X., Yang, W., Xia, Y., Lin, X., Lai, S. H., Weng, D., & Li, J. (2020). Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality. In Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020 (pp. 655-656). Article 9090401 (Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/VRW50115.2020.00178

Liu, Weiquan ; Lai, Baiqi ; Wang, Cheng et al. / Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality. Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020. Institute of Electrical and Electronics Engineers Inc., 2020. pp. 655-656 (Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020).

@inproceedings{f28063c6e04e4270b9629dad1f6ecb77,

title = "Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality",

abstract = "Large-scale Light Detection and Ranging (LiDAR) point clouds provide basic 3D information support for Augmented Reality (AR) in outdoor environments. Especially, matching 2D images across to 3D LiDAR point clouds can establish the spatial relationship of 2D and 3D space, which is a solution for the virtual-real registration of AR. This paper first provides a precise 2D-3D patch-volume dataset, which contains paired matching 2D image patches and 3D LiDAR point cloud volumes, by using the Mobile Laser Scanning (MLS) data from the urban scene. Second, we propose an end-to-end network, Siam2D3D-Net, to jointly learn local feature representations for 2D image patches and 3D LiDAR point cloud volumes. Experimental results indicate the proposed Siam2D3D-Net can match and establish 2D-3D correspondences from the query 2D image to the 3D LiDAR point cloud reference map. Finally, an application is used to evaluate the possibility of the proposed virtual-real registration of AR in outdoor environments.",

keywords = "2D-3D feature representation, Augmented reality, Human-centered computing, Outdoor AR, Visualization, Visualization techniques, cross-domain data matching, virtual-real registration",

author = "Weiquan Liu and Baiqi Lai and Cheng Wang and Xuesheng Bian and Wentao Yang and Yan Xia and Xiuhong Lin and Lai, {Shang Hong} and Dongdong Weng and Jonathan Li",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020 ; Conference date: 22-03-2020 Through 26-03-2020",

year = "2020",

month = mar,

doi = "10.1109/VRW50115.2020.00178",

language = "English",

series = "Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "655--656",

booktitle = "Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020",

address = "United States",

}

Liu, W, Lai, B, Wang, C, Bian, X, Yang, W, Xia, Y, Lin, X, Lai, SH, Weng, D & Li, J 2020, Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality. in Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020., 9090401, Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020, Institute of Electrical and Electronics Engineers Inc., pp. 655-656, 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020, Atlanta, United States, 22/03/20. https://doi.org/10.1109/VRW50115.2020.00178

Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality. / Liu, Weiquan; Lai, Baiqi; Wang, Cheng et al.
Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020. Institute of Electrical and Electronics Engineers Inc., 2020. p. 655-656 9090401 (Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality

AU - Liu, Weiquan

AU - Lai, Baiqi

AU - Wang, Cheng

AU - Bian, Xuesheng

AU - Yang, Wentao

AU - Xia, Yan

AU - Lin, Xiuhong

AU - Lai, Shang Hong

AU - Weng, Dongdong

AU - Li, Jonathan

PY - 2020/3

Y1 - 2020/3

N2 - Large-scale Light Detection and Ranging (LiDAR) point clouds provide basic 3D information support for Augmented Reality (AR) in outdoor environments. Especially, matching 2D images across to 3D LiDAR point clouds can establish the spatial relationship of 2D and 3D space, which is a solution for the virtual-real registration of AR. This paper first provides a precise 2D-3D patch-volume dataset, which contains paired matching 2D image patches and 3D LiDAR point cloud volumes, by using the Mobile Laser Scanning (MLS) data from the urban scene. Second, we propose an end-to-end network, Siam2D3D-Net, to jointly learn local feature representations for 2D image patches and 3D LiDAR point cloud volumes. Experimental results indicate the proposed Siam2D3D-Net can match and establish 2D-3D correspondences from the query 2D image to the 3D LiDAR point cloud reference map. Finally, an application is used to evaluate the possibility of the proposed virtual-real registration of AR in outdoor environments.

AB - Large-scale Light Detection and Ranging (LiDAR) point clouds provide basic 3D information support for Augmented Reality (AR) in outdoor environments. Especially, matching 2D images across to 3D LiDAR point clouds can establish the spatial relationship of 2D and 3D space, which is a solution for the virtual-real registration of AR. This paper first provides a precise 2D-3D patch-volume dataset, which contains paired matching 2D image patches and 3D LiDAR point cloud volumes, by using the Mobile Laser Scanning (MLS) data from the urban scene. Second, we propose an end-to-end network, Siam2D3D-Net, to jointly learn local feature representations for 2D image patches and 3D LiDAR point cloud volumes. Experimental results indicate the proposed Siam2D3D-Net can match and establish 2D-3D correspondences from the query 2D image to the 3D LiDAR point cloud reference map. Finally, an application is used to evaluate the possibility of the proposed virtual-real registration of AR in outdoor environments.

KW - 2D-3D feature representation

KW - Augmented reality

KW - Human-centered computing

KW - Outdoor AR

KW - Visualization

KW - Visualization techniques

KW - cross-domain data matching

KW - virtual-real registration

UR - http://www.scopus.com/inward/record.url?scp=85085359468&partnerID=8YFLogxK

U2 - 10.1109/VRW50115.2020.00178

DO - 10.1109/VRW50115.2020.00178

M3 - Conference contribution

AN - SCOPUS:85085359468

T3 - Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020

SP - 655

EP - 656

BT - Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020

Y2 - 22 March 2020 through 26 March 2020

ER -

Liu W, Lai B, Wang C, Bian X, Yang W, Xia Y et al. Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality. In Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020. Institute of Electrical and Electronics Engineers Inc. 2020. p. 655-656. 9090401. (Proceedings - 2020 IEEE Conference on Virtual Reality and 3D User Interfaces, VRW 2020). doi: 10.1109/VRW50115.2020.00178

Learning to Match 2D Images and 3D LiDAR Point Clouds for Outdoor Augmented Reality

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this