Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction

Yuhui Wu; Yue Liu; Jiajun Wang

doi:10.1109/VRW58643.2023.00158

Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction

Yuhui Wu, Yue Liu^*, Jiajun Wang

^*Corresponding author for this work

School of Optics and Photonics

Beijing Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Citation (Scopus)

Abstract

Hand-object occlusion is crucial to enhance the realism of Aug-mented Reality, especially for egocentric hand-object interaction scenes. In this paper, a hand segmentation-based depth correction approach is proposed, which can help to realize real-time hand-object occlusion. We introduce a lightweight convolutional neural net-work to quickly obtain real hand segmentation mask. Based on the hand mask, different strategies are adopted to correct the depth data of hand and non-hand regions, which can implement hand-object occlusion and object-object occlusion simultaneously to deal with complex hand situations during interaction. The experimental re-sults demonstrate the feasibility of our approach presenting visually appealing occlusion effects.

Original language	English
Title of host publication	Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	631-632
Number of pages	2
ISBN (Electronic)	9798350348392
DOIs	https://doi.org/10.1109/VRW58643.2023.00158
Publication status	Published - 2023
Event	2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023 - Shanghai, China Duration: 25 Mar 2023 → 29 Mar 2023

Publication series

Name	Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023

Conference

Conference	2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023
Country/Territory	China
City	Shanghai
Period	25/03/23 → 29/03/23

Keywords

Computer graphics
Computing methodologies
Human computer interaction (HCI)
Human-centered computing
Image manipulation
Image processing
Interaction paradigms-Mixed / augmented reality

Access to Document

10.1109/VRW58643.2023.00158

Cite this

Wu, Y., Liu, Y., & Wang, J. (2023). Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction. In Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023 (pp. 631-632). (Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/VRW58643.2023.00158

Wu, Yuhui ; Liu, Yue ; Wang, Jiajun. / Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction. Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 631-632 (Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023).

@inproceedings{87fab01f56354a95a16953362ef40064,

title = "Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction",

abstract = "Hand-object occlusion is crucial to enhance the realism of Aug-mented Reality, especially for egocentric hand-object interaction scenes. In this paper, a hand segmentation-based depth correction approach is proposed, which can help to realize real-time hand-object occlusion. We introduce a lightweight convolutional neural net-work to quickly obtain real hand segmentation mask. Based on the hand mask, different strategies are adopted to correct the depth data of hand and non-hand regions, which can implement hand-object occlusion and object-object occlusion simultaneously to deal with complex hand situations during interaction. The experimental re-sults demonstrate the feasibility of our approach presenting visually appealing occlusion effects.",

keywords = "Computer graphics, Computing methodologies, Human computer interaction (HCI), Human-centered computing, Image manipulation, Image processing, Interaction paradigms-Mixed / augmented reality",

author = "Yuhui Wu and Yue Liu and Jiajun Wang",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023 ; Conference date: 25-03-2023 Through 29-03-2023",

year = "2023",

doi = "10.1109/VRW58643.2023.00158",

language = "English",

series = "Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "631--632",

booktitle = "Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023",

address = "United States",

}

Wu, Y, Liu, Y & Wang, J 2023, Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction. in Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023, Institute of Electrical and Electronics Engineers Inc., pp. 631-632, 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023, Shanghai, China, 25/03/23. https://doi.org/10.1109/VRW58643.2023.00158

Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction. / Wu, Yuhui; Liu, Yue; Wang, Jiajun.
Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 631-632 (Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction

AU - Wu, Yuhui

AU - Liu, Yue

AU - Wang, Jiajun

PY - 2023

Y1 - 2023

N2 - Hand-object occlusion is crucial to enhance the realism of Aug-mented Reality, especially for egocentric hand-object interaction scenes. In this paper, a hand segmentation-based depth correction approach is proposed, which can help to realize real-time hand-object occlusion. We introduce a lightweight convolutional neural net-work to quickly obtain real hand segmentation mask. Based on the hand mask, different strategies are adopted to correct the depth data of hand and non-hand regions, which can implement hand-object occlusion and object-object occlusion simultaneously to deal with complex hand situations during interaction. The experimental re-sults demonstrate the feasibility of our approach presenting visually appealing occlusion effects.

AB - Hand-object occlusion is crucial to enhance the realism of Aug-mented Reality, especially for egocentric hand-object interaction scenes. In this paper, a hand segmentation-based depth correction approach is proposed, which can help to realize real-time hand-object occlusion. We introduce a lightweight convolutional neural net-work to quickly obtain real hand segmentation mask. Based on the hand mask, different strategies are adopted to correct the depth data of hand and non-hand regions, which can implement hand-object occlusion and object-object occlusion simultaneously to deal with complex hand situations during interaction. The experimental re-sults demonstrate the feasibility of our approach presenting visually appealing occlusion effects.

KW - Computer graphics

KW - Computing methodologies

KW - Human computer interaction (HCI)

KW - Human-centered computing

KW - Image manipulation

KW - Image processing

KW - Interaction paradigms-Mixed / augmented reality

UR - http://www.scopus.com/inward/record.url?scp=85159705235&partnerID=8YFLogxK

U2 - 10.1109/VRW58643.2023.00158

DO - 10.1109/VRW58643.2023.00158

M3 - Conference contribution

AN - SCOPUS:85159705235

T3 - Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023

SP - 631

EP - 632

BT - Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023

Y2 - 25 March 2023 through 29 March 2023

ER -

Wu Y, Liu Y, Wang J. Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction. In Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 631-632. (Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023). doi: 10.1109/VRW58643.2023.00158

Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this