CoarseBIMLoc: BIM-Integrated Coarse Global Localization via Open-Set Scene Graphs

Chang Chen1,5, Junyou Chen2, Yinqiang Zhang1, Ruihua Han1, Ming Zhang3, Liang Lu4, Jia Pan1,5
1The University of Hong Kong, 2Shandong University, 3City University of Hong Kong, 4Shenzhen University, 5Shenzhen Loop Area Institute

Successful Trials

prediction animation 124567 1

Trial 1

prediction animation 124567 2

Trial 2

prediction animation 572146 1

Trial 3

prediction animation 572146 2

Trial 4

prediction animation 752164 1

Trial 5

prediction animation 752164 2

Trial 6

Failures

Failure Case 1: no object exists

Failure Video 1

Failure Case 2: fail to detect object
(cooking bench)

Failure Video 2

Real-World Experiments

We conduct initial real-world experiments in a completed building using a panoramic camera-based sensor suite mounted on a person's head. As the person moves through the environment, the system captures a real-time RGB images in the scene (left). Meanwhile, the algorithm deployed on a portable computer performs open-set detection and scene graph construction, followed by semantic localization with the BIM hierarchical graph. The estimated global pose at each timestep is projected on the BIM map in a top-down view, accompanied by a rendered image at that pose (right). This online platform is developed by the SYNLOOP startup company.

Real-world experiment 1

Localization by matching a door

Real-world experiment 2

Localization by matching a fire hydrant