AI-ASSISTED 2D-3D SEMANTIC SEGMENTATION FOR STONE LEVEL HBIM
Synopsis
The Scan-to-HBIM process plays a central role in documenting heritage buildings by converting unstructured reality-capture data into semantically enriched BIM models. However, automating this conversion remains challenging, particularly for historic structures with complex and irregular stone masonry. This paper investigates the use of Convolutional Neural Networks (CNNs) to support the segmentation of photogrammetry-derived point clouds from Romanesque stone masonry, aiming to automate the data-processing stage of Scan-to-HBIM workflows. A dual 2D–3D pipeline is proposed in which high-resolution façade images are segmented using U-Net and the Segment Anything Model (SAM) to identify stone blocks, mortar joints, and openings. In this framework, U-Net is trained to classify point-cloud data into defined masonry-related classes, while SAM is employed either to generate training masks for U-Net or to directly project selected segmentations onto the 3D point cloud. The workflow enables the creation of a model capable of automatically segmenting point-cloud data and assigning semantic attributes to specific classes. Applied to a Romanesque church in Portugal, the approach shows that AI can reliably detect primary masonry components and reduce the labour required for detailed classification, although challenges persist in fine-grained stone delineation and model generalisation across diverse datasets. Overall, the study demonstrates a complementary dual-CNN strategy with strong potential to support the generation of high-detail HBIM models.
