Learning Domain Knowledge for Façade Labelling

  • Dengxin Dai
  • Mukta Prasad
  • Gerhard Schmitt
  • Luc Van Gool
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7572)


This paper presents an approach to address the problem of image façade labelling. In the architectural literature, domain knowledge is usually expressed geometrically in the final design, so façade labelling should on the one hand conform to visual evidence, and on the other hand to the architectural principles – how individual assets (e.g. doors, windows) interact with each other to form a façade as a whole. To this end, we first propose a recursive splitting method to segment façades into a bunch of tiles for semantic recognition. The segmentation improves the processing speed, guides visual recognition on suitable scales and renders the extraction of architectural principles easy. Given a set of segmented training façades with their label maps, we then identify a set of meta-features to capture both the visual evidence and the architectural principles. The features are used to train our façade labelling model. In the test stage, the features are extracted from segmented façades and the inferred label maps. The following three steps are iterated until the optimal labelling is reached: 1) proposing modifications to the current labelling; 2) extracting new features for the proposed labelling; 3) feeding the new features to the labelling model to decide whether to accept the modifications. In experiments, we evaluated our method on the ECP façade dataset and achieved higher precision than the state-of-the-art at both the pixel level and the structural level.


Segmentation Method Visual Recognition Content Support Shape Grammar Optimal Labelling 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Müller, P., Wonka, P., Haegler, S., Ulmer, A., Gool, L.V.: Procedural modeling of buildings. In: SIGGRAPH (2006)Google Scholar
  2. 2.
    Müller, P., Zeng, G., Wonka, P., Gool, L.V.: Image-based procedural modeling of facades. In: SIGGRAPH (2007)Google Scholar
  3. 3.
    Shotton, J., Winn, J.M., Rother, C., Criminisi, A.: TextonBoost: Joint Appearance, Shape and Context Modeling for Multi-class Object Recognition and Segmentation. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 1–15. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  4. 4.
    Gould, S., Rodgers, J., Cohen, D., Elidan, G., Koller, D.: Multi-class segmentation with relative location prior. IJCV 80, 300–316 (2008)CrossRefGoogle Scholar
  5. 5.
    Shotton, J., Johnson, M., Cipolla, R.: Semantic texton forests for image categorization and segmentation. In: CVPR (2008)Google Scholar
  6. 6.
    Tighe, J., Lazebnik, S.: SuperParsing: Scalable Nonparametric Image Parsing with Superpixels. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 352–365. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Berg, A.C., Grabler, F., Malik, J.: Parsing images of architectural scenes. In: ICCV (2007)Google Scholar
  8. 8.
    Zhao, P., Fang, T., Xiao, J., Zhang, H., Zhao, Q., Quan, L., Buaa, V.: Rectilinear parsing of architecture in urban environment. In: CVPR (2010)Google Scholar
  9. 9.
    Wendel, A., Donoser, M., Bischof, H.: Unsupervised Facade Segmentation Using Repetitive Patterns. In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) DAGM 2010. LNCS, vol. 6376, pp. 51–60. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Shen, C.H., Huang, S.S., Fu, H., Hu, S.M.: Adaptive partitioning of urban facades. In: SIGGRAPH Asia (2011)Google Scholar
  11. 11.
    Xiao, J., Fang, T., Tan, P., Zhao, P., Ofek, E., Quan, L.: Image-based façade modeling. In: SIGGRAPH Asia (2008)Google Scholar
  12. 12.
    Xiao, J., Fang, T., Zhao, P., Lhuillier, M., Quan, L.: Image-based street-side city modeling. In: SIGGRAPH Asia (2009)Google Scholar
  13. 13.
    Dick, A., Torr, P., Cipolla, R.: Modelling and interpretation of architecture from several images. IJCV 60, 111–134 (2004)CrossRefGoogle Scholar
  14. 14.
    Li, Y., Sharf, A., Cohen-or, D., Chen, B.: 2d-3d fusion for layer decomposition of urban facades. In: ICCV (2011)Google Scholar
  15. 15.
    Musialski, P., Wimmer, M., Wonka, P.: Interactive coherence-based facade modeling. In: Eurographics (2012)Google Scholar
  16. 16.
    Teboul, O., Simon, L., Koutsourakis, P., Paragios, N.: Segmentation of building facades using procedural shape priors. In: CVPR (2010)Google Scholar
  17. 17.
    Teboul, O., Kokkinos, I., Koutsourakis, P., Paragios, N.: Shape grammar parsing via reinforcement learning. In: CVPR (2011)Google Scholar
  18. 18.
    Tu, Z.: Auto-context and its application to high-level vision tasks. In: CVPR (2008)Google Scholar
  19. 19.
    Socher, R., Lin, C.C., Ng, A.Y., Manning, C.D.: Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In: ICML (2011)Google Scholar
  20. 20.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. PAMI 22, 888–905 (2000)CrossRefGoogle Scholar
  21. 21.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. IJCV 59, 167–181 (2004)CrossRefGoogle Scholar
  22. 22.
    Barbu, A., Zhu, S.C.: Generalizing swendsen-wang to sampling arbitrary posterior probabilities. PAMI 27, 1239–1253 (2005)CrossRefGoogle Scholar
  23. 23.
    Szummer, M., Kohli, P., Hoiem, D.: Learning CRFs Using Graph Cuts. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 582–595. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  24. 24.
    Teboul, O.: Shape Grammar Parsing: Application to Image-based Modeling. PhD thesis, Ecole Centrale Paris (2011)Google Scholar
  25. 25.
    Breiman, L.: Random forests. Machine Learning 45, 5–32 (2001)zbMATHCrossRefGoogle Scholar
  26. 26.
    Bosch, A., Zisserman, A.: Bosch, A., Zisserman, A., Muñoz, X.: Image classification using random forests and ferns. In: ICCV (2007)Google Scholar
  27. 27.
    Wu, J., Rehg, J.: Where am i: Place instance and category recognition using spatial pact. In: CVPR (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Dengxin Dai
    • 1
    • 2
  • Mukta Prasad
    • 1
  • Gerhard Schmitt
    • 2
  • Luc Van Gool
    • 1
  1. 1.Computer Vision LabETH ZürichSwitzerland
  2. 2.Information ArchitectureETH ZürichSwitzerland

Personalised recommendations