Seeing Tree Structure from Vibration

  • Tianfan XueEmail author
  • Jiajun Wu
  • Zhoutong Zhang
  • Chengkai Zhang
  • Joshua B. Tenenbaum
  • William T. Freeman
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11213)


Humans recognize object structure from both their appearance and motion; often, motion helps to resolve ambiguities in object structure that arise when we observe object appearance only. There are particular scenarios, however, where neither appearance nor spatial-temporal motion signals are informative: occluding twigs may look connected and have almost identical movements, though they belong to different, possibly disconnected branches. We propose to tackle this problem through spectrum analysis of motion signals, because vibrations of disconnected branches, though visually similar, often have distinctive natural frequencies. We propose a novel formulation of tree structure based on a physics-based link model, and validate its effectiveness by theoretical analysis, numerical simulation, and empirical experiments. With this formulation, we use nonparametric Bayesian inference to reconstruct tree structure from both spectral vibration signals and appearance cues. Our model performs well in recognizing hierarchical tree structure from real-world videos of trees and vessels.


Vibration Tree structure Hierarchical Bayesian model 



This work is supported by NSF #1231216, #1212849, and #1447476, ONR MURI N00014-16-1-2007, Toyota Research Institute, Shell Research, and Facebook. We thank Xiuming Zhang for helpful discussions.

Supplementary material

474192_1_En_46_MOESM1_ESM.pdf (218 kb)
Supplementary material 1 (pdf 217 KB)


  1. 1.
    Bascle, B., Blake, A., Zisserman, A.: Motion deblurring and super-resolution from an image sequence. In: Buxton, B., Cipolla, R. (eds.) ECCV 1996. LNCS, vol. 1065, pp. 571–582. Springer, Heidelberg (1996). Scholar
  2. 2.
    Blei, D.M., Griffiths, T.L., Jordan, M.I.: The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies. JACM 57(2), 7 (2010)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bouman, K.L., Xiao, B., Battaglia, P., Freeman, W.T.: Estimating the material properties of fabric from video. In: ICCV (2013)Google Scholar
  4. 4.
    Braddick, O.: Segmentation versus integration in visual motion processing. Trends Neurosci. 16(7), 263–268 (1993)CrossRefGoogle Scholar
  5. 5.
    Canny, J.: A computational approach to edge detection. IEEE TPAMI 8(6), 679–698 (1986)CrossRefGoogle Scholar
  6. 6.
    Davies, M.N., Green, P.R.: Perception and Motor Control in Birds: an Ecological Approach. Springer, Heidelberg (2012)Google Scholar
  7. 7.
    Davis, A., Bouman, K.L., Chen, J.G., Rubinstein, M., Durand, F., Freeman, W.T.: Visual vibrometry: estimating material properties from small motion in video. In: CVPR (2015)Google Scholar
  8. 8.
    Dijkstra, E.W.: A note on two problems in connexion with graphs. Numer. Math. 1(1), 269–271 (1959)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Farlow, S.J.: Partial Differential Equations for Scientists and Engineers. Courier Corporation, North Chelmsford (1993)zbMATHGoogle Scholar
  10. 10.
    Fleet, D.J., Jepson, A.D.: Computation of component image velocity from local phase information. IJCV 5(1), 77–104 (1990)CrossRefGoogle Scholar
  11. 11.
    Fraz, M.M., et al.: Blood vessel segmentation methodologies in retinal images-a survey. Comput. Methods Programs Biomed. 108(1), 407–433 (2012)CrossRefGoogle Scholar
  12. 12.
    French, A.: Vibrations and Waves. WW Norton, New York (1971)Google Scholar
  13. 13.
    Furoh, T., Fukumori, T., Nakayama, M., Nishiura, T.: Detection for lombard speech with second-order mel-frequency cepstral coefficient and spectral envelope in beginning of talking-speech. J. Acoust. Soc. Am. 133(5), 3246 (2013)CrossRefGoogle Scholar
  14. 14.
    Gautama, T., Van Hulle, M.: A phase-based approach to the estimation of the optical flow field using spatial filtering. IEEE TNN 13(5), 1127–1136 (2002)Google Scholar
  15. 15.
    Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE TPAMI 6(6), 721–741 (1984)CrossRefGoogle Scholar
  16. 16.
    Gershman, S.J., Tenenbaum, J.B., Jäkel, F.: Discovering hierarchical motion structure. Vis. Res. 126, 232–241 (2016)CrossRefGoogle Scholar
  17. 17.
    Grundmann, M., Kwatra, V., Han, M., Essa, I.: Efficient hierarchical graph-based video segmentation. In: CVPR (2010)Google Scholar
  18. 18.
    Hare, S., et al.: Struck: structured output tracking with kernels. IEEE TPAMI 38(10), 2096–2109 (2016)CrossRefGoogle Scholar
  19. 19.
    Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE TPAMI 37(3), 583–596 (2015)CrossRefGoogle Scholar
  20. 20.
    James, K.R., Dahle, G.A., Grabosky, J., Kane, B., Detter, A.: Tree biomechanics literature review: dynamics. J. Arboric. Urban For. 40, 1–15 (2014)Google Scholar
  21. 21.
    James, K.R., Haritos, N., Ades, P.K.: Mechanical stability of trees under dynamic loads. Am. J. Bot. 93(10), 1522–1530 (2006)CrossRefGoogle Scholar
  22. 22.
    James, K., Haritos, N.: Branches and damping on trees in winds. In: Australasian Conference on the Mechanics of Structures and Materials (2014)Google Scholar
  23. 23.
    Jepson, A.D., Fleet, D.J., Black, M.J.: A layered motion representation with occlusion and compact spatial support. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 692–706. Springer, Heidelberg (2002). Scholar
  24. 24.
    Knill, D.C., Richards, W.: Perception as Bayesian inference. Cambridge University Press, Cambridge (1996)CrossRefGoogle Scholar
  25. 25.
    Lee, T.C.: Building skeleton models via 3-D medial surface axis thinning algorithms. CVGIP 56(6), 462–478 (1994)Google Scholar
  26. 26.
    Lee, T.S., Mumford, D.: Hierarchical bayesian inference in the visual cortex. JOSA A 20(7), 1434–1448 (2003)CrossRefGoogle Scholar
  27. 27.
    Liu, C.: Beyond pixels: exploring new representations and applications for motion analysis. Ph.D. thesis, Citeseer (2009)Google Scholar
  28. 28.
    Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: IJCAI (1981)Google Scholar
  29. 29.
    Maninis, K.-K., Pont-Tuset, J., Arbeláez, P., Van Gool, L.: Deep retinal image understanding. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 140–148. Springer, Cham (2016). Scholar
  30. 30.
    Miller, L.A.: Structural dynamics and resonance in plants with nonlinear stiffness. J. Theor. Biol. 234(4), 511–524 (2005)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Moore, J.R., Maguire, D.A.: Natural sway frequencies and damping ratios of trees: concepts, review and synthesis of previous studies. Trees 18(2), 195–203 (2004)CrossRefGoogle Scholar
  32. 32.
    Moreno-Bote, R., Knill, D.C., Pouget, A.: Bayesian sampling in visual perception. PNAS 108(30), 12491–12496 (2011)CrossRefGoogle Scholar
  33. 33.
    Murphy, K.D., Rudnicki, M.: A physics-based link model for tree vibrations. Am. J. Bot. 99(12), 1918–1929 (2012)CrossRefGoogle Scholar
  34. 34.
    Pathak, D., Girshick, R., Dollár, P., Darrell, T., Hariharan, B.: Learning features by watching objects move. In: CVPR (2017)Google Scholar
  35. 35.
    Rubinstein, M.: Analysis and visualization of temporal variations in video. Ph.D. thesis, MIT (2013)Google Scholar
  36. 36.
    Rubinstein, M., Liu, C., Freeman, W.T.: Towards longer long-range motion trajectories. In: BMVC (2012)Google Scholar
  37. 37.
    Spelke, E.S., Breinlinger, K., Macomber, J., Jacobson, K.: Origins of knowledge. Psychol. Rev. 99(4), 605 (1992)CrossRefGoogle Scholar
  38. 38.
    Sun, D., Liu, C., Pfister, H.: Local layering for joint motion estimation and occlusion detection. In: CVPR (2014)Google Scholar
  39. 39.
    Sun, D., Sudderth, E.B., Black, M.J.: Layered segmentation and optical flow estimation over time. In: CVPR (2012)Google Scholar
  40. 40.
    Türetken, E., Benmansour, F., Andres, B., Głowacki, P., et al.: Reconstructing curvilinear networks using path classifiers and integer programming. IEEE TPAMI 38(12), 2515–2530 (2016)CrossRefGoogle Scholar
  41. 41.
    Türetken, E., González, G., Blum, C., Fua, P.: Automated reconstruction of dendritic and axonal trees by global optimization with geometric priors. Neuroinformatics 9(2–3), 279–302 (2011)CrossRefGoogle Scholar
  42. 42.
    Wang, J.Y., Adelson, E.H.: Layered representation for motion analysis. In: CVPR (1993)Google Scholar
  43. 43.
    Wang, Y., Narayanaswamy, A., Roysam, B.: Novel 4-D open-curve active contour and curve completion approach for automated tree structure extraction. In: CVPR (2011)Google Scholar
  44. 44.
    Weiss, Y., Adelson, E.H.: Slow and smooth: a Bayesian theory for the combination of local motion signals in human vision. Technical report, MIT (1998)Google Scholar
  45. 45.
    Wiener, N.: Extrapolation, Interpolation, and Smoothing of Stationary Time Series: with Engineering Applications. MIT Press, Cambridge (1949)zbMATHGoogle Scholar
  46. 46.
    Wu, H.Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.: Eulerian video magnification for revealing subtle changes in the world. ACM TOG 31(4), 65 (2012)CrossRefGoogle Scholar
  47. 47.
    Xue, T., Rubinstein, M., Liu, C., Freeman, W.T.: A computational approach for obstruction-free photography. ACM TOG 34(4), 79 (2015)CrossRefGoogle Scholar
  48. 48.
    Zhou, B., Hou, X., Zhang, L.: A phase discrepancy analysis of object motion. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6494, pp. 225–238. Springer, Heidelberg (2011). Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Tianfan Xue
    • 1
    Email author
  • Jiajun Wu
    • 2
  • Zhoutong Zhang
    • 2
  • Chengkai Zhang
    • 2
  • Joshua B. Tenenbaum
    • 2
  • William T. Freeman
    • 2
    • 3
  1. 1.Google ResearchMountain ViewUSA
  2. 2.MIT CSAILCambridgeUSA
  3. 3.Google ResearchCambridgeUSA

Personalised recommendations