Refraction Wiggles for Measuring Fluid Depth and Velocity from Video

  • Tianfan Xue
  • Michael Rubinstein
  • Neal Wadhwa
  • Anat Levin
  • Fredo Durand
  • William T. Freeman
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8691)


We present principled algorithms for measuring the velocity and 3D location of refractive fluids, such as hot air or gas, from natural videos with textured backgrounds. Our main observation is that intensity variations related to movements of refractive fluid elements, as observed by one or more video cameras, are consistent over small space-time volumes. We call these intensity variations “refraction wiggles”, and use them as features for tracking and stereo fusion to recover the fluid motion and depth from video sequences. We give algorithms for 1) measuring the (2D, projected) motion of refractive fluids in monocular videos, and 2) recovering the 3D position of points on the fluid from stereo cameras. Unlike pixel intensities, wiggles can be extremely subtle and cannot be known with the same level of confidence for all pixels, depending on factors such as background texture and physical properties of the fluid. We thus carefully model uncertainty in our algorithms for robust estimation of fluid motion and depth. We show results on controlled sequences, synthetic simulations, and natural videos. Different from previous approaches for measuring refractive flow, our methods operate directly on videos captured with ordinary cameras, do not require auxiliary sensors, light sources or designed backgrounds, and can correctly detect the motion and location of refractive fluids even when they are invisible to the naked eye.


Particle Image Velocimetry Camera Center Brightness Constancy Background Orient Schlieren Representative Frame 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Adrian, R.J., Westerweel, J.: Particle image velocimetry. Cambridge University Press (2011)Google Scholar
  2. 2.
    Alterman, M., Schechner, Y.Y., Perona, P., Shamir, J.: Detecting motion through dynamic refraction. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 245–251 (2013)CrossRefGoogle Scholar
  3. 3.
    Alterman, M., Schechner, Y.Y., Swirski, Y.: Triangulation in random refractive distortions. In: 2013 IEEE International Conference on Computational Photography (ICCP), pp. 1–10. IEEE (2013)Google Scholar
  4. 4.
    Alterman, M., Schechner, Y.Y., Vo, M., Narasimhan, S.G.: Passive tomography of turbulence strength. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part IV. LNCS, vol. 8692, pp. 47–60. Springer, Heidelberg (2014)Google Scholar
  5. 5.
    Arnaud, E., Mémin, É., Sosa, R., Artana, G.: A fluid motion estimator for schlieren image velocimetry. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006, Part I. LNCS, vol. 3951, pp. 198–210. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  6. 6.
    Atcheson, B., Ihrke, I., Heidrich, W., Tevs, A., Bradley, D., Magnor, M., Seidel, H.P.: Time-resolved 3D capture of non-stationary gas flows. ACM Trans. Graph. 27(5), 132:1–132:9 (2008)Google Scholar
  7. 7.
    Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. International Journal of Computer Vision 92(1), 1–31 (2011)CrossRefGoogle Scholar
  8. 8.
    Dalziel, S.B., Hughes, G.O., Sutherland, B.R.: Whole-field density measurements by “synthetic schlieren”. Experiments in Fluids 28(4), 322–335 (2000)CrossRefGoogle Scholar
  9. 9.
    Ding, Y., Li, F., Ji, Y., Yu, J.: Dynamic fluid surface acquisition using a camera array. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2478–2485 (2011)Google Scholar
  10. 10.
    Elsinga, G., van Oudheusden, B., Scarano, F., Watt, D.: Assessment and application of quantitative schlieren methods: Calibrated color schlieren and background oriented schlieren. Experiments in Fluids 36(2), 309–325 (2004)CrossRefGoogle Scholar
  11. 11.
    Fleet, D., Weiss, Y.: Optical flow estimation. In: Handbook of Mathematical Models in Computer Vision, pp. 237–257. Springer, US (2006)CrossRefGoogle Scholar
  12. 12.
    Hargather, M.J., Settles, G.S.: Natural-background-oriented schlieren imaging. Experiments in Fluids 48(1), 59–68 (2010)CrossRefGoogle Scholar
  13. 13.
    Has, P., Herzet, C., Mmin, E., Heitz, D., Mininni, P.D.: Bayesian estimation of turbulent motion. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1343–1356 (2013)CrossRefGoogle Scholar
  14. 14.
    Jonassen, D.R., Settles, G.S., Tronosky, M.D.: Schlieren “PIV” for turbulent flows. Optics and Lasers in Engineering 44(3-4), 190–207 (2006)CrossRefGoogle Scholar
  15. 15.
    Meier, G.: Computerized background-oriented schlieren. Experiments in Fluids 33(1), 181–187 (2002)CrossRefGoogle Scholar
  16. 16.
    Morris, N.J., Kutulakos, K.N.: Dynamic refraction stereo. In: 2005 IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 1573–1580 (2005)Google Scholar
  17. 17.
    Raffel, M., Tung, C., Richard, H., Yu, Y., Meier, G.: Background oriented stereoscopic schlieren (BOSS) for full scale helicopter vortex characterization. In: Proc. of International Symposium on Flow Visualization (2000)Google Scholar
  18. 18.
    Richard, H., Raffel, M.: Principle and applications of the background oriented schlieren (BOS) method. Measurement Science and Technology 12(9), 1576 (2001)CrossRefGoogle Scholar
  19. 19.
    Ruhnau, P., Kohlberger, T., Schnörr, C., Nobach, H.: Variational optical flow estimation for particle image velocimetry. Experiments in Fluids 38(1), 21–32 (2005)CrossRefGoogle Scholar
  20. 20.
    Ruhnau, P., Schnörr, C.: Optical stokes flow estimation: An imaging-based control approach. Experiments in Fluids 42(1), 61–78 (2007)CrossRefGoogle Scholar
  21. 21.
    Ruhnau, P., Stahl, A., Schnörr, C.: On-line variational estimation of dynamical fluid flows with physics-based spatio-temporal regularization. In: Franke, K., Müller, K.-R., Nickolay, B., Schäfer, R. (eds.) DAGM 2006. LNCS, vol. 4174, pp. 444–454. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  22. 22.
    Schardin, H.: Die schlierenverfahren und ihre anwendungen. In: Ergebnisse der Exakten Naturwissenschaften, pp. 303–439. Springer (1942)Google Scholar
  23. 23.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47(1-3), 7–42 (2002)CrossRefzbMATHGoogle Scholar
  24. 24.
    Settles, G.S.: Schlieren and shadowgraph techniques: visualizing phenomena in transparent media, vol. 2. Springer, Berlin (2001)Google Scholar
  25. 25.
    Settles, G.S.: The penn state full-scale schlieren system. In: Proc. of International Symposium on Flow Visualization (2004)Google Scholar
  26. 26.
    Stam, J.: Stable fluids. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1999, pp. 121–128. ACM Press/Addison-Wesley Publishing Co., New York (1999)CrossRefGoogle Scholar
  27. 27.
    Tian, Y., Narasimhan, S., Vannevel, A.: Depth from optical turbulence. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 246–253. IEEE (June 2012)Google Scholar
  28. 28.
    Wetzstein, G., Raskar, R., Heidrich, W.: Hand-held schlieren photography with light field probes. In: 2011 IEEE International Conference on Computational Photography (ICCP), pp. 1–8 (April 2011)Google Scholar
  29. 29.
    Wetzstein, G., Roodnick, D., Heidrich, W., Raskar, R.: Refractive shape from light field distortion. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1180–1186 (2011)Google Scholar
  30. 30.
    Ye, J., Ji, Y., Li, F., Yu, J.: Angular domain reconstruction of dynamic 3d fluid surfaces. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 310–317 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Tianfan Xue
    • 1
  • Michael Rubinstein
    • 2
    • 1
  • Neal Wadhwa
    • 1
  • Anat Levin
    • 3
  • Fredo Durand
    • 1
  • William T. Freeman
    • 1
  2. 2.Microsoft ResearchUSA
  3. 3.Weizmann InstituteIsrael

Personalised recommendations