Journal of Digital Imaging

, Volume 32, Issue 5, pp 855–864 | Cite as

Image Annotation by Eye Tracking: Accuracy and Precision of Centerlines of Obstructed Small-Bowel Segments Placed Using Eye Trackers

  • Alfredo Lucas
  • Kang Wang
  • Cynthia Santillan
  • Albert Hsiao
  • Claude B. Sirlin
  • Paul M. MurphyEmail author


Small-bowel obstruction (SBO) is a common and important disease, for which machine learning tools have yet to be developed. Image annotation is a critical first step for development of such tools. This study assesses whether image annotation by eye tracking is sufficiently accurate and precise to serve as a first step in the development of machine learning tools for detection of SBO on CT. Seven subjects diagnosed with SBO by CT were included in the study. For each subject, an obstructed segment of bowel was chosen. Three observers annotated the centerline of the segment by manual fiducial placement and by visual fiducial placement using a Tobii 4c eye tracker. Each annotation was repeated three times. The distance between centerlines was calculated after alignment using dynamic time warping (DTW) and statistically compared to clinical thresholds for diagnosis of SBO. Intra-observer DTW distance between manual and visual centerlines was calculated as a measure of accuracy. These distances were 1.1 ± 0.2, 1.3 ± 0.4, and 1.8 ± 0.2 cm for the three observers and were less than 1.5 cm for two of three observers (P < 0.01). Intra- and inter-observer DTW distances between centerlines placed with each method were calculated as measures of precision. These distances were 0.6 ± 0.1 and 0.8 ± 0.2 cm for manual centerlines, 1.1 ± 0.4 and 1.9 ± 0.6 cm for visual centerlines, and were less than 3.0 cm in all cases (P < 0.01). Results suggest that eye tracking–based annotation is sufficiently accurate and precise for small-bowel centerline annotation for use in machine learning–based applications.


Eye tracking Machine learning Small bowel Centerline 


Funding information

Dr. Kang Wang was supported in part by the NIH grant T32EB005970.


  1. 1.
    Alajlan N: HopDSW: An approximate dynamic space warping algorithm for fast shape matching and retrieval. Journal of King Saud University - Computer and Information Sciences 23(1): 7–14, 2011. CrossRefGoogle Scholar
  2. 2.
    Allen BC, Leyendecker JR: MR enterography for assessment and management of small bowel Crohn disease. Radiol Clin N Am 52(4): 799–810, 2014. CrossRefGoogle Scholar
  3. 3.
    Aquina CT, Becerra AZ, Probst CP, Xu Z, Hensley BJ, Iannuzzi JC, Noyes K, Monson JRT, Fleming FJ: Patients with adhesive small bowel obstruction should be primarily managed by a surgical team. Ann Surg 264(3): 437–447, 2016. CrossRefGoogle Scholar
  4. 4.
    Berndt DJ, Clifford J: Using dynamic time warping to find patterns in time series.. In: Proceedings of the 3rd international conference on knowledge discovery and data mining, AAAIWS’94. AAAI Press, Seattle, 1994, pp 359–370.
  5. 5.
    Bruining DH, Zimmermann EM, Loftus EV, Sandborn WJ, Sauer CG, Strong SA: Society of Abdominal Radiology Crohn’s Disease-Focused Panel: Consensus Recommendations for Evaluation, Interpretation, and Utilization of Computed Tomography and Magnetic Resonance Enterography in Patients With Small Bowel Crohn’s Disease. Radiology 286(3): 776–799, 2018. CrossRefGoogle Scholar
  6. 6.
    Chartrand G, Cheng PM, Vorontsov E, Drozdzal M, Turcotte S, Pal CJ, Kadoury S, Tang A: Deep learning: A primer for radiologists. RadioGraphics 37(7): 2113–2131, 2017. CrossRefGoogle Scholar
  7. 7.
    Dale AM, Fischl B, Sereno MI: Cortical surface-based analysis. I. Segmentation and surface reconstruction. NeuroImage 9(2): 179–194, 1999. CrossRefGoogle Scholar
  8. 8.
    Ebner L, Tall M, Choudhury KR, Ly DL, Roos JE, Napel S, Rubin GD: Variations in the functional visual field for detection of lung nodules on chest computed tomography: Impact of nodule size, distance, and local lung complexity. Med Phys 44(7): 3483–3490, 2017. CrossRefGoogle Scholar
  9. 9.
    Feuerstein JD, Cheifetz AS: Crohn disease: Epidemiology, diagnosis, and management. Mayo Clin Proc 92(7): 1088–1103, 2017. CrossRefGoogle Scholar
  10. 10.
    Fischl B, Salat DH, Busa E, Albert M, Dieterich M, Haselgrove C, van der Kouwe A, Killiany R, Kennedy D, Klaveness S, Montillo A, Makris N, Rosen B, Dale AM: Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron 33(3): 341–355, 2002CrossRefGoogle Scholar
  11. 11.
    Frager D, Medwid SW, Baer JW, Mollinelli B, Friedman M: CT of small-bowel obstruction: value in establishing the diagnosis and determining the degree and cause. AJR Am J Roentgenol 162(1): 37–41, 1994. CrossRefGoogle Scholar
  12. 12.
    Frangi AF, Niessen WJ, Vincken KL, Viergever MA: Multiscale vessel enhancement filtering.. In: Medical Image Computing and Computer-Assisted Intervention — MICCAI’98, pp 130–137, 1998Google Scholar
  13. 13.
    Frimmel H, Näppi J, Yoshida H: Fast and robust computation of colon centerline in CT colonography. Med Phys 31(11): 3046–3056, 2004. CrossRefGoogle Scholar
  14. 14.
    Frimmel H, Näppi J, Yoshida H: Centerline-based colon segmentation for CT colonography. Med Phys 32(8): 2665–2672, 2005. CrossRefGoogle Scholar
  15. 15.
    Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Advances In Neural Information Processing Systems, 1–9Google Scholar
  16. 16.
    Lecun Y, Bottou L, Bengio Y, Haffner P: Gradient-based learning applied to document recognition. Proc IEEE 86(11): 2278–2324, 1998. CrossRefGoogle Scholar
  17. 17.
    Li Q, Sone S, Doi K: Selective enhancement filters for nodules, vessels, and airway walls in two- and three-dimensional CT scans. Med Phys 30(8): 2040–2051, 2003. CrossRefGoogle Scholar
  18. 18.
    Litjens G, Toth R, van de Ven W, Hoeks C, Kerkstra S, van Ginneken B, Vincent G, Guillard G, Birbeck N, Zhang J, Strand R, Malmberg F, Ou Y, Davatzikos C, Kirschner M, Jung F, Yuan J, Qiu W, Gao Q, Edwards PE, Maan B, van der Heijden F, Ghose S, Mitra J, Dowling J, Barratt D, Huisman H, Madabhushi A: Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challenge. Med Image Anal 18(2): 359–373, 2014. CrossRefGoogle Scholar
  19. 19.
    Lu L, Zhang D, Li L, Zhao J: Fully automated colon segmentation for the computation of complete colon centerline in virtual colonoscopy. IEEE Trans Biomed Eng 59(4): 996–1004, 2012. CrossRefGoogle Scholar
  20. 20.
    Magdy N, Sakr MA, Mostafa T, El-Bahnasy K (2015) Review on trajectory similarity measures. In: 2015 IEEE 7th international conference on intelligent computing and information systems (ICICIS), pp 613–619.
  21. 21.
    Maiora J, Ayerdi B, Graña M: Random forest active learning for AAA thrombus segmentation in computed tomography angiography images. Neurocomputing 126: 71–77, 2014. CrossRefGoogle Scholar
  22. 22.
    Maiora J, Graña M (2012) Abdominal CTA image analisys through active learning and decision random forests: Aplication to AAA segmentation. In: The 2012 international joint conference on neural networks (IJCNN), pp 1–7.
  23. 23.
    Metz CT, Schaap M, Weustink AC, Mollet NR, Van Walsum T, Niessen WJ: Coronary centerline extraction from CT coronary angiography images using a minimum cost path approach. Med Phys 36(12): 5568–5579, 2009. CrossRefGoogle Scholar
  24. 24.
    Milletari F, Navab N, Ahmadi SA (2016) V-Net: Fully convolutional neural networks for volumetric medical image segmentation, 1–11. arXiv:1606.04797
  25. 25.
    Mittal D, Kumar V, Saxena SC, Khandelwal N, Kalra N: Neural network based focal liver lesion diagnosis using ultrasound images. Comput Med Imaging Graph 35(4): 315–323, 2011. CrossRefGoogle Scholar
  26. 26.
    Papadopoulos DP, Clarke ADF, Keller F, Ferrari V: Training object class detectors from eye tracking data.. In: Computer Vision – ECCV 2014, Lecture Notes in Computer Science. Springer, Cham, 2014, pp 361–376,
  27. 27.
    Paulson EK, Thompson WM: Review of small-bowel obstruction: the diagnosis and when to worry. Radiology 275(2): 332–342, 2015. CrossRefGoogle Scholar
  28. 28.
    Ronneberger O, Fischer P, Brox T (2015) U-Net: Convolutional networks for biomedical image segmentation, 1–8. arXiv:1505.04597
  29. 29.
    Rubin GD, Roos JE, Tall M, Harrawood B, Bag S, Ly DL, Seaman DM, Hurwitz LM, Napel S, Roy Choudhury K: Characterizing search, recognition, and decision in the detection of lung nodules on CT scans: Elucidation with eye tracking. Radiology 274(1): 276–286, 2015. CrossRefGoogle Scholar
  30. 30.
    Santillan CS: MR imaging techniques of the bowel. Magn Reson Imaging Clin N Am 22(1): 1–11, 2014. CrossRefGoogle Scholar
  31. 31.
    Scrima A, Lubner MG, King S, Pankratz J, Kennedy G, Pickhardt PJ: Value of MDCT and clinical and laboratory data for predicting the need for surgical intervention in suspected small-bowel obstruction. AJR Am J Roentgenol 208(4): 785–793, 2017. CrossRefGoogle Scholar
  32. 32.
    Silva AC, Pimenta M, Guimarães LS: Small bowel obstruction: what to look for Radiographics: A review publication of the radiological. Society of North America, Inc 29(2): 423–439, 2009. Google Scholar
  33. 33.
    Spuhler C (2006) Interactive centerline finding in complex tubular structures (16697)Google Scholar
  34. 34.
    Spuhler C, Harders M, Szekely G: Fast and robust extraction of centerlines in 3d tubular structures using a scattered-snakelet approach. SPIE Medical Imaging 6144: 614442–614442–8, 2006. Google Scholar
  35. 35.
    Tall M, Choudhury KR, Napel S, Roos JE, Rubin GD: Accuracy of a remote eye tracker for radiologic observer studies: Effects of calibration and recording environment. Acad Radiol 19(2): 196–202, 2012. CrossRefGoogle Scholar
  36. 36.
    Vaughan N, Gabrys B: Comparing and combining time series trajectories using dynamic time warping. Proc Comput Sci 96: 465–474, 2016. CrossRefGoogle Scholar
  37. 37.
    Vincent G, Guillard G, Bowes M (2012) Fully automatic segmentation of the prostate using active appearance models, Medical image computing and computer-assisted intervention – MICCAI 2012.;type=Public&;file=Imorphics.pdf
  38. 38.
    Wang S, Summers RM: Machine learning and radiology. Med Image Anal 16(5): 933–951, 2012. CrossRefGoogle Scholar
  39. 39.
    Yasaka K, Akai H, Abe O, Kiryu S: Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: A preliminary study. Radiology 286(3): 887–896, 2017. CrossRefGoogle Scholar
  40. 40.
    Yu L, Yang X, Chen H, Qin J, Heng PA (2017) Volumetric convnets with mixed residual connections for automated prostate segmentation from 3d mr images, 31st AAAI conference on artificial intelligence, pp 66–72Google Scholar
  41. 41.
    Zar JH (2014) Biostatistical analysis, 5. ed., pearson new internat. ed edn. Always learning. Pearson Education Limited, Harlow. OCLC: 862984228Google Scholar
  42. 42.
    Zhang W, Liu J, Yao J, Louie A, Nguyen TB, Wank S, Nowinski WL, Summers RM: Mesenteric vasculature-guided small bowel segmentation on 3d CT. IEEE Trans Med Imaging 32(11): 2006–2021, 2013. CrossRefGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2019

Authors and Affiliations

  • Alfredo Lucas
    • 1
  • Kang Wang
    • 2
  • Cynthia Santillan
    • 2
  • Albert Hsiao
    • 2
  • Claude B. Sirlin
    • 2
  • Paul M. Murphy
    • 2
    Email author
  1. 1.Department of BioengineeringUniversity of CaliforniaSan DiegoUSA
  2. 2.Department of RadiologyUniversity of CaliforniaSan DiegoUSA

Personalised recommendations