Abstract
Tracking of anatomical structures has multiple applications in the field of biomedical imaging, including screening, diagnosing and monitoring the evolution of pathologies. Semi-automated tracking of elongated structures has been previously formulated as a problem suitable for deep reinforcement learning (DRL), but it remains a challenge. We introduce a maximum entropy continuous-action DRL neural tracker capable of training from scratch in a complex environment in the presence of high noise levels, Gaussian blurring and detractors. The trained model is evaluated on two-photon microscopy images of mouse cortex. At the expense of slightly worse robustness compared to a previously applied DRL tracker, we reach significantly higher accuracy, approaching the performance of the standard hand-engineered algorithm used for neuron tracing. The higher sample efficiency of our maximum entropy DRL tracker indicates its potential of being applied directly to small biomedical datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Dataset available at: https://www.zenodo.org/record/1182487#.XP2UBS2ZMxc.
References
Alansary, A., et al.: Automatic view planning with multi-scale deep reinforcement learning agents. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 277–285. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_32
Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep reinforcement learning: a brief survey. IEEE Signal Process. Mag. 34(6), 26–38 (2017)
Bass, C., Helkkula, P., De Paola, V., Clopath, C., Bharath, A.A.: Detection of axonal synapses in \(3\rm D\) two-photon images. PLoS ONE 12(9), 1–18 (2017)
Dai, T., et al.: Deep reinforcement learning for subpixel neural tracking. In: Proceedings of the International Conference on Medical Imaging with Deep Learning, pp. 130–150 (2019)
Fraz, M.M., et al.: Blood vessel segmentation methodologies in retinal images-a survey. Comput. Methods Programs Biomed. 108(1), 407–433 (2012)
Ghesu, F.C., et al.: Multi-scale deep reinforcement learning for real-time 3D-landmark detection in \(\rm CT\) scans. IEEE Trans. Pattern Anal. Mach. Intell. 41(1), 176–189 (2017)
Haarnoja, T., Tang, H., Abbeel, P., Levine, S.: Reinforcement learning with deep energy-based policies. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 1352–1361. JMLR.org (2017)
Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 (2018)
Haarnoja, T., et al.: Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 (2018)
Kanski, J.J., Bowling, B.: Clinical Ophthalmology: A Systematic Approach. Elsevier Health Sciences, Edinburgh (2011)
Li, R., Zeng, T., Peng, H., Ji, S.: Deep learning segmentation of optical microscopy images improves 3-D neuron reconstruction. IEEE Trans. Med. Imaging 36(7), 1533–1541 (2017)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2009)
Peng, H., Ruan, Z., Long, F., Simpson, J.H., Myers, E.W.: V3D enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets. Nat. Biotechnol. 28(4), 348 (2010)
Pinto, L., Andrychowicz, M., Welinder, P., Zaremba, W., Abbeel, P.: Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542 (2017)
Pinto, N., Cox, D.D., DiCarlo, J.J.: Why is real-world visual object recognition hard? PLoS Comput. Biol. 4(1), e27 (2008)
Poulin, P., et al.: Learn to track: deep learning for tractography. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 540–547. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_62
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
Skibbe, H., et al.: PAT-probabilistic axon tracking for densely labeled neurons in large 3-D micrographs. IEEE Trans. Med. Imaging 38(1), 69–78 (2018)
Smeulders, A.W., Chu, D.M., Cucchiara, R., Calderara, S., Dehghan, A., Shah, M.: Visual tracking: an experimental survey. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1442–1468 (2013)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press Ltd., Cambridge (2018). https://mitpress.mit.edu/books/reinforcement-learning-second-edition
Uslu, F., Bharath, A.A.: A multi-task network to detect junctions in retinal vasculature. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 92–100. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_11
Zhang, P., Wang, F., Zheng, Y.: Deep reinforcement learning for vessel centerline tracing in multi-modality 3D volumes. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 755–763. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_86
Ziebart, B.: Modeling purposeful adaptive behavior with the principle of maximum causal entropy (2010). http://search.proquest.com/docview/845728212/
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Balaram, S., Arulkumaran, K., Dai, T., Bharath, A.A. (2019). A Maximum Entropy Deep Reinforcement Learning Neural Tracker. In: Suk, HI., Liu, M., Yan, P., Lian, C. (eds) Machine Learning in Medical Imaging. MLMI 2019. Lecture Notes in Computer Science(), vol 11861. Springer, Cham. https://doi.org/10.1007/978-3-030-32692-0_46
Download citation
DOI: https://doi.org/10.1007/978-3-030-32692-0_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32691-3
Online ISBN: 978-3-030-32692-0
eBook Packages: Computer ScienceComputer Science (R0)
