Fully Automated Patch-Based Image Restoration: Application to Pathology Inpainting

  • Ferran PradosEmail author
  • M. Jorge Cardoso
  • Niamh Cawley
  • Baris Kanber
  • Olga Ciccarelli
  • Claudia A. M. Gandini Wheeler-Kingshott
  • Sébastien Ourselin
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10154)


Pathology can have an important impact on MRI analysis. Specifically, white matter hyper-intensities, tumours, infarcts, etc., can influence the results of various image analysis techniques such as segmentation and registration. Several algorithms have been proposed for image inpainting and restoration, mainly in the context of Multiple Sclerosis lesions. These techniques commonly rely on a set of manually segmented pathological regions for inpainting. Rather than relying on prior segmentations for image restoration, we present a combined segmentation and inpainting algorithm for multimodal images. The proposed method is based on an iterative collaboration between two patch-based techniques, PatchMatch and Non-Local Means, where the former is used to estimate the most probable location of the pathological outliers and the latter to gradually fill the segmented areas with the most plausible multimodal texture. We demonstrate that the proposed method is able to automatically restore multimodal intensities in pathological regions within the context of Multiple Sclerosis.


Multiple Sclerosis Lesion Secondary Progressive Multiple Sclerosis Image Inpainting Pathological Region Patch Similarity 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



FP, BK and SO are funded by the National Institute for Health Research University College London Hospitals Biomedical Research Centre (NIHR BRC UCLH/UCL High Impact SO receives funding from the EPSRC (EP/H046410/1, EP/J020990/1, EP/K005278), the MRC (MR/J01107X/1) and the NIHR Biomedical Research Unit (Dementia) at UCL. This work was also supported by the Medical Research Council, the UK Multiple Sclerosis Society (grant 892/08) and the Brain Research Trust.


  1. 1.
    Lladó, X., Oliver, A., Cabezas, M., Freixenet, J., Vilanova, J.C., Quiles, A., Valls, L., Ramió, L., Rovira, A.: Segmentation of multiple sclerosis lesions in brain MRI: a review of automated approaches. Inf. Sci. 186(1), 164–185 (2012)CrossRefGoogle Scholar
  2. 2.
    Gordillo, N., Montseny, E., Sobrevilla, P.: State of the art survey on MRI brain tumor segmentation. Magn. Reson. Imaging 31(8), 1426–1438 (2013)CrossRefGoogle Scholar
  3. 3.
    Rekik, I., Allassonnière, S., Carpenter, T.K., Wardlaw, J.M.: Medical image analysis methods in MR/CT-imaged acute-subacute ischemic stroke lesion: segmentation, prediction and insights into dynamic evolution simulation models. A critical appraisal. NeuroImage Clin. 1(1), 164–178 (2012)CrossRefGoogle Scholar
  4. 4.
    Ceccarelli, A., Jackson, J., Tauhid, S., Arora, A., Gorky, J., Dell’Oglio, E., Bakshi, A., Chitnis, T., Khoury, S., Weiner, H., Guttmann, C., Bakshi, R., Neema, M.: The impact of lesion in-painting and registration methods on voxel-based morphometry in detecting regional cerebral gray matter atrophy in multiple sclerosis. Am. J. Neuroradiol. 33(8), 1579–1585 (2012)CrossRefGoogle Scholar
  5. 5.
    Govindarajan, K.A., Datta, S., Hasan, K.M., Choi, S., Rahbar, M.H., Cofield, S.S., Cutter, G.R., Lublin, F.D., Wolinsky, J.S., Narayana, P.A., MRI Analysis Center at Houston, T.C.I.G.: Effect of in-painting on cortical thickness measurements in multiple sclerosis: a large cohort study. Hum. Brain Mapp. 36(10), 3749–3760 (2015)Google Scholar
  6. 6.
    Sdika, M., Pelletier, D.: Nonrigid registration of MS brain images using lesion inpainting for morphometry or lesion mapping. Hum. Brain Mapp. 30(4), 1060–1067 (2009)CrossRefGoogle Scholar
  7. 7.
    Chard, D.T., Jackson, J.S., Miller, D.H., Wheeler-Kingshott, C.: Reducing the impact of white matter lesions on automated measures of brain gray and white matter volumes. J. Magn. Reson. Imaging 32(1), 223–228 (2010)CrossRefGoogle Scholar
  8. 8.
    Battaglini, M., Jenkinson, M., De Stefano, N.: Evaluating and reducing the impact of white matter lesions on brain volume measurements. HBM 33(9), 2062–71 (2012)CrossRefGoogle Scholar
  9. 9.
    Guizard, N., Nakamura, K., Coupe, P., Arnold, D.L., Collins, D.L.: Non-local MS MRI lesion inpainting method for image processing. In: The endMS Conference (2013)Google Scholar
  10. 10.
    Prados, F., Cardoso, M.J., MacManus, D., Wheeler-Kingshott, C.A.M., Ourselin, S.: A modality-agnostic patch-based technique for lesion filling in multiple sclerosis. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 781–788. Springer, Cham (2014). doi: 10.1007/978-3-319-10470-6_97 Google Scholar
  11. 11.
    Guizard, N., Nakamura, K., Coupé, P., Vladimir, S., Fonov, V., Arnold, D., Collins, D.: Non-local means inpainting of MS lesions in longitudinal image processing. Front. Neurosci. 9, 456 (2015)CrossRefGoogle Scholar
  12. 12.
    Prados, F., Cardoso, M.J., Kanber, B., Ciccarelli, O., Kapoor, R., Wheeler-Kingshott, C.A.G., Ourselin, S.: A multi-time-point modality-agnostic patch-based method for lesion filling in multiple sclerosis. NeuroImage 139, 376–384 (2016)CrossRefGoogle Scholar
  13. 13.
    Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The generalized patchmatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15558-1_3 CrossRefGoogle Scholar
  14. 14.
    Shi, W., Caballero, J., Ledig, C., Zhuang, X., Bai, W., Bhatia, K., Marvao, A.M.S.M., Dawes, T., O’Regan, D., Rueckert, D.: Cardiac image super-resolution with global correspondence using multi-atlas PatchMatch. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8151, pp. 9–16. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40760-4_2 CrossRefGoogle Scholar
  15. 15.
    Ta, V.-T., Giraud, R., Collins, D.L., Coupé, P.: Optimized PatchMatch for near real time and accurate label fusion. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8675, pp. 105–112. Springer, Cham (2014). doi: 10.1007/978-3-319-10443-0_14 Google Scholar
  16. 16.
    Giraud, R., Ta, V.T., Papadakis, N., Manjon, J.V., Collins, D.L., Coupe, P.: An optimized patchmatch for multi-scale and multi-feature label fusion. NeuroImage 124(Pt. A), 770–782 (2016)Google Scholar
  17. 17.
    Prados, F., Cardoso, M.J., Cawley, N., Ciccarelli, O., Wheeler-Kingshott, C.A., Ourselin, S.: Multi-contrast patchmatch algorithm for multiple sclerosis lesion detection. In: ISBI 2015 - Longitudinal MS Lesion Segmentation Challenge, pp. 1–2 (2015)Google Scholar
  18. 18.
    Xu, Z., Bagci, U., Seidel, J., Thomasson, D., Solomon, J., Mollura, D.J.: Segmentation based denoising of PET images: an iterative approach via regional means and affinity propagation. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8673, pp. 698–705. Springer, Cham (2014). doi: 10.1007/978-3-319-10404-1_87 Google Scholar
  19. 19.
    Boyes, R.G., Gunter, J.L., Frost, C., Janke, A.L., Yeatman, T., Hill, D.L.G., Bernstein, M.A., Thompson, P.M., Weiner, M.W., Schuff, N., Alexander, G.E., Killiany, R.J., DeCarli, C., Jack, C.R., Fox, N.C.: Intensity non-uniformity correction using N3 on 3-T scanners. NeuroImage 39(4), 1752–1762 (2008)Google Scholar
  20. 20.
    Buades, A., Coll, B., Morel, J.M.: A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4(2), 490–530 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Cardoso, M.J., Wolz, R., Modat, M., Fox, N.C., Rueckert, D., Ourselin, S.: Geodesic information flows. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7511, pp. 262–270. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33418-4_33 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Ferran Prados
    • 1
    • 2
    Email author
  • M. Jorge Cardoso
    • 1
    • 4
  • Niamh Cawley
    • 2
  • Baris Kanber
    • 1
    • 2
  • Olga Ciccarelli
    • 2
  • Claudia A. M. Gandini Wheeler-Kingshott
    • 2
    • 3
  • Sébastien Ourselin
    • 1
    • 4
  1. 1.Translational Imaging Group, Centre for Medical Image Computing (CMIC)University College LondonLondonUK
  2. 2.NMR Research Unit, Queen Square MS CentreUCL Institute of NeurologyLondonUK
  3. 3.Brain MRI 3T CentreC. Mondino National Neurological InstitutePaviaItaly
  4. 4.Dementia Research CentreUCL Institute of NeurologyLondonUK

Personalised recommendations