Abstract
In this paper, bounded Laplace mixture model (BLMM) is proposed. The parameters of proposed model are estimated by maximum likelihood approach via expectation maximization and Newton–Raphson algorithm. The model is proposed for data modeling to perform clustering using synthetic data for univariate and multivariate examples and real datasets of different medical experiments. BLMM is validated through correctness of estimated parameters for synthetic data and clustering accuracy of medical datasets. A new modeling scheme is also introduced for wavelet coefficients which is based on BLMM. It is applied to image clustering and content-based image retrieval (CBIR) for feature extraction in wavelet domain. For feature extraction in this application, each image is decomposed into a set of wavelet subspaces and BLMM with two components is adopted to model the statistical characteristics of the wavelet coefficients for each wavelet subspace. The model parameters adapted from BLMM represent the image features in wavelet domain for each subspace and selected to formulate the feature space which is further used in clustering and CBIR. In the framework for clustering and image retrieval, features extracted in wavelet domain are further modeled through BLMM to categorize images into different groups and trained model is adopted for CBIR. In order to perform image retrieval with trained model via BLMM, city block distance, posterior probability and Kullback–Leibler divergence are introduced. We also propose a novel solution to compute Kullback–Leibler divergence which is very effective for image retrieval due to its low computational complexity and high retrieval rate. The effectiveness and viability of BLMM in texture image clustering and CBIR are demonstrated through UIUC, KTH-TIPS, DTD, STex and Kylberg databases. Different experiments are performed in the chosen applications, and from the results, BLMM has demonstrated its effectiveness in modeling synthetic data, real datasets from medical experiments, feature extraction in wavelet domain, image clustering and CBIR.
Similar content being viewed by others
References
Allili MS, Bouguila N, Ziou D (2008) Finite general Gaussian mixture modeling and application to image and video foreground segmentation. J Electron Imaging 17(1):013005
Amin T, Zeytinoglu M, Guan L (2007) Application of Laplacian mixture model to image and video retrieval. IEEE Trans Multimed 9(7):1416–1429
Andrearczyk V, Whelan PF (2017) Deep learning in texture analysis and its application to tissue image classification. In: Depeursinge A, Al-Kadi OS, Mitchell JR (eds) Biomedical texture analysis. Elsevier, Amsterdam, pp 95–129
Azam M, Bouguila N (2018) Bounded Laplace Mixture Model with Applications to Image Clustering and Content Based Image Retrieval. In: 2018 17th IEEE international conference on machine learning and applications (ICMLA), pp 558–563, https://doi.org/10.1109/ICMLA.2018.00090
Azam M, Bouguila N (2019) Bounded generalized gaussian mixture model with ICA. Neural Process Lett 49(3):1299–1320. https://doi.org/10.1007/s11063-018-9868-7
Bai S, Bai X, Tian Q, Latecki LJ (2018) Regularized diffusion process on bidirectional context for object retrieval. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2018.2828815
Banerjee I, Kurtz C, Devorah AE, Do B, Rubin DL, Beaulieu CF (2018) Relevance feedback for enhancing content based image retrieval and automatic prediction of semantic image features: application to bone tumor radiographs. J Biomed Inform 84:123–135. https://doi.org/10.1016/j.jbi.2018.07.002
Banfield JD, Raftery AE (1993) Model-based Gaussian and non-Gaussian clustering. Biometrics 49:803–821
Bhowmick D, Davison A, Goldstein DR, Ruffieux Y (2006) A Laplace mixture model for identification of differential expression in microarray experiments. Biostatistics 7(4):630–641
Bradley PS, Mangasarian OL, Street WN (1997) Clustering via concave minimization. In: Advances in neural information processing systems, pp 368–374
Cheeseman PC, Stutz JC et al (1996) Bayesian classification (autoclass): theory and results. Adv Knowl Discov Data Min 180:153–180
Chen Y, Wang JZ (2004) Image categorization by learning and reasoning with regions. J Mach Learn Res 5(Aug):913–939
Chipman HA, Kolaczyk ED, McCulloch RE (1997) Adaptive Bayesian wavelet shrinkage. J Am Stat Assoc 92(440):1413–1421
Choy S, Tong C (2010) Statistical wavelet subband characterization based on generalized gamma density and its application in texture retrieval. IEEE Trans Image Process 19(2):281–289. https://doi.org/10.1109/TIP.2009.2033400
Cimpoi M, Maji S, Kokkinos I, Mohamed S, , Vedaldi A (2014) Describing textures in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)
Cord A, Ambroise C, Cocquerez JP (2006) Feature selection in robust clustering based on Laplace mixture. Pattern Recognit Lett 27(6):627–635
Dang VM (1998) Classification de données spatiales: modèles probabilistes et critères de partitionnement. PhD thesis, Compiègne
Dua D, Graff C (2017) UCI machine learning repository. http://archive.ics.uci.edu/ml
Elguebaly T, Bouguila N (2014) Background subtraction using finite mixtures of asymmetric Gaussian distributions and shadow detection. Mach Vis Appl 25(5):1145–1162
Elter M, Schulz-Wendtland R, Wittenberg T (2007) The prediction of breast cancer biopsy outcomes using two cad approaches that both emphasize an intelligible decision process. Med Phys 34(11):4164–4172
Eltoft T, Kim T, Lee TW (2006) On the multivariate Laplace distribution. IEEE Signal Process Lett 13(5):300–303
Ernst MD (1998) A multivariate generalized Laplace distribution. Comput Stat 13:227–232
Espindola R, Ebecken N (2005) On extending f-measure and g-mean metrics to multi-class problems. WIT Transactions on Information and Communication Technologies 35
Farag A, El-Baz A, Gimel’farb G (2006) Precise segmentation of multimodal images. IEEE Trans Image Process 15(4):952–968. https://doi.org/10.1109/TIP.2005.863949
Figueiredo MA, Jain AK (2002) Unsupervised learning of finite mixture models. IEEE Trans Pattern Anal Mach Intell 24(3):381–396
Franczak BC, Browne RP, McNicholas PD (2014) Mixtures of shifted asymmetric Laplace distributions. IEEE Trans Pattern Anal Mach Intell 36(6):1149–1157. https://doi.org/10.1109/TPAMI.2013.216
Fritz M, Hayman E, Caputo B, olof Eklundh J (2004) The kth-tips database
Gai S, Zhang B, Yang C, Yu L (2018) Speckle noise reduction in medical ultrasound image using monogenic wavelet and Laplace mixture distribution. Digital Signal Process 72:192–207. https://doi.org/10.1016/j.dsp.2017.10.006
Garg L, McClean S, Meenan B, El-Darzi E, Millard P (2009) Clustering patient length of stay using mixtures of Gaussian models and phase type distributions. In: 2009 22nd IEEE international symposium on computer-based medical systems, pp 1–7, https://doi.org/10.1109/CBMS.2009.5255245
Gil D, Girela JL, De Juan J, Gomez-Torres MJ, Johnsson M (2012) Predicting seminal quality with artificial intelligence methods. Expert Syst Appl 39(16):12564–12573
Gorodkin J (2004) Comparing two k-category assignments by a k-category correlation coefficient. Comput Biol Chem 28(5–6):367–374
Hedelin P, Skoglund J (2000) Vector quantization based on gaussian mixture models. IEEE Trans Speech Audio Process 8(4):385–401. https://doi.org/10.1109/89.848220
Hedelin P, Skoglund J, Samuelsson J (1999) Performance bounds for lpc spectrum quantization. In: 1999 IEEE international conference on acoustics, speech, and signal processing. proceedings. ICASSP99 (Cat. No.99CH36258), vol 2, pp 677–680, https://doi.org/10.1109/ICASSP.1999.759757
Jurman G, Furlanello C (2010) A unifying view for performance measures in multi-class prediction. arXiv preprint arXiv:1008.2908
Jurman G, Riccadonna S, Furlanello C (2012) A comparison of MCC and CEN error measures in multi-class prediction. PLoS ONE 7(8):e41882
Khozeimeh F, Alizadehsani R, Roshanzamir M, Khosravi A, Layegh P, Nahavandi S (2017a) An expert system for selecting wart treatment method. Comput Biol Med 81:167–175
Khozeimeh F, Jabbari Azad F, Mahboubi Oskouei Y, Jafari M, Tehranian S, Alizadehsani R, Layegh P (2017b) Intralesional immunotherapy compared to cryotherapy in the treatment of warts. Int J Dermatol 56(4):474–478
Kurgan LA, Cios KJ, Tadeusiewicz R, Ogiela M, Goodenday LS (2001) Knowledge discovery approach to automated cardiac spect diagnosis. Artif Intell Med 23(2):149–169
Kwitt R, Meerwald P (2019) Salzburg texture image database (STex). http://www.wavelab.at/sources/STex/. Accessed 15 Jan 2019
Kylberg G (2011) The Kylberg texture dataset v. 1.0. External report (Blue series) 35, Centre for Image Analysis, Swedish University of Agricultural Sciences and Uppsala University, Uppsala, Sweden. http://www.cb.uu.se/~gustaf/texture/
Law MHC, Figueiredo MAT, Jain AK (2004) Simultaneous feature selection and clustering using mixture models. IEEE Trans Pattern Anal Mach Intell 26(9):1154–1166. https://doi.org/10.1109/TPAMI.2004.71
Lazebnik S, Schmid C, Ponce J (2005) A sparse texture representation using local affine regions. IEEE Trans Pattern Anal Mach Intell 27(8):1265–1278. https://doi.org/10.1109/TPAMI.2005.151
Lindblom J, Samuelsson J (2003) Bounded support Gaussian mixture modeling of speech spectra. IEEE Trans Speech Audio Process 11(1):88–99. https://doi.org/10.1109/TSA.2002.805639
Little MA, McSharry PE, Roberts SJ, Costello DA, Moroz IM (2007) Exploiting nonlinear recurrence and fractal scaling properties for voice disorder detection. Biomed Eng Online 6(1):23
Malik F, Baharudin B (2013) Analysis of distance metrics in content-based image retrieval using statistical quantized histogram texture features in the DCT domain. J King Saud Univ Comput Inf Sci 25(2):207–218. https://doi.org/10.1016/j.jksuci.2012.11.004
Mayhew MB, Petersen BK, Sales AP, Greene JD, Liu VX, Wasson TS (2018) Flexible, cluster-based analysis of the electronic medical record of sepsis with composite mixture models. J Biomed Inform 78:33–42. https://doi.org/10.1016/j.jbi.2017.11.015
Mclachlan G, Basford K (1988) Mixture models: inference and applications to clustering. J R Stat Soc Ser C Appl Stat 38:384–385. https://doi.org/10.2307/2348072
McLachlan G, Krishnan T (2007) The EM algorithm and extensions, vol 382. Wiley, Hoboken
McLachlan G, Peel D (2004) Finite mixture models. Wiley, Hoboken
Medasani S, Krishnapuram R (1998) Categorization of image databases for efficient retrieval using robust mixture decomposition. In: Proceedings. IEEE workshop on content-based access of image and video libraries (Cat. No.98EX173), pp 50–54, https://doi.org/10.1109/IVL.1998.694495
Medasani S, Krishnapuram R (2001) Categorization of image databases for efficient retrieval using robust mixture decomposition. Comput Vis Image Underst 83(3):216–235. https://doi.org/10.1006/cviu.2001.0926
Mitianoudis N, Stathaki T (2005) Overcomplete source separation using Laplacian mixture models. IEEE Signal Process Lett 12(4):277–280. https://doi.org/10.1109/LSP.2005.843759
Najjar A, Gagne C, Reinharz D (2015) Two-step heterogeneous finite mixture model clustering for mining healthcare databases. In: 2015 IEEE international conference on data mining, pp 931–936, https://doi.org/10.1109/ICDM.2015.70
Neal RM, Hinton GE (1998) A view of the Em algorithm that justifies incremental, sparse, and other variants. In: Jordan MI (ed) Learning in graphical models. Springer, Berlin, pp 355–368
Nguyen H, Azam M, Bouguila N (2019a) Data clustering using variational learning of finite scaled dirichlet mixture models. In: 2019 IEEE 28th international symposium on industrial electronics (ISIE), pp 1391–1396, https://doi.org/10.1109/ISIE.2019.8781334
Nguyen H, Kalra M, Azam M, Bouguila N (2019b) Data clustering using online variational learning of finite scaled dirichlet mixture models. In: 2019 IEEE 20th international conference on information reuse and integration for data science (IRI), pp 267–274, https://doi.org/10.1109/IRI.2019.00050
Nguyen TM, Wu QJ, Zhang H (2014) Bounded generalized Gaussian mixture model. Pattern Recognit 47(9):3132–3142
Patrício M, Pereira J, Crisóstomo J, Matafome P, Gomes M, Seiça R, Caramelo F (2018) Using resistin, glucose, age and bmi to predict the presence of breast cancer. BMC Cancer 18(1):29
Peel D, McLachlan G (2000) Robust mixture modelling using the t distribution. Stat Comput 10(4):339–348
Rish I (2001) An empirical study of the naive Bayes classifier. Technical report
Romberg JK, Choi H, Baraniuk RG (2001) Bayesian tree-structured image modeling using wavelet-domain hidden Markov models. IEEE Trans Image Process 10(7):1056–1068. https://doi.org/10.1109/83.931100
Rui Y, Huang TS, Ortega M, Mehrotra S (1998) Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans Circuits Syst Video Technol 8(5):644–655. https://doi.org/10.1109/76.718510
Scricciolo C (2018) Bayes and maximum likelihood for L-1-Wasserstein deconvolution of Laplace mixtures. Stat Methods Appl 27(2):333–362
Shi F, Selesnick IW (2006) Multivariate quasi-Laplacian mixture models forwavelet-based image denoising. In: 2006 International Conference on Image Processing, pp 2625–2628, https://doi.org/10.1109/ICIP.2006.313048
Simoncelli EP (1997) Statistical models for images: Compression, restoration and synthesis. In: Conference record of the thirty-first Asilomar conference on signals, systems & computers, vol 1, pp 673–678. IEEE
Song Y, Li Q, Feng D, Zou JJ, Cai W (2016) Texture image classification with discriminative neural networks. Comput Vis Media 2(4):367–377
Subramaniam AD, Rao BD (2000) PDF optimized parametric vector quantization of speech line spectral frequencies. In: 2000 IEEE workshop on speech coding. proceedings. Meeting the challenges of the new millennium (Cat. No.00EX421), pp 87–89, https://doi.org/10.1109/SCFT.2000.878407
Sun J, Zhou A, Keates S, Liao S (2018) Simultaneous bayesian clustering and feature selection through student’s \({t}\) mixtures model. IEEE Trans Neural Netw Learn Syst 29(4):1187–1199. https://doi.org/10.1109/TNNLS.2016.2619061
Titterington D, Smith A, Makov U (1985) Statistical analysis of finite mixture distributions. Wiley, New York
Usevitch BE (2001) A tutorial on modern lossy wavelet image compression: foundations of jpeg 2000. IEEE Signal Process Mag 18(5):22–35. https://doi.org/10.1109/79.952803
Usevitch BE, Orchard MT (1995) Smooth wavelets, transform coding, and Markov-1 processes. IEEE Trans Signal Process 43(11):2561–2569. https://doi.org/10.1109/78.482107
Varma M, Zisserman A (2005) A statistical approach to texture classification from single images. Int J Comput Vis 62(1–2):61–81
Xu L, Jordan MI (1996) On convergence properties of the em algorithm for gaussian mixtures. Neural Comput 8(1):129–151
Yeh IC, Yang KJ, Ting TM (2009) Knowledge discovery on RFM model using Bernoulli sequence. Expert Syst Appl 36(3):5866–5871
Yuan H, Zhang XP (2004) Texture image retrieval based on a Gaussian mixture model and similarity measure using a Kullback divergence. In: 2004 IEEE international conference on multimedia and expo (ICME) (IEEE Cat. No.04TH8763), vol 3, pp 1867–1870, https://doi.org/10.1109/ICME.2004.1394622
Yuan H, Zhang XP (2010) Statistical modeling in the wavelet domain for compact feature extraction and similarity measure of images. IEEE Trans Circuits Syst Video Technol 20(3):439–445. https://doi.org/10.1109/TCSVT.2009.2031396
Yuan H, Zhang XP, Guan L (2003) Content-based image retrieval using a Gaussian mixture model in the wavelet domain. In: Proceedings of SPIE, vol 5150, pp 422–429, https://doi.org/10.1117/12.503262
Zheng L, Yang Y, Tian Q (2018) Sift meets CNN: a decade survey of instance retrieval. IEEE Trans Pattern Anal Mach Intell 40(5):1224–1244. https://doi.org/10.1109/TPAMI.2017.2709749
Acknowledgements
The completion of this research was made possible thanks to the Natural Sciences and Engineering Research Council of Canada (NSERC).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors declare that they have no conflict of interest.
Additional information
Communicated by V. Loia.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Azam, M., Bouguila, N. Multivariate bounded support Laplace mixture model. Soft Comput 24, 13239–13268 (2020). https://doi.org/10.1007/s00500-020-04737-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-020-04737-7