Medical assistive system for automatic identification of prescribed medicines by visually challenged from the medicine box using invariant feature extraction

  • S. VasaviEmail author
  • P. R. S. Swaroop
  • Roshni Srinivas
Original Research


Developing assistive systems for visually challenged people is an active area of research in computer vision community. Such system provides a medical assistive tool to take the correct medicine at the right time as prescribed by the doctor and makes visually challenged people to live independently for their day to day activities. Many prototypes were developed to deal with misidentification of medicines but are incapable of determining exact pill picked by the person. In our proposed system, feature extraction is done to recognize the pills based on structural, texture and Hu moments. If the pill is picked from the medicine box, the label present on the pill is considered for text recognition. Pill label and expiry date are extracted from the label and classified using Convolutional Neural Network and is converted to speech. This audio is produced to indicate the person about the medicine picked. The proposed system is tested on benchmark datasets and medicines of Diabetes, Hypertension, Arthritis, and Polycystic Ovarian Diseases. Experimental results proved that our system is better than existing works.


Image analysis Pre-processing Feature extraction Artificial neural networks (ANN) CNN Text to speech engine Pill recognition Adverse drug event 



This project work is not funded. This research work is done for a societal cause.

Compliance with ethical standards

Conflict of interest

All the authors has no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. Aravind R, Jagadesh RP, Sankari M, Praveen N, Arokia Magdaline S (2017) Neural network based assistive system for text detection with voice output. IRJET 4(4):44–47Google Scholar
  2. Bourne RRA, Flaxman SR, Braithwaite T, Cicinelli MV, Das A, Jonas JB, Keeffe J (2017) Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. Lancet Global Health 5(9):888–897CrossRefGoogle Scholar
  3. Cai G, Fang Y, Wen J, Han G, Yan X (2019) QoS-aware buffer-aided relaying implant WBAN for healthcare IoT: opportunities and challenges. arXiv preprint arXiv 1902:1–8 (Cai2019QoSAwareBR,journal = {ArXiv}, 2019, vol. abs/1902.04443) Google Scholar
  4. Cook JD (2009) Three algorithms for converting color to grayscale. Accessed 05 Sept 2017]
  5. Ezaki N, Bulacu M, Schomaker L (2004) Text detection from natural scene images: towards a system for visually impaired persons, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004, vol 2, pp 683–686Google Scholar
  6. Hartl A, Arth C (2010) Computer-vision based pharmaceutical pill recognition on mobile phones, Proceedings of CESCG 2010: The 14th Central European Seminar on Computer Graphics, pp 1–8Google Scholar
  7. Lee D, Yoon H, Park C, Kim J, Park CH (2013) Automatic number recognition for bus route information aid for the visually-impaired, 10th International Conference on Ubiquitous Robots and Ambient Intelligence, pp 280–284Google Scholar
  8. Maddala KT, Moss RH, Stoecker WV, Hagerty JR, Cole JG, Mishra NK, Stanley RJ (2017) Adaptable ring for vision-based measurements and shape analysis. IEEE Trans Instrum Meas 66(4):746–756CrossRefGoogle Scholar
  9. Manwatkar PM, Singh KR (2015) A technical review on text recognition from images, IEEE 9th International Conference on Intelligent Systems and Control (ISCO), pp 721–725Google Scholar
  10. Marc’Aurelio R (2014) Tutorial on Large-Scale Visual Recognition CVPR 2014. Accessed 6 June 2018
  11. Ming-Kuei H (1962) Visual pattern recognition by moment invariants. IRE Trans Inf Theory 8(2):179–187CrossRefzbMATHGoogle Scholar
  12. Sobel I, Feldman G (1968) A 3 × 3 isotropic gradient operator for image processing, presented at the stanford artificial intelligence project (SAIL) in 1968Google Scholar
  13. Tay C, Birla M (2016) Pharmaceutical pill recognition using computer vision techniques, School of Informatics and Computing. Indiana University Bloomington, Bloomington, pp 1–12Google Scholar
  14. Wang Y, Ribera J, Liu C, Yarlagadda S, Zhu F (2017) Pill recognition using minimal labeled data, Proceedings of IEEE 3rd International Conference on Multimedia Big Data, pp 346–353Google Scholar
  15. Yang X, Shah SA, Ren A, Zhao N, Zhang Z, Fan D, Zhao J, Wang W, Ur-Rehman M (2019) Freezing of gait detection considering leaky wave cable. IEEE Trans Antennas Propag. 67(1):554–561CrossRefGoogle Scholar
  16. Zhi-Han L, Hui-Yin Y, Makmor-Bakry M (2017) Medication-handling challenges among visually impaired population. Arch Pharm Pract 8(1):8–14CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.VR Siddhartha Engineering CollegeVijayawadaIndia

Personalised recommendations