CNN Based Medical Assistive System for Visually Challenged to Identify Prescribed Medicines

  • P. R. S. Swaroop
  • S. VasaviEmail author
  • Roshni Srinivas
Conference paper
Part of the Learning and Analytics in Intelligent Systems book series (LAIS, volume 3)


Developing assistive systems for visually challenged people is an active area of research in computer vision community. Such system provides a medical assistive tool to take the correct medicine at the right time as prescribed by the doctor and makes visually challenged people to live independently for their day to day activities. Many prototypes were developed to deal with misidentification of medicines but are incapable of determining exact pill picked by the person. This paper presents an automated system where feature extraction is done to recognize the pills based on structural, texture and Hu moments. If the pill is picked from the medicine box, the label present on the pill is considered for Text Recognition. Pill label and expiry date are extracted from the label and classified using Convolutional Neural Network (CNN) and is converted to speech. This audio is produced to indicate the person about the medicine picked. Experimental results proved that our system is better than existing works.


Image analysis Pre-processing Feature extraction ANN CNN Text to speech engine Pill recognition Adverse drug event 


  1. 1.
    Bourne RRA, Flaxman SR, Braithwaite T, Cicinelli MV, Das A, Jonas JB, Keeffe J (2017) Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. Lancet Glob Health 5(9):888–897CrossRefGoogle Scholar
  2. 2.
    Zhi-Han Lg, Hui-Yin Y, Makmor-Bakry M (2017) Medication-handling challenges among visually impaired population. Arch Pharm Pract 8(1):8–14CrossRefGoogle Scholar
  3. 3.
    Wang Y, Ribera J, Liu C, Yarlagadda S, Zhu F (2017) Pill recognition using minimal labeled data. In: Proceedings of IEEE third international conference on multimedia big data, pp 346–353Google Scholar
  4. 4.
    Tay C, Birla M (2016) Pharmaceutical pill recognition using computer vision techniques. School of Informatics and Computing, Indiana University Bloomington, pp 1–12Google Scholar
  5. 5.
    Hartl A, Arth C (2010) Computer-vision based pharmaceutical pill recognition on mobile phones. In: Proceedings of CESCG 2010: The 14th Central European seminar on computer graphics, pp 1–8Google Scholar
  6. 6.
    Aravind R, Jagadesh RP, Sankari M, Praveen N, Arokia Magdaline S (2017) Neural network based assistive system for text detection with voice output. IRJET 4(4):44–47Google Scholar
  7. 7.
    Maddala KT, Moss RH, Stoecker WV, Hagerty JR, Cole JG, Mishra NK, Stanley RJ (2017) Adaptable ring for vision-based measurements and shape analysis. IEEE Trans Instrum Meas 66(4):746–756CrossRefGoogle Scholar
  8. 8.
    Ezaki, N, Bulacu M, Schomaker L (2004) Text detection from natural scene images: towards a system for visually impaired persons. In: Proceedings of the 17th international conference on pattern recognition, 2004. ICPR 2004, vol. 2, pp 683–686Google Scholar
  9. 9.
    Lee D, Yoon H, Park C, Kim J, Park CH (2013) Automatic number recognition for bus route information aid for the visually-impaired. In: 10th international conference on ubiquitous robots and ambient intelligence, pp 280–284Google Scholar
  10. 10.
    Manwatkar PM, Singh KR (2015) A technical review on text recognition from images. In: IEEE 9th international conference on intelligent systems and control (ISCO), pp 721–725Google Scholar
  11. 11.
    Vasavi S, Swaroop PRS, Srinivas R (2018) Medical assistive system for automatic identification of prescribed medicines by visually challenged using invariant feature extraction. IJMSTR 6(1):1–17Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • P. R. S. Swaroop
    • 1
  • S. Vasavi
    • 1
    Email author
  • Roshni Srinivas
    • 1
  1. 1.VR Siddhartha Engineering CollegeVijayawadaIndia

Personalised recommendations