Modeling Human Eye Movement Using Adaptive Neuro-Fuzzy Inference Systems

Abstract

The eye’s muscles are difficult to model to build an eye prototype or an interface between the eye’s movements and computers; they require complex mechanical equations for describing their movements and the generated voltage signals from the eye are not always adequate for classification. However, they are very important for developing human machine interfaces based on eye movements. Previously, these interfaces have been developed for people with disabilities or they have been used for teaching the anatomy and movements of the eye’s muscles. However, the eye’s electrical signals have low amplitude and sometimes high levels of noise. Hence, artificial neural networks and fuzzy logic systems are implemented using an ANFIS topology to perform this classification. This paper shows how the eye’s muscles can be modeled and implemented in a concept prototype using an ANFIS topology that is trained using experimental signals from an end user of the eye prototype. The results show excellent performance for prototype when the ANFIS topology is deployed.

1 Introduction

Movement of the human eye is caused by the action of six extra ocular muscles. Because of the eye’s geometry and nature of movement, its mechanical analysis is complex and difficult to model. Similarly, understanding how these movements occur is challenging. In 1975, David A. Robinson wrote about the mechanics of ocular movement, describing the joint action of the extra ocular muscles [1]. Robinson used the mechanical properties of the extra ocular muscles to describe movement using force-balancing mechanical equations. Using this method, it’s possible to describe how the force is distributed between the different muscles to reach the horizontal, vertical, and torsion movements depending on the length and innervations of each muscle. A physical model that can fully reproduce the ocular movements still does not exist; virtual simulations are instead used by specialists to gain a better comprehension of how ocular movements occur [2]. Artificial intelligence techniques exist that allow linguistic rules to be transformed into an adequate control that leads to the creation of an approximated model close to reality. In this work, we propose a model for describing the human eye’s movement to allow a clear understanding of the joint action of the muscles involved in ocular movement with didactic purposes in areas such as medicine and ophthalmology.

2 General Aspects

2.1 Description of the Human Eye’s Movements

The human eye is our vision organ, and it is located in the orbital cavities. These cavities are bone structures which contain the eyeballs, the extra ocular muscles, nerves, blood vessels, fat, and most of the lacrimal apparatus. The eyeball covers one third of this cavity and consists of five sixths spherical segments, one comprising the fifth posterior parts and one comprising the sixth fore part. Ocular movement is mainly due to the conjunct action of six extra ocular muscles mentioned below: superior rectus, inferior rectus, medial rectus, lateral rectus, and superior and inferior oblique. All these muscles except for the inferior oblique have their origin at a common point called the Annulus of Zinn.

These muscles come in pairs, and therefore every pair has a common plane. This plane is formed by joining the midpoints of the origin of the muscles to the midpoints in the insertions of the eyeball tendons. The plane of the medial rectus and the lateral rectus muscles is a horizontal plane which divides the eye in two parts. The plane of the superior rectus and inferior rectus is a vertical plane that has a 23° angle with the fixation line when the eyes are looking to the front, as shown in Fig. 1. The oblique muscles have a plane that has a 51° angle with the fixation line when the eye is looking in the same position as mentioned [2].

Fig. 1.

Superior and inferior rectus angle with the plane

Ocular movement occurs on three axes (X, Y, Z) better known as Fick’s axes [3] as shown in Fig. 2. The movements generated by the extra ocular muscles are purely rotational due to the action of the ligaments that attach the orbital cavity with the eye. Translational movements are very limited, avoiding the eye to get out of position. Nevertheless, it is considered that the center of rotation is always inside a zone called the centroid. In an adult eye, the center of rotation is located approximately 13.5 mm behind the cornea and 1.6 mm from the eye’s nasal side [4].

Fig. 2.

Fick’s axes

2.2 Vector Analysis of the Human Eye

• θ Horizontal component of ocular rotation

• ϕ Vertical component of ocular rotation

• ψ Ocular torsion component

• O Origin of each muscle

• R Insertion point of each muscle in the eyeball

• $\overrightarrow {O}$ Vector that goes from the center of the eyeball (C) to the origin (O)

• $\overrightarrow {r}$ Vector that goes from the center of the eyeball (C) to the insertion point (R)

• $\vec{F}$ Force vector

• T Point where the force acts

• $\vec{m}$ Action unit vector

As shown in the Fig. 3, vector $\vec{F}$ begins to act from T, which is a muscular trajectory which goes from the insertion point to the separation of itself; therefore, the force is acting completely on the muscle. In order to solve this model, there are two main problems: the first problem is the innervations which consider the assumption that the eye is normal and when a new position is given it is necessary to find the six muscular innervations to keep it in that position. The second problem is when the innervations are known but the eye is abnormal. Therefore, it is necessary to find the deviation of the eye position.

Fig. 3.

Vector analysis of the eye movements

If the eye is moving in a horizontal position with an angle θ, in a vertical position with an angle ϕ, and with a torsion ψ, the eye will remain in this position as long as the sum of forces acting on it are zero. For determining the forces or moments, it is necessary to consider the insertion and origin points of each muscle; these follow a path through the eyeball’s surface ending at T. The exerted force of the muscle is directed from T to O. Therefore, the moment created by this force is a vector perpendicular to the plane formed by C, T, and O with its direction following the right-hand rule. Vector $\vec{m}$ is directed in the same direction of this axis and it is extremely important because its components $m_{x} ,\;m_{y} y m_{z}$ indicate the relative quantities of force distribution needed for acting on the eyeball vertically, horizontally and in torsion.

Vector $\vec{m}$ indicates the direction of the moment which has a magnitude F. Thus, the moment created by each muscle is $F\vec{m}$. There are seven moments in total, one for each extra ocular muscle and a passive moment $\vec{P}$ created for considering the muscular tissues that act for returning the eye to a neutral position close to the primary position. The force equilibrium equation results in the following:

$$ \vec{P} + F_{1} \overrightarrow {{m_{1} }} + F_{2} \overrightarrow {{m_{2} }} + \ldots + F_{6} \overrightarrow {{m_{6} }} = 0 $$

(1)

As mentioned, the force on each muscle in a stationary state depends only on its length L and its innervation I, so it can be expressed as F (L, I). Robinson [1] describes how this relationship is measured experimentally (Fig. 4):

Fig. 4.

Length-tension plot for different innervation levels

The experimental results show how the length-tension curves change for different innervation levels. This plot shows the average of the obtained data in which the curves were normalized allowing validity for all the extra ocular muscles.

It is possible to measure innervation by the force created if a fixed length is chosen, in this case the primary position length $L_{p}$. The length-tension curves are determined by some change in $L_{p}$ expressed as a percentage and defined as $\Delta l$. The resistance of a muscle is proportional to its cross-sectional area and it is defined as $\lambda$. Therefore, the force equilibrium equation can be rewritten as follows:

$$ \vec{P} + \sum\nolimits_{i = 1}^{6} {\lambda_{i} F\left( {\Delta l_{i} ,\Delta {\text{I}}_{\text{i}} } \right)\overrightarrow {{m_{i} }} = 0} $$

(2)

2.3 Artificial Intelligence Systems

The problem with modeling human eye movement is that it requires expressing many equations that could not be understandable by a field expert, like an ophthalmologist, for validation. Hence, we cannot obtain detailed information about the real eye movements or most of the times a precise model may not exist or is very difficult to model. For this, we use artificial intelligence because it facilitates the translation of a series of equations to linguistic rules that are given by the expert on the field whose knowledge and experience can explain the model [5]. In this work, we used a combination of fuzzy logic and neural networks to make an ANFIS system (Adaptative-Network-Based Fuzzy Inference System) because the complexity of most of the biological systems makes traditional quantitative approaches of analysis difficult [5].

Fuzzy Logic

Conventional mathematics tools, like differential equations, are not adequate for modeling systems that are not well defined or are with uncertainty. For this reason, fuzzy logic is useful for modeling systems using if-then rules because they can model qualitative aspects of reasoning and human knowledge without the need of quantitative analysis. Two approaches were created for fuzzy systems, the Mamdani and the Takagi-Sugeno. These models were explored systematically by Takagi and Sugeno, who found many applications in control, prediction, and inference. The Mamdani model is a fuzzy linguistic model that is focused on the interpretability, resulting in a fuzzy set. The Takagi-Sugeno model focuses on accuracy, resulting in crisp values [6].

There is no standard method to transform the human knowledge and experience into rules and databases of an inference system. There is also no completely effective method to tune the membership functions and minimize the output error or maximize its performance index. A fuzzy inference system is formed by five functional blocks [7, 8]:

• A rule base which contains several if-then rules.

• A database that defines the membership functions of the fuzzy sets used in the fuzzy rules.

• A decision-making unit which performs the inference operations on the rules.

• A fuzzification interface which transform the crisp inputs into degrees of match with the linguistic values.

• A defuzzification interface which transforms the fuzzy results of the inference into a crisp output.

Artificial Neural Networks

The theory and model of neural networks was inspired by the structure and function of the human nervous system where the neuron is the fundamental element. The neurons of the nervous system communicate to each other using signals. These signals are received by the dendrites and the cellular body, in which they are combined and emitted as output signals. The information is distributed by the axon to the axon’s terminals to be transmitted again to a new set of neurons. A similarity is created between biological neurons and artificial neurons is which the signals that enter the synapses are the inputs of the neuron, which are weighted depending on the strength of each synapse and can be attenuated or simplified through a parameter called synapse associated weight. The signals that pass through the synapses can be excitatory or inhibitory. The neuron is excited with a positive weight and inhibited with a negative weight; if the weight is zero, the synapse is lost.

Neural Networks have big advantages in the creation of intelligent systems. Their resemblance to the human brain allows them to learn from experience, generalize from previous cases, abstract essential characteristics, among others. Below are some of their advantages [9]:

• Adaptive Learning. The capacity to learn to complete tasks based on training or initial experience. The weights of the interconnections are adjusted to obtain the desired results. Learning continues after initial training. An adequate structure and learning algorithm should be chosen to achieve the desired capabilities.

• Auto organization. Through adaptive learning, networks classify the information they receive and subsequently modify their own organization during learning. This capability provides the faculty to react to information or situations not previously seen.

• Failure Tolerance. When there is a failure in the network, the network should be resilient and immediately begin reorganizing to recover; much like the human brain rewires and builds new pathways.

• Real-Time operation. Neural networks can be run in parallel for efficiency; therefore, machines are often designed with special hardware in mind for efficiency.

• Easy implementation with existing technology. The ease of training, testing, and verification allow neural networks to be implemented with low cost hardware.

ANFIS Systems

ANFIS systems, or Adaptive-Network-Based Fuzzy Inference Systems, combine the use of Artificial Neural Networks with Fuzzy Logic techniques to make an artificial inference system. Adaptive networks, as its name implies, are neural network structures that contain nodes that can be adapted. The adaptability relies on the fact that the output depends on parameters that belong to that specific node that change depending on the rule of learning, minimizing the error. Thus, an adaptive network is a multilayer network with forward connection in which each node carries out a function in the input signal and on the parameters.

Inside the network, the adaptive nodes are represented by squares. Circular nodes do not have parameters because they are not adaptive; their values are fixed. The set of parameters in an adaptive network is the union of all parameters in each node. To obtain the desired output in response to the input, the parameters are updated depending on the training data. The architecture is functionally equivalent to a fuzzy rule base of type Sugeno allowing it to tune the existing rules with a learning algorithm based on the collection of data from the training, optimizing the rule base and adapting to its environment. ANFIS systems use neural networks to optimize the coefficients of the membership functions instead of modifying the weights of the matrix W of the neural network [10, 11].

Unlike using only an artificial neural network to generate a solid knowledge base, ANFIS can construct an input-output mapping based on human knowledge and stipulated input-output data pairs. As ANFIS uses linguistic rules to generate its knowledge, its behavior becomes clearer, so it stops being a “black-box” [8].

3 Implementation

3.1 LabVIEW

The system used for this project is an ANFIS with two inputs and one output. The inputs are coordinates in the X and Y axes in which the eye moves. Three triangular membership functions where used for each axis. To train the system, we researched the percentage of elongation of each muscle for each movement. From these facts, we chose to normalize the minimum value that represents the maximum contraction and maximum values to represent the maximum elongation of each muscle fixing the biggest value as the unity value. With the use of the Anfiseditor library from Matlab software, an ANFIS network was created fixing the number and form of the membership functions. The method used was backpropagation with a maximum of 5% error allowed. The results show nine rules with equal number of output functions of Singleton type, where using the mass center equation, we obtained a crisp output value. The system was programmed in National Instruments LabVIEW software due to its easy connection to acquisition cards and its graphical programming language.

The layer 1 from Fig. 5 shows the membership functions, which are adaptable. Layer 2 and 3 have the suggested rules to obtain the minimum membership as well as its proper normalization, obtain the respective weights. In Layer 4, we have the adaptable output functions that allow its sum to give us the function of the desired muscle. In Layer 5, the functions are summed, making its behavior as a Sugeno type system.

Fig. 5.

Block diagram of the ANFIS system

In the first block of Fig. 6, the position in degrees is obtained; X ranges from −30° to 30°, and Y from 0° to 30°. The next block includes the programming for each muscle. The next step is a block that outputs a crisp value for each muscle. In the last block, the outputs are merged and sent to the acquisition card where 0 volts indicates completely closed and 5 volts completely opened.

Fig. 6.

Block diagram of the general program for the right eye

In the Fig. 7 the first block shows the square which receives position values. The second block shows the membership functions, three for X and three for Y. In the third block, a fuzzy numerical value is generated according to the belonging of the degree of membership in each of the membership functions. In the fourth block, the mass center is calculated by adding and multiplying the constant values generated by Matlab during the training phase using the following equations:

Fig. 7.

Software block of the artificial muscular elongation

$$ O_{2,i} = w_{i} =\upmu{\text{A}}_{\text{i}} \left( {\text{x}} \right) *\upmu{\text{B}}_{\text{i}} \left( {\text{y}} \right) $$

(3)

where i = 1, 2, and

$$ O_{5,1} = \frac{{\mathop \sum \nolimits_{i} w_{i} f_{i} }}{{\mathop \sum \nolimits_{i} w_{i} }} $$

(4)

where w_i is the degree of membership generated in the membership functions and f_i is the constant output function used. In the fifth block, the numerical output which represents the muscular elongation is shown. Next, in Fig. 8. the programming of the triangular membership functions used are described. The triangular functions are created by the values α, β, and γ, generated by Matlab’s ANFIS.

Fig. 8.

Control of a triangular membership function

The first block obtains the desired position which needs the membership value. The second block shows the activation logic, which activates each of the equations to calculate the triangular function. The third block returns a 0 when the value is less than the values of α and γ. Both functions in the middle calculate their positive and negative slope. In the fourth block, the membership degrees are compared by choosing the higher degree resulting in the fuzzy value. All membership functions were generated in the same way, changing only the values of α, β, and γ according to each function.

3.2 Physical Implementation

A model of the right eye with a 5:1 scale was built using a white acrylic sphere of 120 mm diameter for the eye and a transparent resin box for the structure for simulating the eye cavity, as shown in Fig. 9. To make the muscles of the eye, an air muscle was made; when these muscles are filled with air they expand and when the air is removed, they contract. To move the muscles, we used proportional air valves from Dynalloy which are based on Flexinol. These valves can gradually open or close the stem caps to produce proportionally controlled air flow. While heated with internal resistance, the Flexinol contracts and opens the cap; however, as it opens, air begins to flow through and cool the same wire. The equilibrium between the electrical input and the mass of air entering the valve determines the aperture size and air flow.

Fig. 9.

Physical eye model

According to the software and for simplicity of design, we chose to use the control signal for the muscles from 0 to 5 volts to produce 0 to 1 A which assured proper functioning of the valves. We created proportional conversion modules to accomplish this. Each module has a voltage to current converter with load to ground plus a power stage using transistors TIP 41 and TIP 42 for amplifying the current in a push-pull array, an op amp for compensation of the 0.7 volts drop of the emitter followers, and another op amp for impedance coupling and protection of the acquisition cards.

4 Results

Eye movement was controlled by software, in which the movements of the eye are selected by the rotational coordinates of the axes X and Y. However, to demonstrate the usability, we tested the eye’s model movement by imitating the movement of a real eye. For this, an electrooculography circuit was implemented. The signals were acquired by electrodes connected as shown in Fig. 10.

Fig. 10.

Position of electrodes

The movement of the eye is measured in degrees from its initial position to the desired position. To characterize the movement of the model, the degrees moved and time taken were measured for each movement. The measurements where made with the eye in its center position, as the initial position, and to the up, left, and right position; also, from the up position to the left and right. The time was measured in seconds and the velocity as the ratio of degrees and time. The results are shown in Table 1.

Table 1. Model measurements

Compared to the human eye, the model is 45 times slower due to its mechanical limitations. These limitations include the valve’s response time, the program processing time, the acquisition cards voltage response time, and the valves closed state to fully flowing state. The model is limited in the down motion because of the physical restrictions, achieving the 6 of the 9 basic eye positions.

This interface demonstrates to be simple and yet reliable to monitor the movement of the muscles in the human eye. Thus, the interface can be an advantage in many applications where the eye movement is tracked like in computer games [12] where it can be used as a joystick to navigate through a level, or to help severely disabled people [13] to help them control a computer, or a way to measure spatial attention [14] for therapies and studies. and even helping studies of visual marketing [15].

5 Conclusions

Representing human eye movements accurately using vector or mechanical analysis results is difficult because of the complexity of the human vision system. Nevertheless, it turns out to be easier to understand how eye movement is produced with the use of everyday language as “too much” or “too little”. With use of artificial intelligence techniques, such movements can be represented as a simple and easy to understand process, achieving good approximation of the real movement without the use of a mathematical model. Artificial intelligence techniques can be used as the link between physiology and engineering. With the use of this system, medical students can comprehend in a more interactive and didactic way such complexities. The muscles were a challenge since real muscles can be contracted and elongated. The design of the air muscles can elongate easily; but the contraction was made by rubber bands which not always applied the same force. This model proved to be a good alternative for emulating the eye’s movement for potential applications of algorithms designed for robotic systems.