Video-Based Gaze Detection for Oculomotor Abnormality Measurements

Abstract

Measuring oculomotor abnormalities in human subjects is challenging due to the delicate spatio-temporal nature of the oculometric measures (OMs) used to assess eye movement abilities. Some OMs require a gaze estimation accuracy of less than 2 degrees and a sample rate that enables the detection of movements lasting less than 100 ms. While past studies and applications have used dedicated and limiting eye tracking devices to extract OMs, recent advances in imaging sensors and computer vision have enabled video-based gaze detection. Here, we present a self-calibrating neural network model for gaze detection that is suitable for oculomotor abnormality measurement applications. The model considers stimuli target locations while the examined subjects perform visual tasks and calibrate its gaze estimation output in real time. The model was validated in a clinical trial and achieved an axial accuracy of 0.93 degrees and 1.31 degrees for horizontal and vertical gaze estimation locations, respectively, as well as an absolute accuracy of 1.80 degrees. The performance of the proposed model enables the extraction of OMs using affordable and accessible setups—such as desktop computers and laptops—without the need to restrain the patient’s head or to use dedicated equipment. This newly introduced approach may significantly ease patient burden and improve clinical results in any medical field that requires eye movement measurements.

1. Introduction

With the development of computational technology and automated machinery, eye movement measurements—specifically, gaze detection and tracking—have gained increasing interest in recent decades. Some applications of gaze detection aim to improve the user interface and engagement with various external complex controlled systems; for example, gaze-based human–computer interactions [1,2] enable the hands-free operation of software, while safety and awareness gaze-based assessments ensure the optimal functionality of drivers [3], pilots [4], and even surgeons [5,6]. Another set of applications intends to harness the remotely detected gaze to map the saliency of selected items in the overall field of view. Advertisement and marketing-oriented studies [7,8] have extensively implemented gaze detection, and tourism researchers have performed gaze detection along with the measurement of other emotional and physical metrics to estimate reactions to particular items or sights [9].

Another rapidly extending set of gaze detection applications is aimed at non-invasive measurement of eye movements in patients with various diseases. As a primary apparent output of the oculomotor system, gaze detection time-series data allow for further derivation and extraction of the eye movement features directly associated with a subject’s cognitive and motor functionality, enabling the detection of eye movement abnormalities that may be related to the patient’s condition.

Eye movement abnormalities refer to deviations or irregularities observed in the typical characteristic patterns of eye motion in healthy individuals. Such deviations may be indicative of underlying neurological, ophthalmological, or vestibular disorders, making them significant diagnostic indicators for clinicians [10].

The human eye engages in various movement types, including saccades (rapid movements between separate fixation points), smooth pursuit (continuously tracking moving objects), and nystagmus (involuntary rhythmic oscillations of the eyes). Abnormalities in these movements may manifest in various OMs in terms of accuracy, speed, co-ordination, and response patterns with respect to visual stimuli.

Abnormal eye movements may be the result of factors such as damage to the neural pathways governing eye movements [11], dysfunction in the muscles responsible for ocular motion [12], or disruptions in the vestibular system (which processes spatial orientation and balance) [13]. Consequently, eye movement abnormalities are observed across a spectrum of conditions, including those affecting the central nerve system (CNS) such as multiple sclerosis [14,15], Parkinson’s disease [16,17], and amyotrophic lateral sclerosis (ALS) [18,19]. Therefore, extracted eye movement features (i.e., OMs) have potential as new biomarkers for the diagnosis of several CNS diseases [16,18,20,21], strongly correlating with the patient’s condition and disease progression [22].

Clinicians routinely examine and assess eye movements as a diagnostic tool. Monitoring the accuracy, velocity, and co-ordination of eye movements provides valuable insights into the integrity of neural circuits involved in visual processing and motor control. While clinicians have conducted eye movement examinations for decades, the OMs extracted without specialized equipment or laboratory setup are limited, mainly due to the required temporal and spatial sensitivity. For instance, Saccade movement duration (the time from initialization of the movement until the first stop of the eye) may reach values in the order of 100 ms, posing challenges for the manual detection of eye movement abnormalities. Therefore, the introduction of gaze detection tools with high spatio-temporal accuracies is expected to enable the extraction of the relevant OM, allowing for a more confident observation of eye movement abnormalities.

Although humans can quite accurately detect someone else’s gaze visually (i.e., tell where that person looks), the equivalent technology used to locate a human subject’s point of regard (PoR) is still lacking. The most commonly used eye trackers at present are Infrared (IR) active sensors, which track the eye’s position and orientation to estimate the subject’s PoR. These specialized IR eye trackers are currently considered to be the most accurate measurement tool for gaze detection and can be mounted next to a desktop monitor [23] or embedded in wearable head devices (e.g., glasses) [24]. However, even in state-of-the-art IR eye trackers, researchers have observed significant sensitivity to the experimental setup and the diverse characteristics of subjects [25].

Video-based gaze detection has been in development since the 1970s, with many limitations and restrictions imposed on the experimental setup (e.g., fixed head position, an array of mirrors around the subject, multiple camera deployment) and light sources [26,27]. However, advances in hardware since the early 2000s have accelerated the development of video-based gaze detection using standard, affordable web cameras, with their resolution, sensitivity, and frame rate having significantly increased, thus enabling more accurate gaze detection using existing analytical algorithms. In recent years, both computation and storage costs have dropped while data transfer bandwidths have widened, enabling the introduction of efficient and robust neural network (NN) models for video-based gaze detection [28]. This recent leap has encouraged the development of high-accuracy, video-based gaze detection, which may be applied for all the above purposes, with an affordable price tag and a convenient and flexible (ideally seamless) setup.

We introduce and test a gaze detection model that combines several neural network models for eye detection, eye segmentation, and gaze estimation with an ongoing real-time calibration procedure. Our study is aimed at the neurological applications of gaze detection and considers the essential measurement for the extraction of OMs, performed using a simple and affordable setup that presents visual stimulation tasks and processes the video of the gazing subject captured using a webcam. As some OM classes (e.g., saccadic latency) require high spatio-temporal resolution, high accuracy is required when extracting the gaze PoR in each video stream frame.

In this paper, we present our gaze detection model, along with its validation and error estimation results. Section 2 describes the gaze detection model and the clinical study environment, including the experimental setup, examination protocol, and the demographic data of participants. Section 3 provides the results of the video-based gaze detection model, compared to a reference specialized IR eye tracker for validation. In Section 4, we discuss our findings.

2. Materials and Methods

2.1. Gaze Detection Model

The gaze detection model used in this study was developed based on existing published NN models for video-based eye detection, eye segmentation, and gaze detection [22,28], with the addition of information regarding visual stimuli target locations, as depicted in Figure 1.

The initial input—namely, an image containing the subject’s face—is first processed using the MediaPipe neural network. The MediaPipe Face model [29] detects face landmarks from still images or video; specifically, landmarks around the eyes, such as the iris and the eyelid boundaries. These landmarks are used to extract the eye position in each image. Then, using the eye position information, two smaller images of the left and right eye are cropped from the original image, of which one is horizontally flipped, as is common practice in the field [20], in order to optimize training by assuming symmetry of the extracted features. The cropped eye images, along with the eye position in the image, are then passed into the NeuraLight Gaze Estimation Model (NLGEM), the general architecture of which is shown in Figure 2.

The NeuraLight Gaze Estimation Model is a deep neural network (DNN) composed of three distinct modules: (1) A Convolutional Layers Module, including two equally weighted networks (for the two eyes), which extracts relevant features from eye images using convolutional layers; (2) Fully connected layers (Eye Position Data Module), which generate features based on eye position data (i.e., this module processes spatial information); (3) Integration layers (Combination Module), which utilize fully connected layers to combine the features extracted from the image (module 1) and spatial information (module 2). The output is mapped into the target PoR co-ordinates for each eye, denoted as X.

The NLGEM model was trained using 109 subjects as a part of a large healthy subject cohort [30] examined in 925 sessions, with 153,119 frame samples. The NLGEM training cohort included 70 males and 39 females, with a mean age of 41 and a standard deviation of 18.5 years and with the IR eye tracker readings as the training labels.

The NLGEM output gaze point co-ordinates of the two eyes are averaged into a single generic gaze estimation PoR, X, represented with two co-ordinates, which are passed along with the target location into the Calibrated Gaze Model (CGM), which predicts the calibrated gaze PoR, Y, as described in Equations (1)–(3), for multiple frames.

The calibrated gaze matrix $\mathit{Y}$ and generic gaze matrix $\mathit{X}$ are given as follows:

$$\mathit{X}=\left[\begin{array}{c}{x}_h\\ x_v\end{array}\right], \mathit{Y}=\left[\begin{array}{c}{y}_h\\ y_v\end{array}\right]$$

(1)

where ${\mathit{x}}_{\mathit{h}}$ and ${\mathit{x}}_{\mathit{v}}$ are the NLGEM gaze estimation vectors in the horizontal and the vertical axes, respectively, and ${\mathit{y}}_{\mathit{h}}$ and ${\mathit{y}}_{\mathit{v}}$ are the CGM calibrated gaze estimation vectors in the horizontal and vertical axes, respectively.

For each of N known stimulus target points, a group of calibrated point matrices ${\mathit{Y}}_{\mathit{i}=\mathbf{1}\dots \mathit{N}}$ and their corresponding NLGEM generic gaze estimation matrices ${\mathit{X}}_{\mathit{i}=\mathbf{1}\dots .\mathit{N}}$ are extracted, and the calibrated gaze vector is approximated by solving a regression problem defined according to the following equations.

$$y_{i,h}=E_h\left({\mathit{X}}_i\right) ; y_{i,v}=E_v\left({\mathit{X}}_i\right)$$

(2)

where ${\mathit{E}}_{\mathit{h}}$ and ${\mathit{E}}_{\mathit{v}}$ are second-order polynomial regression functions [31,32,33] of the following forms:

$$\begin{gathered} E_h\left(\mathit{X}\right)=E_h\left(x_h,x_v\right)=a_0+a_1\cdot x_h+a_2\cdot x_v+a_3\cdot x_h\cdot x_v+a_4\cdot x_h^2+a_5\cdot x_v^2, \\ {E}_v\left(\mathit{X}\right)=E_v\left(x_h,x_v\right)=b_0+b_1\cdot x_h+b_2\cdot x_v+b_3\cdot x_h\cdot x_v+b_4\cdot x_h^2+b_5\cdot x_v^2. \end{gathered}$$

(3)

Hence, for each calibration point ${\mathit{Y}}_{\mathit{i}}$, with corresponding NLGEM generic gaze estimation matrix ${\mathit{X}}_{\mathit{i}}$, the optimal (least error) coefficients ${\mathit{a}}_{\mathbf{0}\dots \mathbf{5} }$ and ${\mathit{b}}_{\mathbf{0}\dots \mathbf{5}}$ are calculated and further used within the test sequences for ongoing calibration within the examination sequence. This procedure is repeatedly performed during the examination, allowing for supervised learning of the coefficients that may vary between individual subjects, physical setup, and environmental conditions.

2.2. Validation Experiment

The NLGEM and the combination of NLGEM and CGM were validated by comparing the video-based gaze detection results with those of a specialized IR eye tracker (Tobii, Sweden [23]). IR eye tracker and video data were collected from 25 healthy subjects, who were randomly selected from a large cohort of healthy subjects [30].

The validation cohort included 259 subjects: 159 females and 100 males aged 19–86 years, with a mean age of 64 years and a standard deviation of 14 years. A total of 128 of the 259 subjects wore corrective glasses during the test (see

Table 1. The validation cohort groups with their corresponding number of participants; age range, mean, and standard deviation; and fraction of participants who wore glasses during the test.

Group	Number	Age [yr]		Glasses Used/N
		(Min, Max)	(Mean, Std)
Male	100	(19.3, 86.3)	(59.2, 16.6)	44/100
Female	159	(19.3, 83.2)	(65.7, 13.0)	84/159
Total	259	(19.3, 86.3)	(64.5, 13.8)	128/259

). During the data collection sessions, the subjects sat in front of a monitor, at an approximate distance of 50–70 cm (see Figure 3), and were instructed to complete a set of tasks that included horizontal and vertical reflexive pro-saccades.

During the pro-saccade task session, the subjects were directed to stare at a red dot when it was presented. The dot appeared at the center of the screen for some (random) time, then immediately disappeared from the screen center and appeared at an angular distance of ~20 degrees or ~13.5 degrees for horizontal or vertical prosaccades, respectively. The dot stayed at its displaced location for 1500 ms and then disappeared. After another 300 ms, where no dot was displayed on the monitor, the central dot reappeared for the next repetition (see Figure 4).

All tasks were performed on a 1920 × 1080 pixel (52.8 cm × 29.7 cm) monitor presenting a uniform grey background, with a 1.38 cm diameter red circle as the gaze target, representing a target whose angular size (from the subject’s location) was $1.35\pm 0.22$ degrees. The tests included six 60-s sessions, each containing several pro-saccade stimuli. A break of 30 s was given between the sessions.

While the test sessions were performed, all video and IR eye tracker data were collected using an Apple Mac Mini computer (Apple, designed in Cupertino, CA, USA; manufactured in Malaysia). The stimulus monitor and the webcam timestamps were synchronized, and the data were stored in cloud storage in compliance with HIPAA restrictions. The analysis and preprocessing of data were conducted on an AWS EC2 compute-optimized instance, and the NLGEM training was run on an AWS EC2 gpu-ml-GPU-ML-optimized instance. The Tobii Pro Fusion eye tracker was operated using the Tobii Pro SDK software version 1.9.0 (https://www.tobii.com/products/software/applications-and-developer-kits/tobii-pro-sdk, Tobii, Danderyd, Sweden). The system captured the IR eye tracker gaze data at a sampling rate of 120 Hz and the video stream was captured at 60 frames per second.

3. Results

A total of 469 test sessions of horizontal and vertical pro-saccades were analyzed and validated, for a total of 14,768,320 sample frames. Simple qualitative time-series analyses, as shown in Figure 5, clearly demonstrate the improvement in gaze detection accuracy when the CGM was utilized on top of the NLGEM. While the saccade timing (step function in time) was distinctive in both video-based retrievals, the NLGEM + CGM output was much closer to that of the reference IR eye tracker.

Further quantitative analyses with reference to the IR eye tracker showed that, while the mean absolute error (MAE) of the estimated gaze with the NLGEM itself was 4.71 degrees, the CGM decreased the MAE to 1.80 degrees. This indicates that adding the CGM improved the accuracy by a factor of 2.5, reducing the MAE by 62%. An interesting and distinctive difference was observed between the accuracy levels for the horizontal and vertical gaze detections. Adding the CGM reduced the MAE in the horizontal axis by 58% and in the vertical axis by 65%, achieving MAE values of 0.93 degrees and 1.31 degrees, respectively (see

Table 2. Mean average error (MAE) calculations (degrees) for the video-based NLGEM and the NLGEM combined with the CGM gaze estimation location compared with the IR eye tracker. The MAE was calculated separately for the horizontal and vertical axes and the absolute error distance (Norm).

Component	MAE NLGEM [deg]	MAE NLGEM + CGM [deg]
Horizontal	2.20	0.93
Vertical	3.70	1.31
Norm	4.71	1.80

). When comparing the MAE distributions of the gaze detection based on the two models, we found a significantly narrower MAE distribution when the CGM was applied after the NLGEM, allowing better confidence when aggregating multiple samples (see Figure 6).

A closer look at the error distributions of the vertical and horizontal gaze locations provided by the video-based gaze estimation models (see Figure 7) revealed that the CGM significantly improved the error distribution symmetry, in addition to the MAE values. While the error distribution of the NLGEM estimated gaze locations seemed to be multimodal and to vary with different PoR locations, after the CGM was utilized, the gaze estimation error was characterized by a narrow modal distribution.

The gaze spatial error was calculated as the distance vector between the estimated gaze location and the reference (i.e., the IR eye tracker, in this case). Figure 8 presents the 2D gaze spatial error distribution when implementing NLGEM alone (Figure 8a) and when adding the CGM calculation (Figure 8b), exposing the spatial features of the gaze errors. The gaze estimations produced using the NLGEM were widely distributed in various directions and were characterized by an amorphous pattern whose local maxima were in random locations. This unstructured, clustered distribution may have resulted from biases depending on the individual subjects; for instance, glass lenses may bias passive imaging sensors but would not affect active sensors, such as the IR eye tracker used as a reference in this study.

Introducing the stimuli target locations and using the CGM in addition to the NLGEM gaze estimations significantly improved the spatial error (Figure 8b). After implementing the CGM, the spatial error pattern was symmetric and centered on the origin, as expected from a calibrated sensor with a normally distributed measurement error.

Additional analysis on the validation cohort results was conducted to estimate the contribution of the CGM to sensitivity to the age of patients, as shown in Figure 9. While the significant advantage of the NLGEM + CGM over the NLGEM was maintained along all patient ages, the MAE for both NLGEM alone and NLGEM + CGM gaze estimation increased with age, at a mean pace of $0.013 \raisebox{1ex}{$deg$}\!\left/ \!\raisebox{-1ex}{$yr$}\right.$.

4. Discussion

In this study, we introduced a video-based gaze detection model, comprising an independent video-based NeuraLight Gaze Estimation Model (NLGEM) and an additional Calibrated Gaze Model (CGM), which uses the target locations of visual stimuli for ongoing real-time calibration during test sessions. The NLGEM model architecture is based on recently published works (Figure 2) [28], with the exception of training the model on labels measured using an IR tracker and the recruitment of a dedicated cohort for the training and test sets. For validation, we tested our gaze detection model on an independent cohort of 259 participants (see Table 1) who performed horizontal and vertical pro-saccade tasks while sitting at a distance of approximately 60 cm from a display, with no head restraint.

Compared to the use of a specialized IR eye tracker as a reference ground truth, we found that the NLGEM alone achieved an absolute visual angle accuracy (MAE) of 4.80 degrees. While such an accuracy level is comparable to that of some of the leading IR eye trackers available at present [23,24] and may be sufficient for applications that require general visual attention direction or saliency distribution on display, most eye tracking applications require higher accuracy, especially in medical applications (e.g., for the measurement of eye movements in patients with various diseases and oculomotor symptoms).

Most abnormal oculometric measures (OMs) that are present in patients are based on measuring eye movements in response to visual stimulation tasks. Therefore, we added the CGM on top of the NLGEM to leverage the information provided by the visual stimuli, namely the target location. When testing the performance of NLGEM and CGM together (NLGEM + CGM), we achieved an absolute gaze detection accuracy (MAE) of 1.80 degrees in the norm, as well as an MAE of 0.93 degrees in the horizontal direction and 1.31 degrees vertically, as detailed in Table 2. When aiming to measure eye movements, the separation into horizontal and vertical directions is required as, in some movements (e.g., saccades), horizontal and vertical movements involve different pathways [11]. Therefore, the accuracy estimations in this work for the horizontal and vertical axes are useful when assessing OMs based on mechanisms that depend on the movement direction.

The performance of the NLGEM + CGM approach was within the systematic errors of the Tobii Pro Fusion eye tracker (0.3 degrees) when taking into consideration the stimuli target size, which was $1.35\pm 0.22 deg$. Given the accuracy level of our reference measurement (Tobii, 0.3 degrees) and the size of the target displayed (~1.35 degrees), we suggest that our gaze detection model accuracy fulfils the requirements of various OM extraction applications, such as measuring oculomotor abnormalities in patients with different diseases, using only a retail-grade webcam and without the need to fix the subject’s head. With the used video sample rate of 60 frames per second, the maximal error in time for a single sample is 16.6 ms, allowing for the confident detection of abnormalities in OMs, whose time scale is ~100 ms and whose visual angle scale is larger than our calculated errors (e.g., the Square Wave Jerks [34] rate, Saccadic Latency, Saccadic Amplitude, Saccadic Error Rate, and so on).

The impact of age on our video-based gaze detection models was estimated according to an increase in the MAE with the age of the patient, presenting an average rate of $0.013 \raisebox{1ex}{$deg$}\!\left/ \!\raisebox{-1ex}{$yr$}\right.$. The increase in the gaze estimation MAE with age was similar for the NLGEM and NLGEM + CGM models, as well as in all direction variances examined. The increasing uncertainty in video-based eye tracking with age can be influenced by the effects of aging on the physiology and structure of the eye [35]; for instance, changes in pupil size, lens elasticity, and retinal function may lead to higher variability in these landmarks in aged populations, leading to a continuously increasing error in video-based gaze estimation.

Recent video-based gaze detection models have presented high levels of accuracy for free-head visual stimulation tasks within the range of 2–5 degrees [36,37,38]. The model introduced in this study yielded better performance under similar experimental settings. Furthermore, unlike the model presented here, all recent studies have mentioned the requirement of a separate dedicated calibration task for each test. The CGM module in our work eliminates the need for this separate calibration task, instead enabling ongoing real-time calibration during the test. The combination of high-accuracy gaze estimation and flexible environmental settings provides new opportunities for advances in the oculometric measurement analysis of patients with various diseases affecting their oculomotor skills, resulting in eye movement abnormalities.

Furthermore, with the capabilities demonstrated in this study, relevant oculometric measures could be extracted and measured easily using a clinical setup, or even remotely in the patient’s home, assuming that they have access to a computer equipped with a web camera. The ease of operating these applications would increase the number of examined patients, extending the training set of our models and, hence, their accuracy. An accurate, affordable, and accessible medical standard OM extraction tool may lead to significant scientific progress and potential for many patients who strive for a better diagnosis and monitoring of their disease progress.