Stereoscopic facial imaging for pain assessment using rotational offset microlens arrays based structured illumination

Conventional pain assessment methods such as patients’ self‐reporting restrict the possibility of easy pain monitoring while pain serves as an important role in clinical practice. Here we report a pain assessment method via 3D face read‐ ing camera assisted by dot pattern illumination. The face reading camera module (FRCM) consists of a stereo camera and a dot projector, which allow the quantitative measurement of facial expression changes without human subjec‐ tive judgement. The rotational offset microlens arrays (roMLAs) in the dot projector form a uniform dense dot pattern on a human face. The dot projection facilitates evaluating three‐dimensional change of facial expression by improving 3D reconstruction results of non‐textured facial surfaces. In addition, the FRCM provides consistent pain rating from 3D data, regardless of head movement. This pain assessment method can provide a new guideline for precise, real‐ time, and continuous pain monitoring.


Introduction
Clinical procedure requires quantitative and precise assessment of pain because the degree of pain is the treatment result of chronic diseases [1] or non-verbal monitoring in patients [2]. Repeated pain monitoring has poor sustainability due to the heavy workload of hospital nurses, whereas regular pain evaluation improves treatment outcomes [3]. Pain rating methods such as Numerical Rating Scale (NRS) and Visual Analog Scale (VAS) often require self-report or regular visits of a medical team, leading to a limited application range [4][5][6][7]. The Wong-Baker Faces Pain Rating Scale (WBS) is widely utilized for facial expression-based pain assessment for non-communicative patients such as children or people with intellectual disabilities [8]. The WBS estimates the pain intensity by comparing reference images and patients' facial expression. However, conventional methods involve long assessment time, confined applications, or inadequateness of continuous monitoring [9].
Facial Action Coding System (FACS) proposed in 1978 defines the specific facial muscle movements as action units (AUs) classifying the change of muscle movements by facial expression. A well-trained expert directly encodes the facial expression by looking at still images or video of a person [10]. The Prkachin and Solomon pain intensity (PSPI) utilizes several AUs associated with pain, serving as the ground truth of pain intensity [11,12]. Ambiguous and subtle movements of a human face are subjectively classified by human raters for pain assessment using AUs [13]. Several previous works have also demonstrated the 2D facial expression of pain because the facial expression change is closely related to the pain intensity [9,14]. However, the environment disturbance such as lighting, makeup, or eyebrow-free hinders the precise assessment due to 2D information of human faces [6,15,16]. Three-dimensional facial image acquisition further improves the pain assessment but still remains Open Access  [17,18]. As an alternative, bioelectrical signals such as electrocardiograph and electromyograph contribute to the pain monitoring but they still require the clear explanation of the relationship between bioelectrical signals and pain, and overcome high invasiveness [3,19,20]. The distance between a stereo camera and a subject is often estimated by the binocular disparity [21] while non-textured surfaces suchas a human face lower in 3D reconstruction quality [22]. Recently. structured illumination driven stereo camera substantially improves the binocular disparity for a non-textured sample [23].
Here we report a 3D face reading camera module (FRCM) for pain rating (Fig. 1). The FRCM consists of a stereo camera and a dot projector to read facialexpressions in three dimensions for numerical analysis of pain. The dot projector contains rotationaloffsetmicrolensarrays (roMLAs) as a diffractive optical element (DOE), which allows structured illumination for compact stereoscopic imaging system [24,25].The roMLAs provide dense dot patterns of high uniformity and high contrast on a human face and efficientlyimproves the precise 3D reconstruction for quantifying small facial expression changes as well as reducing the influence of head movements under pain.

Results and discussion
The FRCM involves a stereo camera (oCamS-1CGN-U, Withrobot, Korea) for obtaining image disparity and a dot projector for enhancing image quality (Fig. 2a, b). The dot projector contains a laser diode (L785P090, Thorlabs, U.S.), a collimating lens (A390TM-B, Thorlabs, U.S.), and roMLAs as a DOE (Fig. 2b). Near infrared light (785 nm) from the laser diode makes the dot pattern invisible to the human eye for preventing inconvenience during pain assessment. The FRCM obtains clear images within 50-100 cm distance range, and the illumination intensity at 50 cm distance from dot projector is 10µW/cm 2 . The dot projector meets the safety level of Laser class 1 defined by International Electrotechnical Commission (IEC 60825-1 Ed. 3.0), which ensures the safety of eyes and skin under dot projection.The roMLAs were microfabricated by forming two hexagonal microlens arrays with rotational offset angle on both sides of a glass wafer (Fig. 2c) [24]. The microlenses of roMLAs show high lens curvature and high fill-factor to obtain a large field-of-view for diffraction. In particular, the rotational offset of microlens arrays is set to 13.25° for creating dot patterns in hexagonal arrangement with high contrast and density. Figure 2d shows scanning electron micrographs of the one side of the fabricated DOE.
The 3D reconstruction of a human face is substantially improved by using roMLAs driven structured light illumination (Fig. 3). The non-textured surface of a target subject often accompanies distorted or void portions in the reconstructed image due to information errors on the stereo matching algorithm. The dot projection of the FRCM facilitates accurate depth estimation for realizing the facial expression change by adding the texture on subject's skin. The FRCM precisely estimates the feature depth within a fraction of one millimeter.  The numerical pain assessment was performed by using the Mahalanobis distance of geometric features related to pain. Figure 4a explains how the Mahalanobis distances appear in a two-dimensional space. This function fits for quantitative pain assessment because the weight of each variable should be set differently depending on its importance. Several distance features were selected for the pain intensity metric to measure the change of facial expression under pain (Fig. 4b). The distance features are designed to reflect facial expression with a significant pain relationship such as brow lowering and upper lip raising [9,18]. The weight factor of each feature was set to a larger value as the number of corresponding AUs related to pain increases [11].
The pain rating was performed while rotating a plaster cast (Bust of Marcus Vipsanius Agrippa), whose head size is similar to a real person, in order to verify tolerance to head motion. The pain intensities obtained without depth information were simultaneously calculated to show the rotational effect of human head. The pain rating using facial features extracted by the 2D images of significantly depends on the rotation angle while facial expression of the cast is relatively constant. The experimental results clearly indicate that pain intensities from 3D images are relatively consistent unlike those from 2D images (Fig. 5a). The substantial reduction of motion artifacts is very important for the quantitative and precise pain [26]. The evaluation of facial expression change was successfully demonstrated using the FRCM. A volunteer acts three different facial expressions representing neutral, moderate pain, and strong pains. The facial expression for strong pain shows a high value of the pain intensity (Fig. 5b), whose number is more continuous unlike conventional methods such as NRS and WBS indicating the discrete value of pain. Besides, human subjective judgement is not involved in pain assessment process using the FRCM.

Conclusions
The dot pattern illumination based on roMLAs plays an important role for accurate 3D reconstruction of human facial expression. The numerical pain analysis has been successfully performed by using the 3D geometric features extracted by the FRCM. The FRCM achieves consistent pain intensities regardless of face direction due to 3D face reading. The facial image acquisition, feature extraction, and analysis were separately performed in the process of pain assessment. This procedure can be further performed in real-time by employing the detection algorithm for facial landmarks. The roMLA driven structured illumination opens up the potential for precise 3D imaging in order to improve an automatic toolkit for facial expression thanks to the simple and compact implementation. The experimental result clearly implies that 3D facial imaging based on dot projection overcomes the weaknesses of conventional pain assessment such as motion artifact, long assessment time, subjective decision. This new facial reading camera can provide a new direction for facile and real-time pain monitoring for advanced clinical applications. Fig. 4 a Contour lines of theMahalanobis distance in 2D space. b Distance features for pain rating. x 1 , x 2 : from eyebrow to mouth;x 3 , x 4 : from eyebrow to eye; x 5 : mouth height; x 6 : mouth width;x 7 , x 8 : from eye to mouth