Haptic interface with multimodal tactile sensing and feedback for human–robot interaction

Abstract

Novel sensing and actuation technologies have notably advanced haptic interfaces, paving the way for more immersive user experiences. We introduce a haptic system that transcends traditional pressure-based interfaces by delivering more comprehensive tactile sensations. This system provides an interactive combination of a robotic hand and haptic glove to operate devices within the wireless communication range. Each component is equipped with independent sensors and actuators, enabling real-time mirroring of user’s hand movements and the effective transmission of tactile information. Remarkably, the proposed system has a multimodal feedback mechanism based on both vibration motors and Peltier elements. This mechanism ensures a varied tactile experience encompassing pressure and temperature sensations. The accuracy of tactile feedback is meticulously calibrated according to experimental data, thereby enhancing the reliability of the system and user experience. The Peltier element for temperature feedback allows users to safely experience temperatures similar to those detected by the robotic hand. Potential applications of this system are wide ranging and include operations in hazardous environments and medical interventions. By providing realistic tactile sensations, our haptic system aims to improve both the performance and safety of workers in such critical sectors, thereby highlighting the great potential of advanced haptic technologies.

Introduction

The advancement of haptic interfaces has revolutionized human–computer interaction, bridging the gap between virtual and physical reality regarding tactile sensations [1, 2]. Haptic technology, which measures and conveys physical sensations such as pressure, vibration, and texture, has become increasingly critical since the onset of the coronavirus pandemic. In fact, the pandemic imposed social distancing, highlighting the indispensable role of haptic interfaces in facilitating effective non-face-to-face interactions. Haptic technologies have transcended traditional communication barriers and found essential applications across various fields. For instance, telemedicine allows surgeons to perform procedures remotely [3], while virtual training and learning systems offer immersive hands-on experiences [4]. In hazardous environments, haptic interfaces enable remote operations and minimize risks while ensuring task execution efficiency [5]. In each of these domains, haptic interfaces are essential by providing realistic and rich experiences and effectively bridging the sensory gap created by physical distance.

Despite the significant progress in tactile sensing and feedback, this field has primarily focused on the development of isolated components. They include tactile sensors known for their flexibility, high sensitivity, and low power consumption [6,7,8,9] as well as compact high-performance actuators designed for realistic tactile conveyance [10,11,12,13]. However, integrating these components into a cohesive and interactive communication system capable of transmitting real-time sensory information from sensors to users remains challenging in haptic technology. This challenge has been addressed in recent studies [14,15,16]. For example, Pyo et al. developed a haptic system linking fabric-based sensors with linear actuator-based tactile displays [14]. In their system, the pressure sensed by a tactile glove composed of fabric-based sensors is translated into pressure feedback for another user through an actuator-integrated platform. Similarly, Fang et al. developed a haptic system that integrates a highly sensitive hydrogel sensor with a vibrator, suitably detecting and conveying object textures and shapes [16]. The abovementioned systems primarily focus on pressure feedback and often neglect temperature simulation, which is required to elicit comprehensive tactile experiences. This limitation has been further highlighted by recent efforts to integrate temperature feedback, such as in the work by Sun et al., who used NiCr heaters in a multisensory haptic system to deliver temperature sensations along with pressure feedback [17]. Their system was restricted to simulating temperatures below ambient levels, highlighting the need for a more versatile solution that can accurately measure and convey a wider range of tactile stimuli.

To address current challenges, we propose an advanced haptic system. Our system not only integrates advanced tactile sensors and actuators for real-time feedback but also emphasizes temperature simulation alongside force feedback, thus offering a more holistic and realistic tactile experience. The proposed advanced haptic system integrates a robotic hand with a haptic glove, which are meticulously designed to sense and convey intricate tactile stimuli. The robotic hand is driven by servomotors, which are calibrated to replicate the subtle and complex movements of the human hand with high fidelity. Microsensors are incorporated within the robotic hand to detect minute variations in both pressure and temperature. The sensors quickly convert physical interactions into digital signals to provide real-time feedback. A key feature of our system is the haptic glove, which is equipped with a versatile array of flex sensors and two types of actuators. The actuators include eccentric rotating mass (ERM) vibrators and Peltier elements, which are carefully chosen to elicit a wide range of tactile sensations. These components are strategically selected to simulate a broad spectrum of tactile sensations, creating a dynamic and responsive feedback loop finely adjusted to the user movements. Extensive testing and optimization endowed the proposed system with high efficiency and responsiveness in tactile feedback. Our system properly transmits complex tactile information that allows to distinguish between various forces and temperature ranges in real time.

Concept and system design

As depicted in Fig. 1a, the proposed haptic interface integrates two primary components, a robotic hand and a haptic glove, each designed to sense and transmit tactile sensations of force and temperature. This integration enables a comprehensive haptic interaction, which is essential for eliciting realistic tactile experiences. At the core of the system, the robotic hand is driven by precision-engineered servomotors, which can mimic intricate movements of the human hand. This functionality is not limited to motion replication because the hand is equipped with sensors that actively capture tactile data in real time. These sensors detect force and temperature and are crucial for generating the necessary feedback in the haptic loop. Combined with the robotic hand, the haptic glove includes flex sensors that are positioned to accurately detect and transmit the user’s hand movements. The glove is tailored to comfortably fit the five fingers and both capture user movements and convey tactile sensations to the user. This bidirectional information flow between the glove and robotic hand is pivotal for establishing an interactive haptic system.

Fig. 1

Comprehensive view of the proposed haptic interface system. a Integrated haptic interface system presenting the detailed configuration of the robotic hand and the haptic glove. b Photographs of the fully assembled robotic hand and haptic glove with their individual sensors and actuators

Figure 1b shows photographs of the fabricated robotic hand and haptic glove, depicting their structural design and integration of sensors and actuators. The robotic hand was fabricated using a 3D printer (M220, Moment), and each finger houses individual force and temperature sensors, enabling detailed and independent tactile perception. The sensor data are wirelessly relayed to the haptic glove, which establishes a multimodal actuation system consisting of ERM vibrators and Peltier elements. These actuators simulate force and temperature sensations and are finely tuned to replicate the detected stimuli with high fidelity, thereby providing the user with a realistic and nuanced tactile experience.

Replication of human hand movements in robotic systems

Figure 2 shows the control mechanism of the robotic hand in the proposed haptic system. Five servomotors (MG996R, SMG) are central to this mechanism, with each motor controlling the movement of a finger (Fig. 2a). The motors are connected to a wire system designed for precise finger actuation. Upon activation, the servomotors generate a pulling force on the wires coiled around the rotating frame of the motor. This design allows to emulate a diverse range of human finger movements, from basic gestures to complex grasping actions. Flex sensors (SEN-08606, SparkFun Electronics) embedded in the haptic glove are positioned on the back of the user’s hand. These sensors are highly sensitive to different degrees of finger bending. A hand movement results in proportional changes in the sensor electrical resistance, which is then converted into an analog signal. The signals from all the sensors are wirelessly transmitted to the robotic hand to precisely guide the actuation of the servomotor and allow the robotic hand to mirror the user’s movements in real time.

Fig. 2

Control mechanism of the robotic hand in the haptic system. a Configuration for replication of human hand movements. b Correlation between the flex sensor resistance values and the corresponding servomotor driving angles, demonstrating the adaptive response to various finger lengths and motions. c Successful emulation of various human hand gestures by the robotic hand, exemplifying the real-time gesture replication

During design, we carefully considered the variability in human hand movements. The robotic hand underwent meticulous calibration to mirror diverse ranges of motion and adapt to different finger lengths. Figure 2b shows the correlation between the resistance measured by the flex sensors as analog signals and the corresponding servomotor driving angles when a user bent and stretched the fingers. The sensors exhibited various responses depending on the finger length, with longer fingers, such as the middle one, showing more notable changes than shorter fingers, such as the thumb. The system compensates for initial value differences resulting from sensor variance and manufacturing-induced bending by calibrating the servomotor angle changes according to the flex sensor data, ensuring the realistic replication of hand movements. Figure 2c shows a demonstration of the control system with the robotic hand replicating various human hand gestures.

Force detection and feedback

The force feedback loop between the robotic hand and haptic glove is illustrated in Fig. 3a. Force sensors (RA12P, Marveldex Inc.) are integrated within the fingers of the robotic hand, each outfitted with a bump to improve grip and prevent slippage during various interactions. The sensor data are transmitted to the haptic glove and used to adjust the intensity of the vibration motor (ELB060416, YwRobot). This adjustment is essential to effectively convey force sensations to the user. Figure 3b details the quantification of force sensor measurements. A customized compression input tester was utilized to apply force, and the magnitude of force was measured using a force gauge (DGT-1000N, DIGTECH). In the figure, different forces are applied to the sensor for the middle finger, and the sensor output is recorded as an analog signal. These analog values correspond to the changes in electrical resistance of the force sensor when subjected to external forces. Notably, as the force increases, the rate of increase in the analog output of the sensor decreases. To simplify this nonlinear response, we represent it in two linear graphs. Figure 3c illustrates the simplified sensor response, demonstrating a stable output for an applied force ranging from 0.1 to 20 N. The response confirms the sensor reliability under varying forces.

Fig. 3

Detailed overview of the force sensing and feedback system. a Illustration of the force sensor integration within the fingers of the robotic hand and their wireless connection to the haptic glove for vibration motor intensity adjustment. b Response curve of the force sensor, showing the diminishing rate of increase in response to increasing force, simplified using two linear graphs. c Stability of the sensor output across a force range of 0.1 N to 20 N. d Correlation between resistance changes in the force sensor and the increasing duty ratio of the vibration motor. e Force required to produce specific resistance changes in the force sensor, used to calibrate the force output of the vibration motor. f Graph demonstrating the consistency of resistance changes in the force sensor across multiple cycles with varying vibration motor duty ratios

Adequate force feedback relies on the synchronization of the intensity of the vibration motor and the analog output from the force sensor. Synchronization allows the user to perceive a force comparable to that measured by the robotic hand. For synchronization, the motor performance was assessed by measuring resistance changes in an additional force sensor (SEN-09376, SparkFun Electronics) in response to the force generated by the vibration motor. This indirect measurement was used to determine the corresponding force at each level of motor intensity. Figure 3d shows the relation between the resistance changes in the force sensor and duty ratio of the vibration motor. The vibration motor in our system operates at a fixed frequency, and the intensity of vibrations is modulated using pulse-width modulation control. This modulation allows us to adjust the duty cycle of the vibrations, effectively controlling the intensity of the tactile sensations experienced by the user. An increase in the motor intensity correlates directly with increasing resistance in the force sensor. Figure 3e shows the force required to induce a resistance change in the force sensor. Matching the results shown in Fig. 3d and e, the vibration motor can be adjusted to exert a target force on the user. For example, a motor can exert a force of 8 N at a 60% duty ratio. Figure 3f shows consistent changes in the resistance of the force sensor with varying duty ratios of the vibration motor across multiple cycles, validating the accuracy and reliability of the system for delivering force feedback.

Temperature detection and feedback

The proposed haptic system implements temperature control through the interaction of temperature sensors (TS0213, SunFounder) in the robotic hand with Peltier elements (TES1-4903, SMG) in the glove, as illustrated in Fig. 4a. For accurate temperature measurement, a negative temperature coefficient (NTC) thermistor is employed because of its consistent performance across common temperature ranges. A linear response of the sensor to temperature changes can enable precise temperature control. To evaluate the response, the Peltier element was brought in contact with the sensor, enabling controlled temperature variation and resistance measurements. An infrared camera (One Pro LT, FLIR) measured the temperature, confirming a linear decrease in the resistance of the NTC sensor with increasing temperature (Fig. 4b).

Fig. 4

Exploration of the temperature detection and feedback. a Schematic showing the interplay between the temperature sensor in the robotic hand and the Peltier element in the glove for precise temperature modulation. b Linear response of the NTC thermistor to temperature changes. c Visualization demonstrating the ability of the Peltier element to reach specific high and low temperatures. d Control strategy for the Peltier element, including adjustments to current flow duration and off periods. The result demonstrates that a wide range of temperatures is realized linearly, from approximately 30 °C to 80 °C. e Thermal imaging results displaying the effective implementation of segmented temperature control in the glove

The temperature data collected by the NTC sensor are converted into digital signals and transmitted to the haptic glove. These temperature signals are essential for eliciting thermal sensations. In previous studies on haptic interfaces [17], NiCr heaters were adopted owing to their stable operation at high temperatures, but they are less efficient at lower temperatures and show reduced accuracy at extreme temperatures. Therefore, we integrate Peltier elements into the proposed haptic system for heating or cooling according to the direction of current flow. The current applied to the Peltier elements is controlled using a motor driver and monitored with a thermal imaging camera to ensure accurate temperature adjustments. Figure 4c shows the capacity of the Peltier element to achieve specific high and low temperatures. However, the prolonged use of Peltier elements may be problematic. For instance, one side may overheat, causing an overall temperature increase in the element. To address this problem, we carefully control the operation time of the Peltier element. Lengthening the current flow duration increases the temperature, while incorporating a 2.5 s period without current prevents rapid temperature increase. This controlled approach allows achieving specific temperature levels. Figure 4d shows that different temperatures can be achieved by adjusting the current off time and operation time, allowing linear changes in the saturation point temperature, approximately ranging from 30 to 80 °C. Figure 4e shows the successful implementation of segmented temperature control at various target temperatures captured with a thermal imaging camera. Despite the time required for temperature saturation to achieve higher temperatures, this approach ensures that users feel adequate heat while minimizing the risks of overheating or burns owing to the extended use of Peltier elements.

Interaction between robotic hand and haptic glove

Figure 5a demonstrates that the robotic hand can replicate various human hand gestures, including complex gestures such as the V sign (Additional file 1: Video S1). This replication was accomplished by detecting the movements of each of the five human fingers from the glove and transmitting the corresponding motion commands to the individual servomotors of the robotic hand. This ensured that the robotic hand accurately mirrored the movements of each human finger. Nuances of finger movement and corresponding signal translation are shown in Fig. 5b, which depicts the changes in sensor signals corresponding to the bending of individual fingers. Notably, when a finger was fully bent, the sensors produced normalized values that varied according to the finger length, aligning with the explanations depicted in Fig. 2b. The system experienced a brief delay of about 0.6 s, primarily owing to the communication lag between the glove and robotic hand as well as the time required for the motors to move between the initial and final angles. Figure 5c shows the force feedback mechanism resulting from the interaction between the glove and robotic hand. The signals from the force sensors were mapped onto the vibration motors within the glove. This mapping integrated the resistance change data shown in Fig. 2d and was refined using the force–resistance equation depicted in Fig. 2e. This method allowed the conversion of measured force values into equivalent force feedback delivered to the user through the vibration motors in the glove. As shown in Fig. 5d, the temperature feedback accuracy was verified because the temperature measurements at each finger of the robotic hand were relayed to the haptic glove (Additional file 2: Video S2). Experiments involving simultaneous temperature assessments using an infrared camera showed that the temperatures achieved in the Peltier elements on the glove closely matched those recorded by the robotic hand sensors.

Fig. 5

Robotic hand gesture replication and haptic feedback. a Replication of human hand gestures with the robotic hand, showcasing its ability to mimic complex movements. b Analysis of sensor signal changes in the haptic glove corresponding to finger bending, highlighting variations for different finger lengths. c Mapping force feedback from force sensor to vibration motor. d Validation of temperature feedback accuracy by comparing temperature measurements from the thermistor with the Peltier element

As shown in Fig. 6, a comprehensive experiment demonstrated the ability of the robotic hand to discern the pressure and temperature of each finger and provide corresponding feedback through the haptic glove (Additional file 3: Video S3). Figure 6a shows the force sensor and vibration motor signals under varying force applied to the index and ring fingers of the robotic hand. The force sensor signals and vibration motor responses confirmed real-time and responsive interactions. Additionally, a greater response was observed from the sensor and motor of the ring finger owing to the larger applied force. Figure 6b shows the response to temperature stimuli, showing signal variations when a cold pack and hand warmer were applied to the index and ring fingers, respectively, resulting in the cooling and heating of the Peltier elements in the corresponding glove fingers. Finally, Fig. 6c shows the force and temperature feedback during contact of the robotic finger with an object. Different signal responses were observed for each finger, reflecting the varying degrees of contact with the object. The nuanced interactions between the robotic hand and haptic glove led to successfully deliver real-time tactile information of force and temperature. Such advanced interactions can immerse users in a realistic sensory experience, effectively bridging the digital and physical realms of touch sensations.

Fig. 6

Real-time tactile feedback analysis. a Force and vibration response, demonstrating the real-time feedback capability by analyzing force sensor and vibration motor signals when varying forces are applied to the index and ring fingers. b Temperature sensation response showing signal variations when the cold pack and hand warmer are applied to the index and ring fingers, resulting in cooling and heating of the corresponding Peltier elements in the glove. c Nuanced signal responses from different fingers reflecting varying contact degrees during the contact of the robotic finger with an object

Conclusion

We demonstrate an advanced haptic system that offers users a more realistic tactile experience than traditional haptic technologies primarily focused on pressure feedback. The proposed system integrates a robotic hand with a haptic glove and facilitates interactive and remote operations within the range of wireless communication. A key innovation of this system is its multimodal feedback mechanism, which employs both vibration motors and Peltier elements to accurately elicit pressure and temperature sensations. The Peltier elements are correctly controlled to provide a suitable temperature feedback. Hence, the transmission of temperature is safe, enabling users to perceive temperatures similar to those sensed by the robotic hand once equilibrium is reached. Simultaneously, pressure feedback accurately conveys forces to the user based on predefined values. Each sensor and actuator operate independently, allowing the robotic hand to precisely mimic the movements of the user, who can sense tactile information. The tactile information transmitted to the user's fingers via the glove creates a vivid illusion of direct contact with touched objects. The versatility of the proposed system may be conducive to applicability in various fields, particularly where manual operation is challenging or risky, such as in hazardous environments and medical interventions. By eliciting realistic sensations, the proposed system may substantially enhance the efficiency and safety of operators in such scenarios.