- Letter
- Open access
- Published:
Haptic interface with multimodal tactile sensing and feedback for human–robot interaction
Micro and Nano Systems Letters volume 12, Article number: 9 (2024)
Abstract
Novel sensing and actuation technologies have notably advanced haptic interfaces, paving the way for more immersive user experiences. We introduce a haptic system that transcends traditional pressure-based interfaces by delivering more comprehensive tactile sensations. This system provides an interactive combination of a robotic hand and haptic glove to operate devices within the wireless communication range. Each component is equipped with independent sensors and actuators, enabling real-time mirroring of user’s hand movements and the effective transmission of tactile information. Remarkably, the proposed system has a multimodal feedback mechanism based on both vibration motors and Peltier elements. This mechanism ensures a varied tactile experience encompassing pressure and temperature sensations. The accuracy of tactile feedback is meticulously calibrated according to experimental data, thereby enhancing the reliability of the system and user experience. The Peltier element for temperature feedback allows users to safely experience temperatures similar to those detected by the robotic hand. Potential applications of this system are wide ranging and include operations in hazardous environments and medical interventions. By providing realistic tactile sensations, our haptic system aims to improve both the performance and safety of workers in such critical sectors, thereby highlighting the great potential of advanced haptic technologies.
Introduction
The advancement of haptic interfaces has revolutionized human–computer interaction, bridging the gap between virtual and physical reality regarding tactile sensations [1, 2]. Haptic technology, which measures and conveys physical sensations such as pressure, vibration, and texture, has become increasingly critical since the onset of the coronavirus pandemic. In fact, the pandemic imposed social distancing, highlighting the indispensable role of haptic interfaces in facilitating effective non-face-to-face interactions. Haptic technologies have transcended traditional communication barriers and found essential applications across various fields. For instance, telemedicine allows surgeons to perform procedures remotely [3], while virtual training and learning systems offer immersive hands-on experiences [4]. In hazardous environments, haptic interfaces enable remote operations and minimize risks while ensuring task execution efficiency [5]. In each of these domains, haptic interfaces are essential by providing realistic and rich experiences and effectively bridging the sensory gap created by physical distance.
Despite the significant progress in tactile sensing and feedback, this field has primarily focused on the development of isolated components. They include tactile sensors known for their flexibility, high sensitivity, and low power consumption [6,7,8,9] as well as compact high-performance actuators designed for realistic tactile conveyance [10,11,12,13]. However, integrating these components into a cohesive and interactive communication system capable of transmitting real-time sensory information from sensors to users remains challenging in haptic technology. This challenge has been addressed in recent studies [14,15,16]. For example, Pyo et al. developed a haptic system linking fabric-based sensors with linear actuator-based tactile displays [14]. In their system, the pressure sensed by a tactile glove composed of fabric-based sensors is translated into pressure feedback for another user through an actuator-integrated platform. Similarly, Fang et al. developed a haptic system that integrates a highly sensitive hydrogel sensor with a vibrator, suitably detecting and conveying object textures and shapes [16]. The abovementioned systems primarily focus on pressure feedback and often neglect temperature simulation, which is required to elicit comprehensive tactile experiences. This limitation has been further highlighted by recent efforts to integrate temperature feedback, such as in the work by Sun et al., who used NiCr heaters in a multisensory haptic system to deliver temperature sensations along with pressure feedback [17]. Their system was restricted to simulating temperatures below ambient levels, highlighting the need for a more versatile solution that can accurately measure and convey a wider range of tactile stimuli.
To address current challenges, we propose an advanced haptic system. Our system not only integrates advanced tactile sensors and actuators for real-time feedback but also emphasizes temperature simulation alongside force feedback, thus offering a more holistic and realistic tactile experience. The proposed advanced haptic system integrates a robotic hand with a haptic glove, which are meticulously designed to sense and convey intricate tactile stimuli. The robotic hand is driven by servomotors, which are calibrated to replicate the subtle and complex movements of the human hand with high fidelity. Microsensors are incorporated within the robotic hand to detect minute variations in both pressure and temperature. The sensors quickly convert physical interactions into digital signals to provide real-time feedback. A key feature of our system is the haptic glove, which is equipped with a versatile array of flex sensors and two types of actuators. The actuators include eccentric rotating mass (ERM) vibrators and Peltier elements, which are carefully chosen to elicit a wide range of tactile sensations. These components are strategically selected to simulate a broad spectrum of tactile sensations, creating a dynamic and responsive feedback loop finely adjusted to the user movements. Extensive testing and optimization endowed the proposed system with high efficiency and responsiveness in tactile feedback. Our system properly transmits complex tactile information that allows to distinguish between various forces and temperature ranges in real time.
Concept and system design
As depicted in Fig. 1a, the proposed haptic interface integrates two primary components, a robotic hand and a haptic glove, each designed to sense and transmit tactile sensations of force and temperature. This integration enables a comprehensive haptic interaction, which is essential for eliciting realistic tactile experiences. At the core of the system, the robotic hand is driven by precision-engineered servomotors, which can mimic intricate movements of the human hand. This functionality is not limited to motion replication because the hand is equipped with sensors that actively capture tactile data in real time. These sensors detect force and temperature and are crucial for generating the necessary feedback in the haptic loop. Combined with the robotic hand, the haptic glove includes flex sensors that are positioned to accurately detect and transmit the user’s hand movements. The glove is tailored to comfortably fit the five fingers and both capture user movements and convey tactile sensations to the user. This bidirectional information flow between the glove and robotic hand is pivotal for establishing an interactive haptic system.
Figure 1b shows photographs of the fabricated robotic hand and haptic glove, depicting their structural design and integration of sensors and actuators. The robotic hand was fabricated using a 3D printer (M220, Moment), and each finger houses individual force and temperature sensors, enabling detailed and independent tactile perception. The sensor data are wirelessly relayed to the haptic glove, which establishes a multimodal actuation system consisting of ERM vibrators and Peltier elements. These actuators simulate force and temperature sensations and are finely tuned to replicate the detected stimuli with high fidelity, thereby providing the user with a realistic and nuanced tactile experience.
Replication of human hand movements in robotic systems
Figure 2 shows the control mechanism of the robotic hand in the proposed haptic system. Five servomotors (MG996R, SMG) are central to this mechanism, with each motor controlling the movement of a finger (Fig. 2a). The motors are connected to a wire system designed for precise finger actuation. Upon activation, the servomotors generate a pulling force on the wires coiled around the rotating frame of the motor. This design allows to emulate a diverse range of human finger movements, from basic gestures to complex grasping actions. Flex sensors (SEN-08606, SparkFun Electronics) embedded in the haptic glove are positioned on the back of the user’s hand. These sensors are highly sensitive to different degrees of finger bending. A hand movement results in proportional changes in the sensor electrical resistance, which is then converted into an analog signal. The signals from all the sensors are wirelessly transmitted to the robotic hand to precisely guide the actuation of the servomotor and allow the robotic hand to mirror the user’s movements in real time.
During design, we carefully considered the variability in human hand movements. The robotic hand underwent meticulous calibration to mirror diverse ranges of motion and adapt to different finger lengths. Figure 2b shows the correlation between the resistance measured by the flex sensors as analog signals and the corresponding servomotor driving angles when a user bent and stretched the fingers. The sensors exhibited various responses depending on the finger length, with longer fingers, such as the middle one, showing more notable changes than shorter fingers, such as the thumb. The system compensates for initial value differences resulting from sensor variance and manufacturing-induced bending by calibrating the servomotor angle changes according to the flex sensor data, ensuring the realistic replication of hand movements. Figure 2c shows a demonstration of the control system with the robotic hand replicating various human hand gestures.
Force detection and feedback
The force feedback loop between the robotic hand and haptic glove is illustrated in Fig. 3a. Force sensors (RA12P, Marveldex Inc.) are integrated within the fingers of the robotic hand, each outfitted with a bump to improve grip and prevent slippage during various interactions. The sensor data are transmitted to the haptic glove and used to adjust the intensity of the vibration motor (ELB060416, YwRobot). This adjustment is essential to effectively convey force sensations to the user. Figure 3b details the quantification of force sensor measurements. A customized compression input tester was utilized to apply force, and the magnitude of force was measured using a force gauge (DGT-1000N, DIGTECH). In the figure, different forces are applied to the sensor for the middle finger, and the sensor output is recorded as an analog signal. These analog values correspond to the changes in electrical resistance of the force sensor when subjected to external forces. Notably, as the force increases, the rate of increase in the analog output of the sensor decreases. To simplify this nonlinear response, we represent it in two linear graphs. Figure 3c illustrates the simplified sensor response, demonstrating a stable output for an applied force ranging from 0.1 to 20 N. The response confirms the sensor reliability under varying forces.
Adequate force feedback relies on the synchronization of the intensity of the vibration motor and the analog output from the force sensor. Synchronization allows the user to perceive a force comparable to that measured by the robotic hand. For synchronization, the motor performance was assessed by measuring resistance changes in an additional force sensor (SEN-09376, SparkFun Electronics) in response to the force generated by the vibration motor. This indirect measurement was used to determine the corresponding force at each level of motor intensity. Figure 3d shows the relation between the resistance changes in the force sensor and duty ratio of the vibration motor. The vibration motor in our system operates at a fixed frequency, and the intensity of vibrations is modulated using pulse-width modulation control. This modulation allows us to adjust the duty cycle of the vibrations, effectively controlling the intensity of the tactile sensations experienced by the user. An increase in the motor intensity correlates directly with increasing resistance in the force sensor. Figure 3e shows the force required to induce a resistance change in the force sensor. Matching the results shown in Fig. 3d and e, the vibration motor can be adjusted to exert a target force on the user. For example, a motor can exert a force of 8 N at a 60% duty ratio. Figure 3f shows consistent changes in the resistance of the force sensor with varying duty ratios of the vibration motor across multiple cycles, validating the accuracy and reliability of the system for delivering force feedback.
Temperature detection and feedback
The proposed haptic system implements temperature control through the interaction of temperature sensors (TS0213, SunFounder) in the robotic hand with Peltier elements (TES1-4903, SMG) in the glove, as illustrated in Fig. 4a. For accurate temperature measurement, a negative temperature coefficient (NTC) thermistor is employed because of its consistent performance across common temperature ranges. A linear response of the sensor to temperature changes can enable precise temperature control. To evaluate the response, the Peltier element was brought in contact with the sensor, enabling controlled temperature variation and resistance measurements. An infrared camera (One Pro LT, FLIR) measured the temperature, confirming a linear decrease in the resistance of the NTC sensor with increasing temperature (Fig. 4b).
The temperature data collected by the NTC sensor are converted into digital signals and transmitted to the haptic glove. These temperature signals are essential for eliciting thermal sensations. In previous studies on haptic interfaces [17], NiCr heaters were adopted owing to their stable operation at high temperatures, but they are less efficient at lower temperatures and show reduced accuracy at extreme temperatures. Therefore, we integrate Peltier elements into the proposed haptic system for heating or cooling according to the direction of current flow. The current applied to the Peltier elements is controlled using a motor driver and monitored with a thermal imaging camera to ensure accurate temperature adjustments. Figure 4c shows the capacity of the Peltier element to achieve specific high and low temperatures. However, the prolonged use of Peltier elements may be problematic. For instance, one side may overheat, causing an overall temperature increase in the element. To address this problem, we carefully control the operation time of the Peltier element. Lengthening the current flow duration increases the temperature, while incorporating a 2.5 s period without current prevents rapid temperature increase. This controlled approach allows achieving specific temperature levels. Figure 4d shows that different temperatures can be achieved by adjusting the current off time and operation time, allowing linear changes in the saturation point temperature, approximately ranging from 30 to 80 °C. Figure 4e shows the successful implementation of segmented temperature control at various target temperatures captured with a thermal imaging camera. Despite the time required for temperature saturation to achieve higher temperatures, this approach ensures that users feel adequate heat while minimizing the risks of overheating or burns owing to the extended use of Peltier elements.
Interaction between robotic hand and haptic glove
Figure 5a demonstrates that the robotic hand can replicate various human hand gestures, including complex gestures such as the V sign (Additional file 1: Video S1). This replication was accomplished by detecting the movements of each of the five human fingers from the glove and transmitting the corresponding motion commands to the individual servomotors of the robotic hand. This ensured that the robotic hand accurately mirrored the movements of each human finger. Nuances of finger movement and corresponding signal translation are shown in Fig. 5b, which depicts the changes in sensor signals corresponding to the bending of individual fingers. Notably, when a finger was fully bent, the sensors produced normalized values that varied according to the finger length, aligning with the explanations depicted in Fig. 2b. The system experienced a brief delay of about 0.6 s, primarily owing to the communication lag between the glove and robotic hand as well as the time required for the motors to move between the initial and final angles. Figure 5c shows the force feedback mechanism resulting from the interaction between the glove and robotic hand. The signals from the force sensors were mapped onto the vibration motors within the glove. This mapping integrated the resistance change data shown in Fig. 2d and was refined using the force–resistance equation depicted in Fig. 2e. This method allowed the conversion of measured force values into equivalent force feedback delivered to the user through the vibration motors in the glove. As shown in Fig. 5d, the temperature feedback accuracy was verified because the temperature measurements at each finger of the robotic hand were relayed to the haptic glove (Additional file 2: Video S2). Experiments involving simultaneous temperature assessments using an infrared camera showed that the temperatures achieved in the Peltier elements on the glove closely matched those recorded by the robotic hand sensors.
As shown in Fig. 6, a comprehensive experiment demonstrated the ability of the robotic hand to discern the pressure and temperature of each finger and provide corresponding feedback through the haptic glove (Additional file 3: Video S3). Figure 6a shows the force sensor and vibration motor signals under varying force applied to the index and ring fingers of the robotic hand. The force sensor signals and vibration motor responses confirmed real-time and responsive interactions. Additionally, a greater response was observed from the sensor and motor of the ring finger owing to the larger applied force. Figure 6b shows the response to temperature stimuli, showing signal variations when a cold pack and hand warmer were applied to the index and ring fingers, respectively, resulting in the cooling and heating of the Peltier elements in the corresponding glove fingers. Finally, Fig. 6c shows the force and temperature feedback during contact of the robotic finger with an object. Different signal responses were observed for each finger, reflecting the varying degrees of contact with the object. The nuanced interactions between the robotic hand and haptic glove led to successfully deliver real-time tactile information of force and temperature. Such advanced interactions can immerse users in a realistic sensory experience, effectively bridging the digital and physical realms of touch sensations.
Conclusion
We demonstrate an advanced haptic system that offers users a more realistic tactile experience than traditional haptic technologies primarily focused on pressure feedback. The proposed system integrates a robotic hand with a haptic glove and facilitates interactive and remote operations within the range of wireless communication. A key innovation of this system is its multimodal feedback mechanism, which employs both vibration motors and Peltier elements to accurately elicit pressure and temperature sensations. The Peltier elements are correctly controlled to provide a suitable temperature feedback. Hence, the transmission of temperature is safe, enabling users to perceive temperatures similar to those sensed by the robotic hand once equilibrium is reached. Simultaneously, pressure feedback accurately conveys forces to the user based on predefined values. Each sensor and actuator operate independently, allowing the robotic hand to precisely mimic the movements of the user, who can sense tactile information. The tactile information transmitted to the user's fingers via the glove creates a vivid illusion of direct contact with touched objects. The versatility of the proposed system may be conducive to applicability in various fields, particularly where manual operation is challenging or risky, such as in hazardous environments and medical interventions. By eliciting realistic sensations, the proposed system may substantially enhance the efficiency and safety of operators in such scenarios.
Availability of data and materials
Data will be made available on request.
References
Pyo S, Lee J, Bae K, Sim S, Kim J (2021) Recent progress in flexible tactile sensors for human-interactive systems: from sensors to advanced applications. Adv Mater 33:e2005902. https://doi.org/10.1002/adma.202005902
Huang Y, Yao K, Li J, Li D, Jia H, Liu Y, Yiu CK, Park W, Yu X (2022) Recent advances in multi-mode haptic feedback technologies towards wearable interfaces. Mater Today Physics 22:100602. https://doi.org/10.1016/j.mtphys.2021.100602
Abdi E, Kulic D, Croft E (2020) Haptics in teleoperated medical interventions: force measurement, haptic interfaces and their influence on user’s performance. IEEE Trans Biomed Eng 67:3438–3451. https://doi.org/10.1109/TBME.2020.2987603
Biswas S, Visell Y (2021) Haptic perception, mechanics, and material technologies for virtual reality. Adv Funct Mater. https://doi.org/10.1002/adfm.202008186
Xiao C, Woeppel AB, Clepper GM, Gao S, Xu S, Rueschen JF, Kruse D, Wu W, Tan HZ, Low T, Beaudoin SP, Boudouris BW, Haris WG, Wachs JP (2023) Tactile and chemical sensing with haptic feedback for a telepresence explosive ordnance disposal robot. IEEE Trans Rob 39:3368–3381. https://doi.org/10.1109/TRO.2023.3278455
Pyo S, Lee J-I, Kim M-O, Lee H-K, Kim J (2019) Polymer-based flexible and multi-directional tactile sensor with multiple NiCr piezoresistors. Micro and Nano Syst Lett 7:5. https://doi.org/10.1186/s40486-019-0085-6
Bae K, Jeong J, Choi J, Pyo S, Kim J (2021) Large-area, crosstalk-free, flexible tactile sensor matrix pixelated by mesh layers. ACS Appl Mater Interfaces 13:12259–12267. https://doi.org/10.1021/acsami.0c21671
Javidi R, Moghimi Zand M, Alizadeh Majd S (2023) Designing wearable capacitive pressure sensors with arrangement of porous pyramidal microstructures. Micro Nano Syst Lett 11:13. https://doi.org/10.1186/s40486-023-00178-7
Park B-S, Im S-M, Lee H, Lee YT, Nam C, Hong S, Kim M (2023) Visual and tactile perception techniques for braille recognition. Micro and Nano Syst Lett 11:23. https://doi.org/10.1186/s40486-023-00191-w
Ankit HTYK, Nirmal A, Kulkarni MR, Accoto D, Mathews N (2022) Soft actuator materials for electrically driven haptic interfaces. Adv Intell Syst. https://doi.org/10.1002/aisy.202100061
Pyo S, Eun Y, Sim J, Kim K, Choi J (2022) Carbon nanotube-graphene hybrids for soft electronics, sensors, and actuators. Micro Nano Syst Lett 10:9. https://doi.org/10.1186/s40486-022-00151-w
Lin W, Wei C, Yu S, Chen Z, Zhang C, Guo Z, Liao Q, Wang S, Lin M, Zheng Y, Liao X, Chen Z (2023) Programmable and ultrasensitive haptic interfaces enabling closed-loop human-machine interactions. Adv Funct Mater. https://doi.org/10.1002/adfm.202305919
Panda S, Hajra S, Rajaitha PM, Kim HJ (2023) Stimuli-responsive polymer-based bioinspired soft robots. Micro Nano Syst Lett 11:2. https://doi.org/10.1186/s40486-023-00167-w
Pyo S, Lee J, Kim W, Jo E, Kim J (2019) Multi-layered, hierarchical fabric-based tactile sensors with high sensitivity and linearity in ultrawide pressure range. Adv Funct Mater. https://doi.org/10.1002/adfm.201902484
Jin H, Kim Y, Youm W, Min Y, Seo S, Lim C, Hong CH, Kwon S, Park G, Park S, Kim HJ (2022) Highly pixelated, untethered tactile interfaces for an ultra-flexible on-skin telehaptic system. NPJ Flexible Electr. https://doi.org/10.1038/s41528-022-00216-1
Fang P, Zhu ML, Zeng ZB, Lu WX, Wang FX, Zhang L, Chen T, Sun LN (2023) A multi-module sensing and bi-directional HMI integrating interaction, recognition, and feedback for intelligent robots. Adv Funct Mater. https://doi.org/10.1002/adfm.202310254
Sun Z, Zhu M, Shan X, Lee C (2022) Augmented tactile-perception and haptic-feedback rings as human-machine interfaces aiming for immersive interactions. Nat Commun. https://doi.org/10.1038/s41467-022-32745-8
Acknowledgements
This work was supported by the National Research Foundation of Korea (NRF) granted by the Korea government (MSIT) (Grant No. RS-2023-00209937 and NRF-2022R1A4A1032030).
Funding
Not applicable.
Author information
Authors and Affiliations
Contributions
MK: experiments, figure preparation, data analysis, writing—original manuscript; CGG: experiments, software; SKR: experiments, software; HJK: experiments; DYJ: experiments; SP: supervision, writing—review & editing. All authors have read and approved the final version of the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Additional file 1. Replication of human hand gestures with the robotic hand.
Additional file 2. Validation of temperature feedback accuracy.
Additional file 3. Demonstration of haptic interface with multimodal tactile sensing and feedback.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kang, M., Gang, CG., Ryu, SK. et al. Haptic interface with multimodal tactile sensing and feedback for human–robot interaction. Micro and Nano Syst Lett 12, 9 (2024). https://doi.org/10.1186/s40486-024-00199-w
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s40486-024-00199-w