Telepointer technology in telemedicine : a review

Abstract

Telepointer is a powerful tool in the telemedicine system that enhances the effectiveness of long-distance communication. Telepointer has been tested in telemedicine, and has potential to a big influence in improving quality of health care, especially in the rural area. A telepointer system works by sending additional information in the form of gesture that can convey more accurate instruction or information. It leads to more effective communication, precise diagnosis, and better decision by means of discussion and consultation between the expert and the junior clinicians. However, there is no review paper yet on the state of the art of the telepointer in telemedicine. This paper is intended to give the readers an overview of recent advancement of telepointer technology as a support tool in telemedicine. There are four most popular modes of telepointer system, namely cursor, hand, laser and sketching pointer. The result shows that telepointer technology has a huge potential for wider acceptance in real life applications, there are needs for more improvement in the real time positioning accuracy. More results from actual test (real patient) need to be reported. We believe that by addressing these two issues, telepointer technology will be embraced widely by researchers and practitioners.

Review

Introduction

Fast and accurate long-distance communication have been very fundamental factors to humankind advancement. Telemedicine is one of the areas that benefited from the recent innovation in network and communications, which proved to be crucial in saving many lives [1]. A basic system of telemedicine typically involves communication between two or more persons that are located in different places. Some examples of telemedicine-based applications are education, surgery, consultation and many more.

Without doubt, the best mode of communication is face-to-face communication due to natural presence of gesture, interaction, deictic instructions, face expression and voice intonation, which helps in explaining the real meaning of the speech. However, long-distance communication lacks of natural presence contrary to face-to-face communication, which usually leads to misunderstand and misinterpretation of the information, especially when dealing with deictic gestures. To improve the communication quality, telepointer technology has been widely used to help the speaker in conveying the real meaning of the words. It is used as the support tool which is capable of boost up the power of distance communication through a sense of presence.

Telepointer technology can be defined as “an interaction style for presentation system interactive television, and other systems, where the user is positioned at a remote site from the display” [2]. The main function of a telepointer is to point at the specific display so that its motion could represent the human gesture. Meanwhile, display devices allow the collaborator to view the same scene as seen by the other parties. Greenberg et al.[3] stated that “telepointers are a natural focus of attention for group participants, and they can be leveraged to show information vital for smooth collaboration: interaction modes, system state, identity, actions of others, and so on.“ In other words, telepointer can be used 1) to point to an object or region of interest and 2) to create a pattern to signify something depending on movement, location and temporal information at the remote display.

Nowadays, a lot of applications use a camera-based physical gesture to improve the communication quality, especially hand gestures that can be used for pointing, overlaying hands and sketching [4]. Telepointer can be broadly classified into two categories; low level and high level. Low level system refers to a pointer which provides the coordinate information only such as cursor and laser pointer. Meanwhile, high level system provides more than just coordinate information by providing instruction and complicated data such as hand gestures, sketching, drawing and overlaying hands.

A variety of technologies have been developed to facilitate telepointer gesturing, such as GestureMan [5], DOVE-Drawing over Video Environment [6], Mixed Ecology [7] and Head mounted display (HMD) [8]. GestureMan employs a robotic system that represents the gesture through a laser pointer. DOVE system creates the sketching gestures by overlaying pen-based gestures on a tablet personal computer (PC) and Mixed Ecology employs unmediated gestures. HMD based on augmented reality by superimposed pointer on a video channel.

Basic system telepointer

Telepointer is vital elements in remote collaboration and computer supported co-operative work (CSCW). CSCW is a combination of hardware and software resources to allow groups to collaborate either in static or mobile environments. Static refer to fix confined workspace while mobile refer to moving workspace either remote or local user or both. Hence, technical setup and hardware system almost inherit from both areas. A simple mechanism is used to set up a telepointer system which consists of a video display; a pointer device; a computer; and a camera as a means of communication with the collaborator. A camera is used to capture the visual information at the local site which is then transmitted to the remote site. The local site will have a pointing device (e.g.: cursor mouse) to point to a particular object on the computer display which should be synchronous between the local and remote sites. As a result, the other parties will be able to view the exact scene as viewed by the sender in order to reduce false information. Figure 1 shows the typical telepointer system applied in telemedicine.

Figure 1

Typical telepointer system applied in telemedicine and laser as a pointing device (adapted from Ereso , et al.[9]).

Motivation

A telepointer system offers many advantages such as 1) to provide coordinate information of the target object, 2) create the sense of presence and 3) increase the audience attention by using the multiple form of the telepointer such as colour, size and image [10]. It is difficult to rely on verbal instruction alone while giving the instruction through the internet. By adding the pointer information, the collaborator will be able to understand the command easily. Previous researches [11–14] have demonstrated that a telepointer system has been well accepted as the support tool for sending gesture information and enhance the performance of the collaborator. In addition, it also reduces the travel cost since both collaborators are not required to be at the same place for undergo their activities.

Sharing and exchanging knowledge about the treatment and diagnosis of a disease is a common practice in the medical world. It is crucial in the medical field to have effective communication mode such as discussion and consultation in order to come out with the best treatment. Lack of expert and specialist in rural areas is of great concern for the government, especially when diagnosing a rare disease and difficult symptoms. Therefore, telemedicine can be used to overcome this shortcoming. The problem becomes more complicated for an online-based surgery where a specialist needs to assist a general surgeon in a remote site who may lack the required skills. An example of successful telesurgery was given by Brévart et al. [15] where an emergency surgery needs to be performed to a two-year-old boy, who had a severe vertex epidural hematoma (VEDH).

As reported by McLauchlan et al.[16], only 32% of the junior doctors managed to successfully diagnose a trauma case based on X-ray’s data, while 80% of the senior doctors correctly diagnosed the same disease. This issue can be attributed to low confidence among the junior doctors where a second opinion is needed to help them in doing the diagnose [17–19]. Hence; a telemedicine can be employed as a support tool for facilitating the knowledge sharing. A lot of researches [20, 21] believe that the telepointer technology has a great potential in the medical field, since it 1) improves the performance of doing a task, 2) increase the accuracy of the diagnose and 3) avoid the tragedy during the golden rescue minutes due to lack of specialist. According to Sachpazidis et al.[22], the telepointer is an important feature in collaborative applications, especially in medical collaboration. It helps a lot during long range diagnosis, consultation and mentoring by enhance the medical services and computer-mediated instructions.

However, there are two main limitations to the telepointer. Firstly, network condition will heavily affect the quality of transmit signal. Several studies [23–25] show that a major contributor to error of pointing is due to delay, which result in jitter and latency. A delay to the pointer system may cause a fatal consequence, especially in tele-surgery where a wrong location may be pointed. Secondly, current telepointer systems lack of tracking capability. Tracking allows the system to be updated by using prediction data in case of short-period signal loss.

The paper is organized into 5 sections. A brief description of Telemedicine is given in Section 2. Section 3 outlines the telepointer technologies in telemmedicine. Issues and challenges in section 4. Conclusion and future works are summarized in section 5 respectively.

A brief description of telemedicine

Initially, telemedicine was inspired by evolution of the space technology which happened around 1960s. The system was first used to monitor astronauts’ physiological parameters by using real time wireless approach during the outer space expedition. Presently, clinical medicine that allows patient medical information to be shared through interactive multimedia has been the driving force of telemedicine application development. Rizou et al. [26] defines a telemedicine system as “Telemedicine is the use of electronic information and communication technologies to provide and support health care when distance separates the participants (physicians, providers, specialists and patients)” as illustrate at Figure 2. Thus, we can infer from the definition that a telemedicine system has a very limited capability to interact physically with all the involved parties except through a telecommunication mean.

Figure 2

Participants of telemedicine communicate at distance thru through internet network (adapted from[27]).

Telemedicine can be broadly divided into two types of communication mode; online and offline. Offline method is the simplest approach, which requires less sophisticated equipments and technical facilities. Basically, the system just records the patient data first and then transferred to the interested party. For a teleconsultation system, medical data such as images, sound, and text are collected and stored, which is then forwarded to the medical specialist. The specialist will then diagnose the data at any convenient time and of course within the allowable time frame. Commonly, offline method was only applicable for non-critical diseases such as small skin issue and minor pathology problem. Besides, it is used to obtain a second opinion to strengthen the first doctor deduction. Several examples of popular offline communication mediums are email [28], iphone [29], Multimedia Messaging Service (MMS) [30], and Short Message Service (SMS) [18] which do not require a telepointer system in place.

Contrary to the offline method; online approach involves two ways of communication simultaneously by sending and receiving the feedback instantaneously. It provides more convenient and satisfaction [31] due to additional sense of presence. It is normally employed for the critical cases (heart disease, diabetes mellitus, cardiac), which requires face to face communication. The usual modes of online communication are video conferencing technology that integrates both multimedia elements of video and audio [32, 33], desktop to desktop [34]. The setup requirement of online method is more extensive because of the extra technical equipments required such as video camera, video conferencing equipment, telepointer, computer and high-speed network.

Telemedicine has also been applied to both clinical and nonclinical applications. Clinical term refers to the task that requires direct diagnosis and treatment of the patients such as general healthcare delivery (nursing, follow up, trauma, rehabilitation, pharmacy) and specialist care delivery (cardiology, pathology, dermatology, surgery). On the other hand, nonclinical is any task that does not involve treatment and diagnosis of the patient yet still related to the patient care such as distance learning for sharing knowledge and information, pre-operation meeting, medical appointment and many more. Both clinical and nonclinical job can be enhanced by implementing a telepointer system for more efficient communication.

There are several subbranches under the telemedicine system such as teleconsultation, telementoring, teleproctoring, telesurgery, telediagnose, telepresence, teletrauma and many more. All of these branches can be perceived as a cheaper alternative to the traditional methods [35] except for the telesurgery. Telesurgery is the most sought over service and requires the most advance facilities to operate on. The surgeons will remotely control the robot action that located in the operation theatre by using a computer interface. One of the earliest successful telesurgery was achieved by operation Lindbergh in 2001 [36]. It was the first surgery where the operating surgeon was removed from the operating room to perform minimally invasive surgery on a human patient. The surgeon was in New York City while the patient was in Strasbourg, France. The setup cost was very expensive, which can go up to $100,000 per terminal [34]. However, most hospitals or clinics that need the system most are located in rural area of poor and developing countries, where majority of them cannot afford these terminals due to limited health care budget [37]. Besides, the system is still in early development phase, which can be very risky because of the 1) failure possibility due to clinician lack of skills, 2) technical error of the communication system and 3) quality inconsistency across geographic or economic boundaries. Hence, a good network and communication technology is a vital component for a successful telesurgery system.

As we stated before, a good communication system is very critical for an accurate telemedicine system since it will act as the medium for delivering and exchanging information from one place to another place. Performance of a telemedicine system always dependent on the network bandwidth which gives the upper limit of information transferred. In order to receive instantaneously accurate information, a telemedicine system required a high bandwidth network to transmit complex medical information in a real-time. The main downside of such a system is a higher installation and maintenance cost. For example, Nagata and Mizushima [34] has shown that a medical image in Joint Photographic Experts Group (JPEG) format of 1000 x 1000 pixels requires 10 to 65 seconds to be uploaded to four different clients, while an image of 640 x 480 pixels had a much faster response time (2-5 s) with 1.5Mbit/s connection. Besides, choice of the equipments, transmission media and network bandwidth are dependent on the remote hospital location. As for example, a telemedicine system in Madagascar cannot operate on a high bandwidth system due to limited technologies [38]. They can only manage a system with low cost equipments for 25 kbit/s bandwidth connection. Therefore, only small-sized data such as electrocardiogram (ECG) and blood pressure can be transmitted for a real time teleconsultation.

Telepointer technology in telemedicine

Current technology in telemedicine allows the patient image to be transmitted in real time to the expert by using a camera. A two-way communication is established via an audio-visual tool, which enables the general clinicians and the expert to view the same video as displayed simultaneously on the expert’s terminal. There are several approaches in telepointer where the most popular methods utilize a laser pointer, cursor movement or sketching to generate a mutually visible remote pointer in the shared workspace. Figure 3 shows the division of the telepointer technology used in telemedicine, which will be discussed in the following subsections.

Figure 3

Division of the telepointer technology used in telemedicine.

Cursor pointer

Cursor pointer can be regarded as the simplest pointer. It is just a small graphical pointer on the screen display, which normally takes an arrow form. Location and movement of the cursor are controlled by an external computer device such as mouse or touchpad. Numerous studies have explored the usage of the cursor pointer for the remote collaboration [24, 39, 40] and have been proven to be very important in enhancing the performance of the collaborative tasks. Kirk and Fraser [7] found that the performance of overlaying hand approach was better than the actual sketch devices (for example, cursor pointer, pen) in terms of the effectiveness. However, sketch device has a superior performance compared to a simple laser pointer.

In telemedicine, cursor pointer plays the main role in showing and stressing the details of the examined image. It can be used to highlight any specific location and distribution of a lesion which has been applied in the majority of the teleconsultation systems. In [41], Julsrud et al. explored the capability of a cursor pointer system for congenital heart disease consultation. They have performed an initial test of 54 sessions of teleconsultation comprised of 38 patients with various types of congenital heart disease. The results were promising since 72 (67%) respondents out of 108 observations believed that a cursor pointer was helpful during the consultation. Hence, the study suggested that the implementation of a cursor pointer will enhance the consultation process, especially for congenital heart disease. Moreover, the system has also been implemented in teleradiology consultation [42, 43] as shown in Figure 4(A). A similar approach has been adopted in telementoring of gynecological surgery where a cursor pointer is used to highlight the structure and landmark anatomy on the monitor of the operating room [44]. For this case, the cursor movements have been synchronized between the local and the remote sites.

Figure 4

(A) cursor pointer applied in teleradiology (adapted from[43]) (B) laser pointer applied in telementoring (adapted from[45]) (C) telestrator applied in laparoscopic (adapted from[46]) (D) hand pointer (adapted from[21].

Generally, a cursor pointer system in telemedicine can be classified into two modes of communication scheme; 1) master and slave and 2) groupware. Both schemes apply the concept of “What you see is what I see”. It means that each user could see what the other parties were pointing. Master and slave scheme involves two parties where the collaboration only occurs if one person situated at the local site while the other person at the remote site. This scheme was designed to prevent any competition while operating the telepointer such as the cursor and laser pointer [22, 47]. Hence, only one person will be granted the authority to control the cursor pointer at one time. The slave site can become the master just by pressing a special button to reverse the authority. Groupware scheme involves more than two parties where each user can see their own action and other users’ cursor pointer. Anyone can control the other parties’ pointers. In the work by Lee et al.[43], they faced inconsistency issue with the cursor pointer when too many users activate the system simultaneously. This issue can be solved by using a simple solution such as token passing scheme.

Some studies have focused on improving the attention and gaze awareness among the collaborators by enhancing the visual properties of the cursor. By default, a cursor pointer is usually small and sometimes its movement may not be noticeable by others. It will be worse if the display is situated far away from the user. Alternatively, Sachpazidis et al.[22] proposed changeable graphical symbol to represent the cursor when it is in active mode. The changeable scheme has also been implemented in the hand cursor system. A blink of yellow flash is added below the cursor to indicate the authority transfer to the remote site. If the transfer is successful, the cursor colour will be changed to grey. Usually, a cursor pointer has an embedded capability to zoom in and out just by clicking the right and left side of the mouse. In the system developed by Nagata and Mizushima [34], the cursor shape was changed to an arrow, and the user can define their own cursor colour. For a groupware application, cursor pointers of each participant are marked by their name and host address of the remote computers [48].

Real time communication software have been developed to improve high-level telemedicine system. Some examples of existing methods are; 1) wavelet-based interactive video communication system (WinVicos) [49] 2) TeleDicom [50] 3) REmote Patient Education in a Telemedicine Environment Architecture (REPETE) [51]. All of these methods used sophisticated communication software, which consists of all minimum requirements for a collaboration scheme such as audio, interactive video conference system, still images and a telepointer. All of these softwares used cursor as a pointer for deictic referencing and gesturing. However, Huang et al.[21] claimed that a simple cursor pointer was insufficient for an effective collaboration.

Laser pointer

The first laser pointer was invented in 1960 [52] with the intention of replacing the traditional pointer such as a hand-held wooden stick for more flexible presentation. The main advantages of the device are long range pointer capability and ability to function well in a low ambient surroundings. It produces a bright spot of light to attract audience attention. It can be broadly classified according to its power consumption, either low or high. A low power pointer usually requires 1mWto 500 mW while a high power device can consume power from 1000 mW up to 3000 mW.

Laser pointers have three primary colours; red, green and blue. Each colour have a different range of wavelength with red laser has the highest wavelength, followed by green and blue. Red and green colours are most suitable to be implemented in a task that requires the attention from the audience [53]. Unfortunately, red laser have similar colour to blood and human tissue, which makes it not suitable for pointing in clinical applications. Hence, the usual practice is to implement green laser, which have a good contrast to blood and tissue colour. Paper by Schneider, et al.[54] claimed that green laser performed well enough in the operative field for clinical remote consultation.

During its early development, laser pointer system is usually coupled with the monitor interface. It just not acting as a pointer, but able to navigate the display as well as function as a mouse [55–57]. The advantages telepointer usage in telemedicine are 1) it can directly point to the interest object, 2) it requires a very low bandwidth, 3) it is not an invasive procedure, 4) it can be easily incorporated into the modern operating theatre and 5) low cost installation.

Nowadays, laser technology is used widely in telemedicine not just for pointing purpose since a high power device have been used in surgical operation to cut through tissue, seal and cauterize wounds while a low power device can be used to treat injury and speed-up the healing process. Usually, for a pointer application, a diode laser pointer will have a power less than 1 mW power [45]. It has been used to improve teaching efficiency during surgical education of a laparoscopic surgery. Ursic et al.[58] used a common pen-sized laser pointer to mark on the video screen to support and accelerate interaction between the surgeons and their assistants. Laser pointer can also pinpoint the correct entry points as demonstrated by Racz and Kao [59]. Moreover, it is used to guide the trainees while learning the anaesthesia procedures. The surgeon also benefitted from the system since a laser pointer can act as the user input to a robotic system before they perform the medical procedure autonomously [60]. Similarly, laser pointers have also been applied in Computer Assisted Orthopaedic Surgery for locating landmarks intra-operatively [61]. Figure 4(B) shows laser pointer marks on patient body which applied in telementoring.

Preliminary research for the remote laser pointer was started by Yamazaki et al.[62] when they realized a fixed computer does not support the interaction between space and the real objects. Output of this research has allowed them to come out with a new idea specially for remote medical instruction. They developed a system that allows the remote expert to control laser pointer movement by using a mouse cursor, which acts as a pointer that can be directed to any particular area of interest on the patient’s body [45]. By using cameras, the system enabled both local and remote sites to share the same visual data. A similar approach was employed by Ko and Razvi [63] and Schneider et al.[54], in which a laser pointer was integrated to a camera that will point out the relevant anatomical points of interest as directed remotely by an expert. In [9], Ereso, et al. developed a system that allowed the camera to track the hand of the local surgeon. However, none of the previous publications provide much detail on the image processing part of the laser pointer.

There are other technologies that are able to support two ways communication directly such as tabletop [64], Wearable Active Camera/Laser (WACL) [8], and HandsOnVideo [65]. However, most of these systems require a complex technical setup and can only function well under limited environments for the gesture and interaction to be recognizable. Moreover, it is a difficult task for the surgeon to maintain the stability of the camera during a surgery, which might distract their concentration. Besides, it is difficult to make an accurate diagnosis and observation on the remote site when the captured image is blurred due to unstable camera effect. Furthermore, laser pointer movement depends only on the cursor movement by the expert which might lead to an error, especially while clicking the interface or taking a glance at the patient’s body. This error will lead to the inaccurate area of interest.

Results in [9, 45] have shown that the presence of a laser pointer device can increase the performance speed of the surgeons. As an example, such a device can enable inexperienced surgeons to identify correctly the intended location of the surgery and enhance their chance of doing the job right on the first trial. Without the support by a laser pointer system, the novice surgeons took a longer time to complete the task and even some of them need a second trial.

Sketching

Sketching is a basic skill in painting and drawing. In a telepointer system, sketching can be regarded as a gesture system with the added ability of pointing. The user is given the freedom to draw anything without any restriction such as any shape, line and even writing a word. Normal implementation of a sketch pointer is by sharing the drawing activities or “virtual sketchbook” among the collaborators [66]. Video input is used to provide a common drawing surface where each collaborator can view the combined image of all the sketches made by the collaborators. The work by Ou et al.[6] proposed a method to integrate visual information and gesture beyond the static surface like a sketchbook. They implemented a system of dynamic surface in the collaborative physical task environment. Their system can support pointing and gesture activity by using the sketch pointer on a video stream. Stylus, touch screen and telestrator are used as a drawing tool that allows its operator to sketch an overlay over an image generated by the camera.

Telestrator has a special ability that allows the user to draw on the television screen by using a particular stylus pen. It was invented in the late 1950s and widely used in advanced application of telemedicine such as telesurgery [67], teleproctoring and telementoring [68]. Telestrator managed to improve performance of a clinician who has the fundamental knowledge about the surgery yet limited or no experience in the operation room. Ali et al.[69] stated that telestrator is an important teaching tool, especially for minimally invasive surgery or also known as laparoscopic. Telestrator commonly used as annotation tools for directing the surgeon by highlighting the point of interest that allows better demonstration of the anatomical structure. It can also be used to lead the surgeon where to place the instruments, other than pointing out any operative threat that he might not aware. An example of such a system is used in telementoring of an adrenalectomy laparoscopic [46] as indicate in Figure 4(C). A surgeon who is an expert in open surgery and skiilful in laparoscopic procedure but limited experienced in laparascopic adrenalectomy has been guided by another expert from a remote place to perform the laparoscopic successfully. He needs to identify and recognize all the difficult structures, such as vena cava, adrenal veins, and in spleen medialization in order to complete the operation. Here, telestrator is used to guide the surgeon to identify and recognize the difficult structures. The result showed that the operating time was shorter compared to the other authors report and at the same time no complication to the patient is reported. We strongly believed that the implementation of a telestrator enhances the learning experience and accelerates the learning curve of for the laparoscopic procedure.

Conventional telestrator have a limitation on the number of participants. It operates on one-to-many relationship where only the remote site is allowed to manipulate the pointer while the local site can only view the output. The system will feed the information back to the remote site through audio information. This restriction makes a telestrator system unsuitable for a groupware application that requires multiple input from all the users. Besides, communication architecture of a conventional telestrator does not allow the users to recover back the original video. As the technology evolves, new form of multimedia framework for the telestrator was developed. Qiru and Dong [70] presented an approach that provides a platform for n-way Interactive Visual Content Sharing and Telestration (IVCST). The architecture of the system allows multiple users at remote sites to sketch on a shared visual content. Each user is identified by using a name tag and different sketch colour. A conventional telestrator system utilizes 2-dimensional visual system which overlaps the pointer with the points of interest. This approach reduces the accuracy of the telestrator from the user point of view. To lessen this effect, Ali et al. [69] investigated a 3-dimensional system for potential integration with the telestrator system. Early results showed that the system is feasible for further development since it receives positive feedback from the doctors. In addition, it provides more diverse spatial information such as 3-dimensional sense of presence and can be used for depth perception, which results in more accurate pointing position.

The popularity of a stylus pointer in touch screen tablet and personal digital assistant (PDA) technology were increased due to wide spread usage in telepointer and telemedicine applications. Both technologies allow the remote user to be at more dynamic position compared to the telestrator technology which requires the remote user to be at a static place. The main advantage of a pen-based pointer compared to other telepointer modalities (cursor, laser pointer) is the user friendly factor since it imitates the way human write on a piece of paper. Kim et al.[71] claimed that “pen-based computing provides people with an intuitive way to use a computer” because the user can leverage their writing skill without having to learn how to operate the keyboard and mouse. Therefore, a patient with low computer skill can still be able to provide gesture feedback to the doctor during a teleconsultation session. Dante [72] examined the usability of a pen-based pointer among the novice older users. The results revealed that majorities of the participant were satisfied with the pen-based pointer compared to the cursor since they can focus their hands and eyes at the same location. Meanwhile, cursor pointer requires them to focus on two different parts; 1) a mouse for moving the cursor and 2) view the display in order to locate the cursor on the screen. Hence, it is suitable for the patient with Parkinson’s disease [73, 74]. In the work by Tu et al.[75], they explored the feasibility of using pen and finger-based pointer for the touch screen application. They found out that both pointers have their own advantages depending on the types of applications.

Hand pointer

Hand pointer is a subset of the hand gesture system that mainly utilizes finger as a pointer. Fussell, et al.[4] classified the hand gesture system into four categories; 1) deictic (pointing), 2) iconic, 3) spatial or distance and 4) kinetic motion. It is considered as a nonverbal communication method since it is portrayed by human action and body language. There are many forms and shapes of the hand gesture that dependent heavily on the culture and norm of the society. Several examples of the human gesture technology are implemented in head tracking [76] and eye tracking [77] systems where the users are allowed to interact directly with the computer without using the mouse.

Hand pointer system is an active research area, especially for remote collaboration on a physical task [4, 7, 13, 78]. Basically, the input data or the hand gestures are captured by a camera that will be projected to a local workspace and displayed on the monitor. This set up requires the user to be at certain location, which limits the user mobility. Therefore, Alem, et al.[65] and Huang, et al.[21] a wearable system that allows the user to move freely since the camera is not fixed anymore as illustrate in Figure 4(D). Their systems operate on one to one relationship. Both remote and local sites are allowed to send the gestures, but only the local site can view the projected image. Alem and Li [20] then investigated the possibility of combining hand gesture and cursor pointer together for video-mediated collaboration. The results revealed that both gestures had almost similar performance since all participants performed the tasks well with only minor mistakes. However, most of the participants preferred a hand gesture pointer instead of the cursor pointer because a gesture is more flexible and can represent more symbols. On the other hand, a pointer can only represent a deictic point and can be misunderstood easily by the other parties.

Current hand pointer applications are still in early stage for teleconsultation usage due to high requirement of the telecommunication bandwidth to convey the images. Hence, it requires an expensive installation cost, which is not feasible for rural area implementation. Moreover, a hand pointer system also requires a more accurate algorithm to perform well such complex modeling of the scene. Paper by Argyros and Lourakis [79] proved that the preprocessing steps required is very challenging such as detection, tracking and recognizing the hand gestures. Gallo and Ciampi [80] improved the system robustness by using a hand glove for more recognizable gestures. However, usage of a glove is unpractical for a telemedicine application due to heavy and bulky size of the glove that hinders the doctor ability to write the diagnosis report at the same time. Besides, the high-tech glove contains a lot of electronic components that require careful handling. The user needs to be trained on how to operate the devices optimally for a safety reason.

However, the usage of a hand pointer system in telemedicine is encouraging, especially for human-computer interface (HCI) applications. An example of hand pointer implementation in telemedicine is to perform sterility procedures in order to avoid bacteria and virus contamination. Hand pointer is manipulated to direct the inexperienced doctor to perform the procedures as shown by Grätzel, et al.[81], Grange, et al.[82] and Wachs, et al.[83]. The gesture assisted system works as follows 1) “push-to-click” is represented by hand pressing movement 2) “wait-to-click” is inferred from the absence of hand movement for a particular length of time. 3) “turn left” command is performed by moving the hand towards left direction. In [84], the authors improved the hand pointer system for dynamic environment usage by considering both hand motion and posture simultaneously for realistic gesture representations. A hand pointer system is also used to zoom in or out the camera perspective remotely by manipulating all five fingers movements [85]. According to Wachs, et al.[83], this technology managed to improve surgeon performance by reducing unnecessary movements while discussing with his assistant and browsing through the patient data. Therefore, we strongly believed that a hand pointer system will be further developed given such a vast potential application. Table 1 shows the comparison of each pointer technology.

Table 1 Comparison between pointer technologies in telemedicine

Issues and challenges

In the previous subsections, we have discussed the main role and current technology available for the telepointer system. Although it has the ability to increase the quality of health care, issues such as clinician’s concentration, insufficient gesture information and real-life applications should also be addressed.

Clinician’s concentration

The usage of a cursor pointer on the projector screen or display device is a common practice in medical applications. Because of this practice, a clinician needs to divide their focus between looking at the pointer on the display device and the patient. A clinician needs to look repeatedly at the projected image to see the exact location of the expert’s pointer. Difficulty arises when the expert asks the clinician to search for a specific point on the patient body. This may cause a serious problem, especially during a telesurgery where the focus of attention should be on the patient instead of the display device. Therefore, a telepointer system which is capable of transferring digital gestures to the physical workspace is more suitable for this type of telemedicine application.

Insufficient gesture information

Laser pointer has the ability to point directly to the specific point on the patient. Thus, it will reduce the search time and a more accurate localization can be obtained. Furthermore, pattern of the laser spot movement can also represent some form of instruction. However, a clinician claimed that normal telepointer without any gesture information is not sufficient enough to lead the direction in complex teleconsultation process [86]. Moreover, laser pointer movement depends only on the cursor movement by the expert who might lead to an error, especially while clicking the interface or taking a glance at the patient’s body. This error will lead to an inaccurate area of interest.

Real-life applications

A telemedicine system requires more accurate and precise technology since it deals with living things. Most paper [43, 45] only considered static patients. However, this assumption does not represent precise real-life applications since human does move a lot. Certain parts of the body such as mouth and eyes move unintentionally as reported by Jon and Bowden [87]. In surgery, internal organs also move unconsciously such as heart pumping, colons contraction and lungs expanding motion. Real-life applications must consider the ability of the pointer to track and point to the accurate spot on the object at a local site. It should reflect the same point as pointed out by the expert at the remote site regardless of the object movement at a local site. Salient features are needed to track the moving components. Tracking process will be more complicated due to poor texture on the human skin and tissue, which will degrade the performance of the salient feature’s detection.

Conclusions & future works

In this paper, we have discussed state of the art of telepointer technology in telemedicine. Based on recent publications, we can infer that telepointer system is a very valuable support tool for telemedicine applications that improves communication quality between 1) clinicians and other clinicians, 2) clinicians and patients, and 3) medical students and expert clinicians. Telepointer system is a nonverbal communication mode that enhanced the communication capability of the remote user so that more accurate information will be delivered. There are many types of telepointer hardware such as cursor, laser, hand and sketching tool where every tool has their own unique advantages and disadvantages. In addition, we have identified the environments and surroundings where the tools will perform the best. For the teleconsultation purpose, cursor pointer and sketching were usually used as the support tool by the specialists to convey the instruction to the junior clinicians. On the other hand, laser pointer is used to enhance the communication between clinicians and patients.

Currently, several works in CSCW field [23, 24] have focused on robustness improvement of the system to unstable network. We also found out that there is a limited study in telepointer performance with respect to the localization, actual patients and outdated instruments, which result in the limited report on real object implementations.

As for future improvement, we believe there are two areas that will enhance the current system significantly. Firstly, the system can be implemented in a parallel processor instead of sequential approach for faster processing speed. Most of the existing telepointer hardwares utilize central processing unit (CPU) as the main processor, which performs moderately slow if the image resolution is high. High performance processor is required for real-time applications, especially in telesurgery where the decision must be made instantly. Therefore, a graphics processing unit (GPU) is a good alternative to CPU for real-time telemedicine system. GPU consists of hundreds of smaller core, while current CPU technology typically has four to eight cores. This feature makes GPU as a powerful computing tool for medical image information. Besides, image resolution and interpolation can be executed at a faster speed. Due to bandwidth limitation, most of the captured images need to be transmitted at a low resolution. Jie et al.[88] implemented their down sample method in GPU so that the captured images can be transmitted at a lower resolution. Then, the down sample can be enhanced through image sharpening, which also requires heavy computational load. As a result, their method can perform five times faster than the conventional compression method. By having greater computational ability, hybrid telepointer with high resolution will be utilized more in the telemedicine field. One obvious advantage of using higher-resolution image is better localization of the pointer information such that more accurate pointing can be obtained.

Secondly, a telepointer system can be improved by using a more robust algorithm, especially for the image-processing part. A statistical approach will result in better learning compared to heuristic approach [89]. It will reduce the bottleneck effect when the decision rules are out of the input set such as during illumination changes. Statistical approaches are based on a collection of quantitative data manipulation and interpretation on its statistical properties to discover the pattern, relationship and distribution of the data. Most of the medical images contain nonrigid object and it is hard to model it by using fixed modelling. Statistical method offers the ability to learn from a bundle of data and come out with more robust formulation. This approach has been used in many high level applications [90] and has been applied in various medical systems [91, 92]. By using this approach, a better pointer tracking system will be developed to keep track the movement of the pointer device at the remote site. At the same time, the local site pointer can be tuned to follow the remote site movement in order to obtain more accurate feedback data.