The feasibility of using pattern recognition software to measure the influence of computer use on the consultation

Background

A key feature of a good general practice consultation is that it is patient-centred. A number of verbal and non-verbal behaviours have been identified as important to establish a good relationship with the patient. However, the use of the computer detracts the doctor's attention away from the patient, compromising these essential elements of the consultation. Current methods to assess the consultation and the influence of the computer on them are time consuming and subjective. If it were possible to measure these quantitatively, it could provide the basis for the first truly objective way of studying the influence of the computer on the consultation.

The aim was to assess whether pattern recognition software could be used to measure the influence and pattern of computer use in the consultation. If this proved possible it would provide, for the first time, an objective quantitative measure of computer use and a measure of the attention and responsiveness of the general practitioner towards the patient.

Methods

A feasibility study using pattern recognition software to analyse a consultation was conducted. A web camera, linked to a data-gathering node was used to film a simulated consultation in a standard office. Members of the research team enacted the role of the doctor and the patient, using pattern recognition software to try and capture patient-centred, non-verbal behaviour. As this was a feasibility study detailed results of the analysis are not presented.

Results

It was revealed that pattern recognition software could be used to analyse certain aspects of a simulated consultation. For example, trigger lines enabled the number of times the clinician's hand covered the keyboard to be counted and wrapping recorded the number of times the clinician nodded his head. It was also possible to measure time sequences and whether the movement was brief or lingering.

Conclusion

Pattern recognition software enables movements associated with patient-centredness to be recorded. Pattern recognition software has the potential to provide an objective, quantitative measure of the influence of the computer on the consultation.

Rationale for the study

Although the computer is now ubiquitous in the UK primary care consultation, there is as yet no consensus as to how its impact on the consultation should be assessed. The NHS information strategy [1] has accelerated the process of computerisation, stating that all practices should be computerised by 2005. This strategy aimed to improve the quality of data recorded, drive evidence-based practice and enable easier audit of practice data to explore if national targets are being met. By 1995 it was reported that around 90% of general practitioners (GPs) were using computers during their consultations [2]. The new GP contract [3], implemented in 2003, has quality targets that can only be met through the use of general practice computers to record markers of the quality of care.

The study team has previously used traditional 'single channel' video recordings of the consultation, to assess the effectiveness of new software in a consultation [4]. However, it proved difficult to carry out this assessment. No gold standard existed for an effective consultation, in the context of this particular software, so a model had to be derived and a rating scale constructed [5]. Assessors had to be trained, and reliability testing performed. The rating scale produced, only achieved borderline reliability. One of the reasons for this was that the assessors found on occasions that it was difficult to determine exactly what was going on in the consultation and how the computer was being used at the time. Often, what was going on and what was being recorded at that precise moment had to be interpreted by the nurse involved whilst replaying the video recording.

A subsequent study was conducted to see if using 'three video channels' overcame the problems associated with using a single video camera [6]. This experiment involved setting up one video camera to record the doctor-patient interaction, the "standard" view as used in the previous study. A second camera focused on the consulting GP's head and shoulders, so that it was easy to discern body language. A third video feed extracted what was entered into the computer. This 'three channel video' set up provided an enormous amount of information, and good insight into how the clinician integrates information. Its drawback was that it was extremely time consuming to analyse. Therefore there is considerable attraction in assessing whether it is possible to use web-cam technology to automate the objective assessment of non-verbal communication between doctor and patient and the degree of attention given to the computer in the consultation.

What is known about the impact of the computer on the consultation?

Herzmark et al [7] observed that the computer screen requires more attention than paper. Warshawsky [8] concluded that GPs spent less time interacting with the patient when they used their computer in the consultation. Pringle et al [9] reported that use of computers can lengthen the duration of the consultation. The computer can be used in different ways during a consultation. Fitter and Cruickshank[10] have identified three patterns of computer use;

1. Minimal users: Clinicians who only record information at the end of the consultation after the patient has left. This has lead to concerns of memory load affecting the completeness of the patient record and the final diagnosis.

2. Conversational users: Clinicians who record information throughout the consultation. Requiring the ability to alternate between tasks.

3. Block users: Clinicians that interrupt the consultation to use the computer, often leaving the patient sitting quietly.

Although there is no hard evidence as to which approach is optimal, styles that mean that the GP does not miss cues from the patient, are thought to be more desirable. Specific training has been developed to convert clinicians from conversational to block styles of computer use, and to identify communication skills that assist in maintaining rapport with the patient whilst using the computer in the consultation [11]. It is possible that this approach will improve the use of the computer in the consultation without prolonging it [12]. Ridsdale and Hudd have highlighted how patients wish to see some, but not necessarily all of the information contained in their computer record [13], and that generally they think favourably of doctors who use computers during their consultations [14].

Is there any evidence that patient centred behaviours are beneficial?

Aspects of non-verbal communication such as affirmative head nodding, gaze focused on the patient, leaning forward, affectionate touching and smiling have been found to have an important influence on patients perceptions and satisfaction of the consultation [17]. High levels of patient satisfaction are desirable, as more satisfied patients are associated with higher levels of adherence to treatment, understanding of their condition, adaptive coping, quality of life and health outcome [15, 16]. However, these important elements of the doctor-patient interaction may be compromised if the GP is focused on the computer. Therefore, an effective technique needs to be developed to identify, explore and quantify the balance between making best use of the computer, whist minimising its negative influence on the patient-centred tone of the consultation.

Does use of a camera in the consultation have an adverse influence?

There is no evidence that the use of a video camera interferes significantly with the consultation, it is therefore likely that the same will apply to a physically smaller web-cam. The use of the former during consultations has been widely researched and little impact on practitioner behaviour [17] or patient satisfaction with consultations [18], has been found. Video recordings have been used effectively across many academic fields for teaching and training purposes [19], they are mandatory for the Summative Assessment (a final test of fitness to practice) for doctors wishing to enter General Practice in the UK [20].

Assessment of the influence of the computer on the consultation

Assessment of the effect of the computer on the consultation can be carried out by exploring when the computer is used and through analysis of verbal and non-verbal behaviour.

It is accepted that there remains a somewhat limited research base about the effective way to use the computer in the consultation [21]. Models have been proposed [5, 22–24] but these are based on consensus and opinion, rather than rigorous scientific method. Much has been described about the use of the computer, but very little has been rigorously evaluated [2, 25].

Assessments of verbal behaviour have been revealed to be reliable and valid. For example, The RIAS (Roter Interaction Analysis System)[26] is typical of the type of coding system that can be used to code verbal behaviour in the GP interaction. Categorising each utterance (either a word or sentence conveying one meaning from the GP and the patient) into 5 categorises; social communication, affective communication, structural communication, health communication and Lifestyle/psychosocial communication. The percentage of verbal communication that focuses on each of these five categories can then be calculated. However, traditional assessments of non-verbal behaviour in consultations use assessments based on video recordings. The recordings lack sufficient detail, are based on subjective assessment and are time consuming to analyse. For example, timing how long the GP leans towards the patient or nods their head [27]. Use of three video channels captures the finer details of the interaction, although the assessment of these recordings remain based on subjective and time consuming analysis [6]. Therefore it is important that assessment methods are developed to increase the objectivity in evaluating the effect of the computer on patient-centred consultations.

Pattern recognition software

Pattern recognition software (PRS) is being increasingly used as a means of interpreting complex movements. One of these uses outside medicine is to interpret video signals such as the monitoring of traffic flows [28]. This technique has also been used in the medical domain for the recognition of gait disorders [29, 30]. However, in general the use of video pattern recognition is at a distance and as yet there are no published reports of its potential use to monitor the influence of the computer on the consultation.

The purpose of this feasibility study was to see if using a standard web-cam and PRS, it would be possible to determine and quantify use of the computer in the consultation. In addition, the investigation sets out to see whether elements of standard assessments of the patient-centred behaviour of the consulting clinician could also be measured using this technique, to overcome some of the problems associated with manually rating consultations. PRS will be an extension of the three channel approach previously described [6] with the aim to provide quantitative output of non-verbal behaviour.

The equipment consists of a web camera linked to a data-gathering node (DGN). The DGN is a powerful enclosed computing device capable of capturing the RGB (red, green, blue) pixel data from the camera, analysing it and outputting it for display and presentation. In this study the output was transferred via a local network, however, it could also be transferred (encrypted) to a web-server or via a mobile phone (GPRS – General Packet Radio Service) connection. The study used standard off-the-shelf equipment with no special adjustment made. The camera used in this pilot was 20 cm × 10 cm × 10 cm, a smaller version is available for internal use. It used standard video for windows (VFW) drivers. The information it collected was passed to the DGN, which was 28 cm square. These are the only items that would need to be in the consulting room. The data gathered was transferred via an Ethernet network to a personal computer (PC) of standard specification. Any PC with a 2400 MHz processor, or better can run the software. The output data was transferred to a MySQL database (the most popular open source database server worldwide [31]). Here the data output is interogated by a presentation programme. Standard outputs can be pre-programmed.

The camera and DGN were set up in a small office (to simulate a standard general practice consulting room). The office was smaller than an average size consulting room to test PRS at the minimum distance it was likely to be used (measurements of patterns at short distance is a less usual use of this technology). The resulting image is shown in figure 1.

The GP and patient with the trigger line between them.

Full size image

This tehcnical setup, and its associated algorithms, is the culmination of two years work, including testing PRS over different distances. The photographs presented in this article were based on final testing of PRS in a simulated consulting room, filmed over 2 sessions in one day. This study demonstrates the use of this technlolgy over the shortest distance thus far. Members of the research team enacted different combinations of movements such as moving of the head, whole body, moving GP alone, moving the patient alone and moving the GP and the patient to recreate a range of possible situations during a consultation.

Trigger lines, event counting and identifying the centre of gravity of a movement

An important part of the way PRS works is illustrated by the angled blue line in figure 1. This is the "trigger line" used by the pattern recognition technology to detect an event. Movement across this line is detected and counted. The line can be positioned anywhere within the frame at any angle. Systems can include irregular lines and, potentially, multiple lines.

A proprietary Cats Eye Network Systems (CENS) algorithm, used by the DGN, analyses colour changes in the pixels of the RGB (red-green-blue) signal to distinguish between moving foreground objects and static background objects [32]. The program then "identifies" the moving object by "wrapping" it and further delineating its presence by calculating its centre of gravity. This process can be seen in Figures 2 and 3. As the GP moves a wrap emerges. The wrap is delineated by a green line and the centre of gravity by a yellow line.

Wrap emerges as the GP moves his arms and chair and the patient nods his head.

Full size image

The movement of the GP towards the patient has a characteristic wrap as well as involving crossing the trigger line.

Full size image

The affirmative "nod" of the doctor's head also produces its own wrap. As he moves further and extends his arm the wrap captures the entire upper body. Note that the arm has now crossed the trigger line. The leaning forward towards the patient, had a characteristic "wrap" as well as involves the hand crossing the trigger line.

Once identified, the boundaries of the recognised object are used to register an event when the wrapped boundary of the object crosses the trigger line. A count is then incremented by one. Counts collect data about the time spent in an particular posture as well as differentiating between "passing" (i.e. brief) and "lingering" movements.

Ethics

The "GP" and "patient" in the illustrations are members of the study team, role-playing consultations. No real patients or confidential information was used. The research complies with the Helsinki Declaration [33].

The outputs of this feasibility study are that it is possible to capture movements that are associated with computer use and patient-centred behaviour on behalf of the GP. It is also possible to count the number of these and to measure time sequences. The software contains a whole host of sensitivity triggers which enables its use to be optimised for particular circumstances.

Movements captured

The following movements were captured. They can be identified by their characteristic wrap, and trigger lines can be set that enable them to be timed and counted.

1. Leaning towards the patient

This has been illustrated in the methods section and in figures 2 and 3.

2. Turning towards the computer

These movements and their associated trigger lines have to be carefully studied. They are illustrated in Figure 4. Note that the yellow line plots the movement of the centre of gravity of the GP. This pattern of movement is characteristic of keyboard work. Importantly the shape of the wrap and centre of gravity of the GP are different from those of a GP who has turned towards the patient, as illustrated in figure 3.

The wrap and centre of gravity associated with computer use.

Full size image

The patient in this picture is sitting totally still so the wrapping has disappeared, another marker of concentration and interaction with what the GP is doing on the computer screen. An agitated patient would show a wrap and a centre of gravity.

3. Hands covering the computer keyboard

The setting up of a trigger line that would enable measurement of when the GP's hands were over the keyboard could not be achieved from the camera angle used above. Adopting a different position allowed this to be measured. This is illustrated in figure 5.

If multiple trigger lines were used, it would be possible to compare the use of mouse, and other parts of the keyboard. There would be no preclusion to using more than one camera if this view was particularly important in developing an understanding of what had a positive and negative influence on the consultation.

4. Affirmative nodding, by both patient and GP

This can be detected, and trigger lines inserted to allow it's monitoring. This is most clearly illustrated in the "patient's" nod in figure 2.

Event counting

Event counting was performed for all the four movements listed above – one event is counted when a trigger line is crossed. The event counter appears in the bottom right-hand corner of the figures. For example, in figure 5 the hands had crossed the trigger line in front of the keyboard 145 times in that simulated consultation.

Trigger lines to allow counting of when the computer keyboard is used.

Full size image

The software allows event counting, time to be measured, and type of event "passing" – where the moving object is never stationary; and lingering where it is stationary for at least a pre-defined fixed interval.

There was not the time or resources to set up graphical outputs of the data within the context of this feasibility study. Although time and motion data could be graphically represented by a histogram bar capturing the time interval over each minute when recording consultations. This would provide a "density map" of activity, which is stored in a database for later analysis.

Principal findings

Pattern recognition software (PRS) can operate in the simulated surgery environment, and there is no reason why it should not operate as well within a real surgery. Aspects of the GP's computer use were captured using PRS. For example, the GP's hands covering the keyboard and turning towards the computer. In addition, the pilot demonstrated that some of the GP's non-verbal behaviours that increase or decrease the patient-centred tone of the consultation could be measured such as leaning towards the patient and nodding their head. An unexpected by-product of the work was observing the patient's body language.

Implications of the findings

The consultation is the single most important event in clinical practice [34] and therefore measures need to be in place to ensure optimum quality of the consultation. There is currently a lack of methods available to accurately measure the effect of computers on the consultation. Consequently, the impact of the computer has been rarely evaluated [12]. The feasibility of PRS to assess the impact of the computer on the patient-centredness of the consultation will therefore be a valuable approach in the bid to ensure the quality of patient care.

The implications of the findings are that objective, quantitative data can be collected about the consultation. If other aspects of the consultation were controlled – for example, pre-loaded medical history on the GP computer system, and actors with carefully prepared scripts were used as patients – it would be possible to quantitatively compare different GPs' consulting styles and different computer systems. This technique allows the patients' movements to be studied as well as the clinicians. This has the potential to provide valuable additional insight into the consultation and computer use within it, over and above traditional approaches based on subjective assessment. However, there needs to be further development of PRS to capture all of the aspects of non-verbal behaviour that has been found to be important to the patient-centredness of the consultation such as smiling, looking at the patient and affectionate touching [17].

Limitations of the methodology

It has not been possible to collect sufficient data to make comparisons with others' reports about the consultation. However, this method would enable the observations, for example, of Fitter and Cruickshank [10] to be validated by completely objective, quantitative measurement using today's clinical systems. The purpose of this study was to examine whether it would be feasible to use PRS to quantitatively measure body language, and the extent to which the computer distracts the GP from being patient-centred. However, the study team recognises that it may not be the only technique that can be deployed to measure these behaviours in the consultation. In addition, PRS will need to be conducted alongside three channel video recordings using traditional methods to assess verbal communication (such as the RIAS) to ensure that all aspects of patient-centred behaviour are captured. The layout of consulting rooms is uncontrolled, and whilst it may be possible within a standardised setting (maybe a set of consulting rooms used to examine candidates at the end of their training) it would be more difficult to set up in any consulting room, than video. Some of the movements detected using the wrapping, centre of gravity, and trigger lines may be better detected by other means discussed below.

A further limitation of this small pilot is that it was not possible to research real consultations, or to validate the PRS output with analysis using established assessment measures. The pilot involved capturing movements individually, but simultaneous collection would only require small modification of the software that has been achieved in other fields. No audio feed was taken, and it is recommended that this is simultaneously recorded in any future research, as it can be used to help clarify exactly what was happening at any given moment in the consultation.

Call for further research

The use of pattern recognition software will allow the deployment of advanced signal analysis techniques. Careful research is needed into what trigger lines are used, what the implications of crossing them are, and what agreement there is between the digital results obtained and those obtained from direct observation and rating scales.

PRS should not be seen as the only technology that should be deployed here. The video footage revealed that the clinician usually changed chair position when turning from patient to computer. Possibly the use of a low cost magnetic sensor attached to the chair – active when orientated towards the computer, and inactive in other positions may provide data is a more cost-effective way. A more expensive alternative would be to use an electronic compass.

Other sensors that might be useful are ones that would detect the hands over the keyboard (capacitive sensor) or its use by direct measurement. Voice recognition software could also be used, not to detect what was said – but to detect the proportion of time the GP and the patients' spoke for in the consultation. This could also be linked to computer use.

Further research into the automated, quantitative evaluation of the consultation needs to take into account the existence of other modalities that exist in addition to PRS. Our current view is that we will be looking for a mix of what are collectively termed "change recognition techniques". Elements of the following technologies will be required, as part of a broadly based data capture concept:

Visual motion detection (using a web-cam and PRS as set out in this paper)

• Audio capture and analysis

• Keystroke capture.

• These components of the data capture concept are set out in what we currently see as their relative importance.

Pattern recognition offers for the first time a chance to automate the collection of quantitative data about the consultation, and the use of the computer within it. It allows the patient, as well as the clinician to have their behaviour and reaction monitored. Further research and development, and the incorporation of other sensing technologies, is needed to turn this into a readily useable application.

This pilot was carried out as "own account" research with no grant or commercial funding. Main Highway provided equipment and time at no cost.

Affiliations

1. Primary Care Informatics, Department of Community Health Sciences, Hunter Wing, St. George's Hospital Medical School, London, SW17 ORE, UK
   • Simon de Lusignan
   • , Alice Theadom
   •  & Tom Chan
2. Main Communications, 39 Southgate Street, Winchester, SO23 9EH, UK
   • Eben Wilson
   • , Andy Dyble
   •  & Tristan Grant

Corresponding author

Correspondence to Simon de Lusignan.

Competing interests

Eben Wilson is an employee at Main Highway (part of Main Communications), the company responsible for the development of the technology.

Andy Dyble is an employee at Main Highway (part of Main Communications), the company responsible for the development of the technology.

Tristan Grant is an employee at Main Highway (part of Main Communications), the company responsible for the development of the technology.

None of the other authors have any competing interests to declare

Authors' contributions

S de L and EW conceived the study. AD and TG adapted and recalibrated equipment usually used count people or traffic at a greater distance to use in the confines of a consulting room. S de L and AT took the lead, but all authors contributed to the observations about the data, and the writing of the paper. All authors read and approved the final manuscript.

Reprints and Permissions