Abstracts and Presentations 2007
The Royal College of Surgeons, Edinburgh - 16th November 2007
Research Methods Symposium
Observing and Measuring Behaviour:
Non-technical Skills in the Operating Theatre
Abstracts – see participant list for contact details
The future of safety in health care: Lessons from NASA and the Columbia accident
Institut for Ergonomics, Ohio State University
Midwest Center for Inquiry on Patient Safety
Veterans Administration Medical Center, Cincinnati, Ohio
I. Tools for Measuring Behaviour of Surgical Team Members
The Observational Teamwork Assessment for Surgery (OTAS):
Development and Feasibility
Shabnam Undre, Andrew Healey, Nick Sevdalis, Maria Koutantji, Charles Vincent
Clinical Safety Research Unit, Imperial College, London
Teamwork is one of the important aspects of good practice and important for safety in surgery. To improve teamwork, assessment measures and training interventions are necessary. High reliability organisations have stressed the importance of teamwork for safety and regularly provide such training to their team members. We have developed a comprehensive assessment for teamwork in surgery – namely the Observational Teamwork Assessment for Surgery (OTAS) and tested its feasibility.
This tool was tested in 50 general surgical and 50 urology procedures. OTAS comprises a task checklist (patient, equipment/provisions and communications tasks), and ratings on team behaviours (communication, co-operation, co-ordination, leadership and awareness). In Urology, teamwork was assessed separately in the surgical, anaesthetic, and nursing sub-teams in the operating theatre. We also assessed the reliability of the behavioural scoring.
Results showed that a number of communication and equipment/provisions tasks were not routinely performed. Regarding team behaviours, adequate reliability scores were obtained. In sub team behaviours, anaesthetists and nurses obtained their lowest scores on communication. In addition to low scores on communication, surgeons' teamwork behaviours appeared to deteriorate as the procedures were finishing.
Our findings suggest that OTAS is feasible and applicable to different branches of surgery. Separate assessment of the different sub-teams in the operating theatre provides useful information that can be used to build targeted teamwork training aiming to improve surgical patients' safety and surgical outcomes. Team training interventions such as briefing and simulations may be applied to routine surgery to enhance communication and team working in theatre.
Development and validation of a non-technical skills assessment tool for operating theatre teams
P. McCulloch, K. Catchpole, A. Mishra, T. Dale*
Nuffield Department of Surgery, University of Oxford and Attrainability plc*
The specific behaviours, tasks and roles of nursing, surgical and anaesthetic sub-teams, and their asymmetrical relationships need to be recognised by any adequate description of non-technical skills in theatre. We report the development of a tool to allow assessment of these skills.
We adapted the aviation NOTECHS tool, developing a taxonomy of non-technical skills for operating theatre personnel through iterative observation and discussion with domain experts over 2 years. The behavioural markers used to anchor scale dimensions were modified to recognise the asymmetric nature of working relationships between surgeons, nurses and anaesthetists. The scale was evaluated in teams performing laparoscopic cholecystectomy operations. Two observers parallel scored 24 procedures to test reliability. Validity was tested by (a) comparing NOTECHS and OTAS scores (b) analysis of successive cohorts (c) observing NOTECHS scores and technical skills before and after training. We analysed the whole team and sub-teams separately.
We studied 65 laparoscopic cholecystectomies, 26 before and 39 after training. Interobserver agreement was excellent, (Cronbach's alpha = 0.938). OTAS and NOTECHS agreed well (Pearson's r=0.886, p=0.046, n=5). Cohort of observation did not affect mean NOTECHS scores. The mean NOTECHS score was 37.4, increasing after non-technical skills training from 35.5 to 38.6 (t=-3.019 (P=0.005)).
Technical errors correlated with NOTECHS scores (rho=-0.267 (P=0.045)), but more strongly with surgical sub-team NOTECHS (rho=-0.412 (P=0.001)). Important effects of surgical and anaesthetic leadership on operative duration, of surgical situational awareness on technical errors, and of nursing leadership on procedural errors were seen.
The Oxford NOTECHS scale shows good validity and reliability, is able to evaluate whole teams and sub-teams, and reveals interesting insights into team behaviour when subteam and subscale data are analysed.
Non-Technical Skills for Surgeons (NOTSS)
S. Yule1, R. Flin1, N. Maran2, D. Rowley3, G. Youngson4,
1School of Psychology, University of Aberdeen Departments of 2Anaesthesia and 5Surgery, Royal Infirmary, Edinburgh, 3Royal College of Surgeons of Edinburgh, 4Royal Aberdeen Children's Hospital
Surgical patients may be involved in up to 60% of adverse medical events. Studies of behaviour in the operating room show that breakdown in non-technical skills such as team working, leadership, communication, situation awareness, and poor decision making are not uncommon and can lead to errors, poor outcomes, and higher compensation payouts. The irony is that non-technical skills are still not formally trained or assessed in surgical curricula and the evidence-base regarding which skills are most influential on surgical performance and outcome is only beginning to emerge. To start addressing this problem, we developed the NOTSS (Non-Technical Skills for Surgeons) behaviour rating system. The tool was developed using task analysis with subject matter experts and allows trained surgeons to structure observations and feedback on observed behaviours during the intraoperative phase of surgery. The NOTSS system is based on a skills taxonomy consisting of four categories (situation awareness, decision making, teamwork & communication, leadership) divided into twelve elements, each with example behaviours. The system has been tested in an experimental trial using video scenarios of simulated cases (n=44 surgeons participated). A usability trial (n=11 trainee-surgeon dyads) tested the feasibility of implementing NOTSS, and current research (n=22 surgical Specialist Registrars) is establishing the impact that regular feedback has on trainee surgeons' attitudes, behaviours and technical skills. This talk will briefly outline the development process and the main features of the system.
Assessing the surgical skills of trainees in the operating theatre
J. Beard1, H. Purdie2, J. Marriott3
1University of Sheffield and College of Surgeons, 2Royal Hallamshire Hospital, Sheffield, 3University of Sheffield.
The Procedure Based Assessment (PBA) has been adopted by the Intercollegiate Surgical Curriculum Project (ISCP) as the main workplace based assessment for surgical trainees (www.iscp.uk). The Objective Structured Assessment of Technical Skills (OSATS) is an alternative assessment tool, used by RCOG. There has been little validation of these work placed based assessment tools, especially regarding their transferability to a wide range of procedures and specialties.
There is increasing awareness that 'higher-order' skills such as situation awareness, decision making, team-working and leadership, should underpin technical proficiency. The NOTSS tool has been developed to evaluate the non-technical behaviour of surgeons in theatre using a rating system of observed behaviours.
To demonstrate the validity, reliability and user satisfaction of three methods of assessing the surgical skills of trainees in the operating theatre.
Subject Group & Sample Size
Trainee surgeons in Upper GI, Vascular, Orthopaedic, Cardiothoracic and Obstetric & Gynaecological Surgery
50 assessments for each of 10 index procedures within the five surgical specialties. The aim is to complete a total of 500 assessments over 16 months.
Methods of Working
Three assessments tools (PBA, OSATS and NOTSS) will be compared using direct observation in the operating theatre. Ten index procedures in five surgical specialties will be assessed to ensure that the methodology is not procedure or specialty specific. Assessments will be undertaken by the supervising consultant, an independent assessor(s) and consultant anaesthetist to study inter-rater variation. Videos of some cases will be reviewed by specialty experts to assess their fidelity.
The trainee will be provided with feedback on their surgical and non-technical skills by the supervising consultant/senior registrar. User satisfaction and educational impact will be studied using a follow up questionnaire sent to trainees and consultants one month after completion of assessments in theatre(Kirkpatrick's Model).
Recruitment began in June 2007 and we have completed assessments in Upper GI. Trainees and consultants have begun completing feedback questionnaires. We plan to perform an interim data analysis of the Upper GI assessments and feedback questionnaires. We are exploring the potential of using videos with structured assessment forms as a feedback tool.This study will evolve as work-place based assessments become more widely used within the new surgical curriculum
Methods for measuring team communication and coordination
Stephanie Guerlain†, Thomas Shin†, Beth Turrentine* and James Calland*
†Department of Systems and Information Engineering, University of Virginia
*Department of Surgery, University of Virginia School of Medicine, Charlottesville, VA USA
Two independent observers used a software system called RATE to observe and score operating teams during 48 laparoscopic cholecystectomy surgery procedures at the University of Virginia hospital. 1) Pre-defined tasks of interest were tracked using a checkbox list when the task was completed. 2) Technical and coordination errors were marked, based on a predefined (but expandable) list of errors. 3) Communications involving either the Surgery Attending (senior surgeon) or the Surgery Resident (junior surgeon) were tracked using a 4-column "pick list" format. Sample coded communications were: "Surgery Attending à Surgery Resident: Teaching-Anatomy" or "Surgery Resident à Scrub Nurse: Requesting-Tools". Thus, classes of conversations were predefined, with the 1st column defining the initiator of the conversation, the 2nd column indicating the recipient(s) of the conversation, the 3rd column defined the type of communication (e.g., teaching, requesting, discussing, socializing) and the 4th column defined the content of the communication (e.g., Anatomy, Surgical Approach, Tools, Equipment, Other Cases). Conversations could be annotated using a free-form textbox, however the scorers wished. This was usually a shorthand transcription.
An algorithm was developed to "align" the two lists of conversations as recorded by the 2 observers. Conversations that were marked by both observers were then compared to see if there was an exact or similar match to get a measure of inter-rater agreement. These inter-rater agreement scores can be compared to those obtained by observers scoring transcriptions of the conversations, completed after thefact. Thus, conclusions can be drawn as to the viability of "live" scoring vs. the added work of transcribing and then scoring.
A final measure of team communication and coordination was gathered by having each team member fill out a post-procedure questionnaire. This enables measuring "team situation awareness" (e.g., the overlap and lack thereof of knowledge about the happenings of the just completed procedure) and to collect demographics of the team members (including experience level and hours of sleep in the prior 24 hours).
This methodology was applied to measure and compare performance before and after certain interventions (e.g., crew resource management training). The methodology has also been adapted to other healthcare domains, specifically the content and quality of handoffs during inpatient rounds.
II. Using Behaviour Rating Tools in Clinical Practice
Incorporation of Anaesthetists' Non-Technical Skills system into a workplace based assessment tool
Scottish Clinical Simulation Centre/ NHS, Glasgow
The Anaesthetists' Non-Technical Skills system has received the support of the Royal College of Anaesthetists as an assessment tool to be used in the battery of workplace based assessment methods. However, the clinicians involved in the development of the system have noted reluctance on the part of those who have had some formal training in the use of system to apply it when working with trainees. Feedback from those who have had training suggests that they lack the confidence in trying the system because they find it quite daunting to use (F Dodds, personal communication). One personal solution to this is to incorporate the system into a system that assesses aspects of clinical performance with which clinical anaesthetists have more experience. The Mini Clinical Evaluation Exercise is one such assessment system that shows promise. An initial pilot session with anaesthetists in Auckland and Wellington suggested that clinical anaesthetists could use this system to identify and rate behaviours that came under the category of non-technical skills. A further study will be carried out using the amended Mini-CEX form and also obtaining feedback from the trainees on whom the exercise is being carried out and the consultants using the form. Special focus will be placed on the non-technical skills components of the assessment tool.
ANTS – Putting a tool into practical use
Jodi Graham, Emma Giles,Graham Hocking
Sir Charles Gairdner Hospital, Perth, Australia
Anaesthesia Non-Technical Skills (ANTS) is currently being investigated for use as a summative assessment tool by the Australian and New Zealand College of Anaesthetists (ANZCA). For use on a large scale, such a tool must be highly reliable, valid, acceptable and feasible. Our study aimed to investigate inter-reliability following an 8-hour training programme.
We have designed a one-day rater training package. Video footage of anaesthetists working in real time was produced specifically for this purpose. Our rater training was based on extensive research previously performed in this area. We trained 15 raters in one day using our programme and aimed to increase inter-rater reliability to a level of rwg>0.7. At the same time a rater training programme for ANTS has been developed.
At the time of writing this abstract, the study and data collection had not been fully completed. Complete results will be available for the meeting.
We expect ANTS to be a highly reliable assessment tool. Further research surrounding its introduction into practical use is yet to be conducted.
An attempt to equal rating of all OR team members´ team skills
Carl-Johan Wallin1Lisbet Meurling1, Leif Hedman1,2, Gunnar Söderdahl1, Li Tsai1
1Dept of Clinical Science, Intervention and Technology (CLINTEC),
Karolinska Institutet, Stockholm, Sweden. 2Dept of Psychology, Skill Acquisition Lab, Umeå University, Umeå, Sweden
In an earlier study we have shown that team training in full-scale medical simulation in emergency medicine improves team behaviours. Others have shown that effective teamwork can help decrease the number of small problems and prevent them from escalating to more serious situations during complex surgical procedures. Our hypothesis is that team training in full-scale medical simulation should improve (i) global team skills, (ii) medical behaviour, and, (iii) outcome for the simulated patient. However, in order to successfully evaluate training we must be able to rate team skills of all members of the team; physicians as well as nurses. Thus, we need an instrument feasible to rate both leaders´ and followers´ behaviour. In a pilot study on medical students training in emergency medicine scenarios we tested a recently developed behaviour rating scale for all team members. Our intention is to test this scale also in the operating theatre on full professionals. Liver transplantation is a complex surgical procedure. Residents or specialists in anaesthesia and transplantation surgery together with anaesthesia and operating nurses who will enter the transplantation organisation will be recruited to the study. Mixed groups of physicians and nurses will have standardized patient simulator training in anaesthesia and blood-saving procedures related to liver transplantation including pre-clamping, clamping, de-clamping, and management of massive bleeding. Video recordings for retrospective review of videotapes during team training for analysis of team skills and medical behaviour will be analyzed together with outcome for the simulated patient. Data will be compared and triangulated.
Anaesthetist's prospective memory
Danish Simulation Centre, Herlev University Hospital, Denmark
This study investigated failures of prospective memory (PM) as a relevant but neglected error type in medicine. A patient simulator was used to investigate PM failures. The influence of subjective importance and type of intention (educational, internal, external) on the (missed) execution of intention was investigated in a 262 design. The effects on missed executions by importance and type of intention were hypothesized. A total of 64% of unimportant and 80% of important intentions were executed 79% of educational 67% of external and 72% of internal intentions were executed. Neither difference was statistically significant. Interaction was significant for missed executions and for executions. Despite low statistical support and some methodological limitations, it was possible to show that PM failures are relevant to patient safety and that patient simulators are a suitable but so far unused tool for their investigation.
Simulator-based evaluation of clinical guidelines, exemplary of a modified rapid-sequence-induction (RSI) technique for infants as compared to the classic technique
C. Eich, A. Timmermann, S.G. Russo, H. Wagner-Berger & A. Nickut
Georg-August University, Göttingen, Germany
A valid evaluation of clinical guidelines in acute medicine frequently collides with ethical considerations, particularly when children are involved. Question is whether a paediatric patient simulator in an authentic clinical environment would be able to relieve this dilemma.
Rapid sequence induction (RSI) in infants is a time critical and potentially harmful procedure. The classic RSI technique comprises various difficulties: Inadequate preoxygenation of a wiggling child, prolonged apnoea without intermittent mask-ventilation, and cricoid pressure impeding endotracheal intubation. Hence most infants become more or less hypoxic which frequently triggers a cascade of subsequent incidents, such as bradycardia, increased rate of failed intubations, forced mask-ventilation with gastric distension and high stress levels for the anaesthetist.
With respect of these unfavourable effects, more and more professional bodies rather recommend a modified RSI technique: Shallow and rapid mask-ventilation after induction of apnoea with minimal airway pressure to achieve hyperoxia, unhurried intubation under complete neuromuscular blockade, and abandonment of cricoid pressure.
Despite inherent limitations we aim to prove that an evaluation of clinical guidelines in a simulator environment is generally feasible. We compare the modified RSI technique for infants with the classic technique by using an infant simulator in an operating room setting. We record critical events and unsafe actions (e.g. failed intubation, forced mask-ventilation) as well as stress markers of the performing anaesthetist (respiratory rate, minute volume, O2-consumption and heart rate, salivary cortisol and alpha-amylase).
Conflicts of interests:
The Göttingen SimTeam receives regular technical support for workshops, courses and research projects by Laerdal Medical Germany.
Current projects at McMaster University's Centre for Clinical Simulation: CanMEDS competencies and telemedicine
Centre for Clinical Simulation, McMaster University, Ontario, Canada
Our group is in the early stages of two separate studies involving the development of domain-specific behavioural markers. The first is a Royal College of Physicians and Surgeons of Canada (RCPSC) funded study to examine the relationship between CanMEDS and existing behavioural marker systems such as ANTS. CanMEDS is a system of 7 core competencies mandated by the College for all medical specialty trainees, and includes: expert clinician, communicator, collaborator, manager, health advocate, scholar and professional. Similar core competencies have been identified in other national governing bodies, yet teaching and evaluating such competencies has yet to be clearly defined. This project also explores the use of high fidelity simulation to train and evaluate those competencies in anesthesia and critical care. The second project uses high fidelity simulation to recreate certain healthcare settings for the purpose of analyzing team behaviour and communication. Most would agree that "real life" observations are more desirable than simulated recreations for conducting assessments of team performance; however, certain events or settings are too rare, too remote, or too difficult to observe directly.
III. Analysis of Behaviour in the Operating Theatre/ Anaesthetic Room
Adaptive coordination in anaesthesia teams
Michaela Kolbe, Gudela Grote, Barbara Künzle & Enikö Zala-Mezö
There is growing evidence for the importance of adaptive coordination in the face of changing demands on performance of medical teams, clear links between particular patterns of coordination and performance are difficult to establish, however, in our work, we attempt to fill this gap by analyzing coordination behavior and performance in routine and non-routine events based on a theory-based taxonomy of coordination behavior (previously validated in a comparative study on coordination of cockpit crews and anaesthesia teams; Grote, Zala-Mezö & Grommes, 2004) and a measure of technical team performance based on different reaction times in the face of non-routine events (e.g. Weinger, Slagle, & Jain, 2003).Data on coordination behavior and performance were obtained from 15 inductions of general anaesthesia in a simulated setting where after minor changes in blood pressure an asystole occurred. The taxonomy consists of three main groups: 1) explicit and implicit coordination (Entin & Serfaty, 1999; Wittenbaum, Vaughan, & Stasser, 1998) 2) leadership, and 3) heedful interrelating (Weick & Roberts, 1993). Explicit coordination involves devoting extra resources, such as communication, to organize activities (e.g. request information, ask for help). Implicit coordination occurs when actions harmonize based on a shared understanding of the task and the anticipation of other's needs for task fulfilment - therefore, little noticeable effort for coordination is required (e.g. provide unsolicited information, offer assistance). Leadership includes behaviours that are expressions of personal forms of coordination (or direct supervision) building upon the hierarchical structure of the team (e.g. give order, assign task). Heedful interrelating is a form of lateral coordination and includes the continuous (re)consideration of the situation where all team members work for the common team goal (e.g. consider other team members, consider the future). Results will be presented with a focus on the theoretical and empirical grounding of the taxonomy of coordination behaviors.
Observing communication and problem solving in critical incidents
Gesine Hofinger & Cornelius Buerschaper
University of Bamberg, Germany
Two tools for observing anaesthetists' communication and problem solving behaviour are described and compared. They were both developed and tested in a project on training anaesthetists non-technical skills when coping with critical incidents. Data were collected in the patient simulator of the University of Erlangen (Germany) using three different scenarios. The behavioural observations were categorised by psychologists of the University of Bamberg (Germany). Medical management was judged by expert anaesthetists.
One observation tool is for observing problem solving behaviour in teams. It is based on action psychology and comprises 24 items that allow to categorise every utterance during a scenario. The items are grouped into "action organisation", "team and process management", "conflict resolution". Additionally, formal characteristics of the utterance are categorised. Inter-coder reliability was trained until we reached 0.61-0.80 (Cohen's Kappa), depending on the scenario.
The other tool is a list of behavioural markers for adequate communication. As using general markers lead to no satisfying results we developed a list of markers specific for the scenarios used (16-22 items). For every phase of the incident a class of communication behaviours was defined that operationalise non-technical skills needed for a good management of the situation. Inter-coder reliability was 0.82% (Cohen's Kappa). The presentation will report some results and compare the tools.
An ethnographic approach to communication in anaesthesia teams
Royal Lancaster Infirmary
Our focus has been on how professional knowledge is acquired and used in anaesthesia. Our project 'The problem of expertise in anaesthesia' used ethnographic methods and brought together two clinical and two sociological perspectives in the research team . We have already published on communication routines in anaesthesia teams as one aspect of shared professional knowledge . Not only do the communicative routines tend to fall into specific types, but also that other members of the anaesthetic team would often join in. Although the talk is usually directed at the patient, it also allows staff to 'signal' to each other to enable the patient's safe passage through induction of general anaesthesia. Instances where the routines are broken can adversely affect patient safety.
My presentation will deal with aspects of method but will focus on some data we are currently preparing for publication dealing with handover in the recovery room. We observed 45 handovers taking place between 18 anaesthesiologists and 15 nurses. These were conducted in an environment that was time-pressured and prone to concurrent distractions (for instance, connecting monitoring, attending to the waking patient and operating equipment). Anaesthesiologists and nurses often had differing expectations of content and timing of information transfer. The anaesthesiologist's handover was typically brief, often condensing a complex case into a terse summary. The point at which the transfer of professional responsibility for the awakening patient was also highly variable and depended on the condition of the patient but also on the professional relationship between the nurse and doctor concerned. The transition was often marked by the anaesthetist's question 'Are you happy?' The informality of such handovers, and the use of 'tribal' coded markers, offers the potential for misunderstanding and contrasts with the standardised approaches adopted in high-risk industries.
1 Smith AF, Goodwin D, Mort M, Pope C. Expertise in practice: an ethnographic study exploring acquisition and use of knowledge in anaesthesia. British Journal of Anaesthesia 2003; 91: 319-28.
2 Smith AF, Pope C, Goodwin D, Mort M. Communication between anesthesiologists, patients and the anesthesia team: a descriptive study of induction and emergence. Canadian Journal of Anesthesia 2005; 52: 915-20.
Analyzing dialog: Balanced advocacy and inquiry
Dan Raemer, Jenny Rudolph, May Pian-Smith, Rebecca Dunkaillo Minehart, Toni Walzer, Roxane Gardner, Robert Simon
Center for Medical Simulation, Massachusetts General Hospital/
Harvard Medical School
When evaluating teamwork in crisis situations it is especially important to use measures that exploit a practical behavioral skill. The consequence of this approach is that team members can readily appreciate what is expected of them and implement a tangible behavioral change to improve teamwork. Rather than simply say, "improve communication" we want to provide a rubric for improving a verbal interaction in a particular way. The specific communication skill we focus upon is a balanced advocacy and inquiry with joint planning.
Our studies have analyzed dialog between two practitioners in a simulated crisis cases. We have chosen cases where there is a difficult decision to be made where each party has information known only to them and the consequences of actions by either party affect the other. Using retrospective video analysis, we have taken a short time segment one-minute prior and three minutes after the crisis onset for analysis. Two raters independently evaluate any statement made by either party and categorize that statement as an inquiry or an advocacy and then further whether the statement primarily provided information or formulated a plan of action. If the statement represented an advocacy that formulated a plan, it was further categorized as a "for self" or joint plan. A third rater resolved disagreement between raters.
We have found that this analysis method is easy to implement and is not time consuming because it focuses on a small number of statements during a short time segment. In one implementation of this technique on 50 tape segments we found the need to resolve coding differences as below 15%. Importantly, the results send a clear message to the parties as to their relative strengths and weaknesses in following a generic communication rubric in a crisis.