Objective Evaluation of Sonographic Skill in FAST Examinations

With the recent uptake of competency-based medical education, there has been extensive study of objective skill evaluation for needle-based and laparoscopic interventions on phantoms simulating human anatomy. In these contexts, it has been shown that skill evaluation metrics can distinguish novice from expert operators. Upon further validation, these metrics could be used to supplement expert knowledge in the medical education process, providing standardization and feedback during unsupervised training.

Sonographic skill, however, has received significantly less attention and studies have been limited to motion economy metrics. Here, a method is proposed for objective evaluation of sonographic skill by assessing whether relevant anatomical structures are scanned. To test this method, Focussed Assessment with Sonography for Trauma (FAST) human will be considered. FAST is a quick screening test for internal bleeding in the pericardium and abdominal cavity after trauma, for which ultrasound skill is of great importance. Because ultrasound has no known adverse effects on humans, the FAST examinations can be performed on a human volunteer, rather than a phantom, thus improving the system’s fidelity.

A healthy volunteer subject was recruited to be scanned by ultrasound. Fourteen novice residents and fifteen residents with experience in FAST performed the examination on the volunteer. The participants were asked to perform a complete FAST examination, scanning each of the four regions of interest (hepatorenal, splenorenal, pericardial, and pelvic regions) to whatever extent they believed to be sufficient for a complete assessment (Fig. 1). Their ultrasound probe motion was tracked relative to a reference sensor affixed to the volunteer’s back, and their ultrasound image sequence was recorded. The tracking and image recording was performed separately for each region using the open-source SlicerIGT (www.slicerigt.org) with the embedded PLUS software library (www.plustoolkit.org). Fig. 1. Photograph (left) and schematic diagram (right) of experimental setup for tracking and recording ultrasound image sequences for the FAST examinations.

To evaluate whether each participant scanned all of the relevant structures in each region of interest, an expert scanned each region. Subsequently, the expert reviewed their ultrasound image sequence using the open-source Multi-Dimensional Data extension for 3D Slicer (www.slicer.org). They indicated points of interest (POIs) in 3D space on their ultrasound sequence which should be scanned during any sufficiently thorough FAST examination (Fig. 2). Between five and twelve POIs were identified for each region. To calculate whether a resident user scanned a POI, the ultrasound plane’s sweep was reconstructed using the tracking information and a prior ultrasound calibration using SlicerIGT. Any POI intersected by a plane in the user’s ultrasound sweep was considered to be scanned. A small 10mm threshold was allowed around each POI to account for motion due to breathing, deformation, or ultrasound beam width.

The percentage of POIs scanned was used as a metric for sonographic skill evaluation. This metric is automatically calculated for each participant by analyzing their recorded probe trajectories using the open-source Perk Tutor software (www.perktutor.org). Additionally, the time of procedure and path length of the probe were calculated by Perk Tutor. All data was assessed for normality and subsequently analyzed using a Wilcoxon rank sum test. Fig. 2. 3D view (left) and ultrasound slice view (right) of user scanning POIs in the hepatorenal region, visualized using the 3D Slicer platform. POIs are indicated in red.

In total, the novice group scanned a lesser percentage of POIs than the experienced group, with a median (interquartile range) of 50% (10% – 86%) vs. 100% (83% – 100%), and this difference was significant (p<0.001). Furthermore, relative to the novice group, the experienced group scanned these POIs in significantly less time (62s, 47s – 89s, vs. 43s, 27s – 69s, p<0.001) and had significantly shorter path length (2113mm, 1199mm – 2902mm, vs. 892mm, 619mm – 1804mm, p<0.001).

The proposed metric for objectively evaluating sonographic skill, based on the percentage of POIs scanned, was demonstrated to correlate with experience of residents performing FAST examinations. Furthermore, experienced residents performed these examinations in shorter time and had less ultrasound probe motion. This provides initial evidence that the percentage of POIs, as an objective metric, can be used to determine skill level in diagnostic ultrasound examinations. These results, respectively, suggest that experienced residents perform both more complete and more efficient FAST examinations, though this has not been validated in the current study.

The system used for measuring objective evaluation metrics for the FAST examinations was composed of commodity hardware and completely open-source software. As a result, the setup can be reproduced rapidly and inexpensively. Pending further validation experiments, objective evaluation metrics could be implemented in the context of competency-based medical education programs for training in FAST examinations. Additionally, they may be applicable to evaluation of sonographic skill in similar procedures.

Related Publication

Bell, C., McKaigney C., Holden M. S., Fichtinger G., & Rang L. (2017). Sonographic accuracy as a novel tool for point of care ultrasound competency assessment. Academic Emergency Medicine Education and Training. 1, 316-324.

Holden, M. S., Ungi T., McKaigney C., Bell C., Rang L., & Fichtinger G. (2015). Objective Evaluation Of Sonographic Skill In Focussed Assessment With Sonography For Trauma Examinations. Computer Assisted Radiology and Surgery, 29th International Congress and Exhibition. 10(Supplement 1), S79-S80.