Medical Student NYU Grossman School of Medicine Green Village, New Jersey, United States
Introduction: Blameless culture, constructive feedback, effective communication, and teamwork are key to improving medical culture and subsequently patient care. Perceptions of culture differ among team members and currently there is no formal standardized method of evaluation. We created a novel Quality Culture Score to grade the quality culture of a surgical team based on perioperative communication through a mobile platform.
Methods: The frequency and sentiment of communication between faculty and residents was captured in a perioperative communication platform and weighted. Three independent reviewers manually rated the positivity of text exchange in 680 neurosurgery cases on a scale of 0 to 3, with 0 representing negative sentiment, 1 neutral, 2 positive, and 3 very positive. A machine learning algorithm was developed to rate sentiment using natural language processing.
Results: Sentiment-specific interobserver reliabilities for negative, neutral, positive, and very positive sentiments were 80%, 60%, 34%, and 57% respectively. The algorithm had an overall sensitivity of 0.68 (SD 0.02), positive predictive value of 0.70 (SD 0.02), and F1-score 0.68 (SD 0.01). Sensitivity was higher for negative sentiment scores (0.90, SD 0.03, p < .001) vs. very positive sentiment scores (0.51, SD 0.02, p < .001). Similarly, the positive predictive value was highest for negative sentiment scores (0.94, SD 0.01, p < .001). Overall, accuracy was highest for low sentiment scores (F1 0.92, SD 0.02, p < .001), and lowest for very positive scores (F1 0.51, SD 0.05, p < .001).
Conclusion : The quality culture of an institution or team governs quality of care. Our Quality Culture Score evaluates the culture of surgeons utilizing AI technology to enable automation. The algorithm performed best in the evaluation of negative culture, which is arguably the most important metric when determining improvement needs. By capturing and automatically analyzing communication, culture may not only be evaluated, but improved. Most importantly, these improvements will lead to an increased quality of patient care.
