SA2 - The 2 am Page Versus the 2 am ChatGPT: How Accurately Do Large Language Models Triage Oral and Maxillofacial Surgery Emergencies When Prompted as Clinicians Versus Patients?

Abstract: Large language models (LLMs) are an increasingly popular tool among both clinicians seeking decision support and patients self-triaging symptoms; yet, whether or not these models provide safe, accurate triage recommendations for oral and maxillofacial surgery (OMS) emergencies, and whether the framing of the query affects the response, remains unknown. This study evaluated triage accuracy, management quality, and safety across three frontier LLMs when the same clinical scenarios were presented from a perspective of either a physician or a patient.

Twenty-five clinical scenarios spanning 5 categories were written, including postoperative complications (n = 5), emergency department consults (n = 5), referring dentist calls (n = 5), direct patient questions (n = 5), and intraoperative anesthesia emergencies (n = 5). Each scenario was presented to 3 frontier LLMs (OpenAI GPT-5.4, Anthropic Claude Opus 4.6, Google Gemini 3.1 Pro) in 2 distinct framings: 1) a physician consult call containing structured clinical data, and 2) a patient self-report containing the same clinical information embedded in colloquial language with emotional content, imprecise anatomical descriptions, and buried critical findings. Anesthesia emergencies were tested in the physician frame only (n = 15). Each response was scored by three independent raters (1 OMS attending, 2 OMS residents) on triage accuracy (emergent, urgent, routine), critical management elements (physician frame only), dangerous recommendations, and calibration of stated confidence. Inter-rater reliability was excellent (Fleiss' kappa = 0.875, 95% unanimous agreement). The primary analysis used McNemar's exact test to compare paired triage accuracy between framings.

Across 135 total runs, overall triage accuracy was 80% in the physician frame and 88% in the patient frame. By model, physician-frame accuracy was 84% (GPT-5.4), 84% (Claude Opus 4.6), and 72% (Gemini 3.1); patient-frame accuracy was 85%, 95%, and 85%, respectively. McNemar's test did not reach significance for any model (all P >= .125), though a consistent directional trend favored the patient frame. Mean management element accuracy in the physician frame was 86% (GPT-5.4), 91% (Claude Opus 4.6), and 83% (Gemini 3.1). Dangerous recommendations were identified in 7% of all responses (10/135), with Gemini producing the highest rate (20% in physician frame) compared to Claude Opus 4.6 (12%) and GPT-5.4 (4%). All three models correctly triaged all 5 anesthesia emergencies, but dangerous recommendations appeared in 6 of the 15 anesthesia emergency responses. Examples included recommending deepening sedation during laryngospasm and administering full-dose naloxone to an elderly patient with congestive heart failure. The most frequent error types were over-triage to the emergency department (20 instances) and dangerous specific recommendations (8 instances). Notably, no dangerous recommendations occurred in any patient-frame response from GPT-5.4 or Gemini, suggesting models may exercise more caution when addressing patients directly. Additionally, response verbosity varied substantially. Claude generated a median of 2,306 tokens in physician-frame responses versus 734 for GPT-5.4. Expected calibration error was lowest for Claude (0.028), moderate for GPT-5.4 (0.048), and highest for Gemini (0.236), indicating substantial overconfidence in Gemini's self-assessed certainty.

These findings demonstrate that frontier LLMs provide clinically useful but imperfect triage recommendations for OMS emergencies. Patient-framed queries produced equal or higher triage accuracy than physician-framed queries across all models, suggesting these models may be more robust to informal clinical language than expected. However, the persistence of dangerous recommendations, including in correctly triaged anesthesia emergencies and model-specific miscalibration of confidence raise important safety concerns for both clinical decision support and direct patient-facing applications.