Large language models like GPT-4 have demonstrated remarkable capabilities in various domains, including medicine. This study evaluates GPT-4 on medical competency examinations, such as the USMLE, and benchmark datasets like MultiMedQA. The research examines GPT-4's performance without any specialized prompt crafting and investigates the model's ability to predict the likelihood that its answers are correct.

Results show that GPT-4 exceeds the passing score on USMLE by over 20 points, outperforming earlier general-purpose models like GPT-3.5 and models fine-tuned on medical knowledge, such as Med-PaLM. The study also finds GPT-4 to be better calibrated than GPT-3.5, which is crucial for high-stakes applications like medicine. A case study explores GPT-4's ability to explain medical reasoning, personalize explanations for students, and interactively craft new counterfactual scenarios around medical cases.

These findings suggest that GPT-4 has the potential to be applied in medical education, assessment, and clinical practice. However, it is essential to consider the challenges of accuracy and safety when utilizing GPT-4 in the medical field.