This project was in partnership with the Johns Hopkins Dermatology department to evaluate the accuracy of GPT-4 vision in evaluating dermatology images.

The goal was comprised of several folds:

1) Understand the current progress in vision AI in performing medical diagnostics,

2) Compare that with expert Dermatologist,

3) Perform segment analysis to understand which categories of images or types of diseases does vision AI perform best and worst in, and lastly,

4) Understanding prompting techniques to improve performance. 

This project required a robust and accurate method of scraping, cleaning and pre-processing dermatological images from specific publicly available sources. The team used these images with specialized few-shot and chain-of-though prompting to GPT-4 vision API and save the responses. Similar images were also evaluated by a team of Johns Hopkins Dermatologists as human expert reference. A custom AI model was built to quantify these qualitative evaluations and compared across AI vs human experts and through segmentation analysis. 

Research paper to be published within early 2024