Integrating Large Language Models into Qualitative Methods in Health Services Research: A Proof-of-Concept Study (Preprint)

Source abstract

BACKGROUND Qualitative methods are widely used in health services research to understand human experiences and contextual factors. Traditional qualitative methods are valued for their depth of interpretation and ability to derive context-specific insights. Manual coding, a foundational technique in rigorous qualitative methods, is highly resource and time-intensive and difficult to scale up. Natural Language Processing (NLP), specifically Large Language Models (LLMs), have shown potential to enhance efficiency in qualitative analysis. However, there is limited research providing guidance and methodology for determining how best to integrate LLMs while maintaining rigor and trustworthiness of results. Specifically, methodologies are needed in which researchers are in conversation with the text and can review results for credibility. OBJECTIVE In this proof-of-concept study, we propose, apply, and evaluate a method using LLMs for automated coding, and present a use case specific to health services research. METHODS We analyzed 22 interviews among public health officials, law enforcement, community organizers, and medical professionals at one county in the State of California to examine existing service gaps and needs. A primarily deductive codebook was iteratively refined until two coders achieved an inter-coder reliability (ICR) > 0.95 and was applied to the transcripts using ATLAS.ti. We developed an NLP-assisted method that uses a semantic shift algorithm to segment transcripts which are then passed to GPT-4 for code assignment and explanation using the codebook and coding guidelines developed during the manual process. We conducted both a quantitative assessment of agreement between human coders and the NLP-assisted method, and a qualitative assessment of the soundness of the codes assigned by the NLP-assisted method. RESULTS The NLP-assisted method had moderate agreement with human coding (modified pooled Cohen’s Kappa = 0.66), and 71.8% of codes were rated as sound by reviewers. Sound codes were more often observed for high-certainty and straightforward codes, and when text chunks were semantically well defined. The NLP-assisted method had more difficulty with non-linear conversation and entity-dependent codes. Coding time was reduced significantly from ~40 hours for the traditional method to ~1 hour for the NLP-assisted method. CONCLUSIONS These findings suggest that LLMs can be effectively incorporated into qualitative processes while maintaining rigor if humans are embedded into the process. By maintaining a human-in-the-loop workflow, our methodology allows for researchers to maintain familiarity with the data, define the research question(s) and codebook, and determine if there are results that are not sound. By incorporating LLMs into the coding stage of the process, key limitations of traditional qualitative methods can be addressed, such as scalability, and resource and time limitations.