Are Small Language Models Enough for Biomedical QA Tasks?

Abstract

This paper presents a specialized fine-tuning approach for the Mistral-7B Large Language Model (LLM) tailored for biomedical applications. We employ Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, to adapt the model to the intricacies of biomedical language and domain-specific knowledge. By integrating LoRA, we aim to preserve the general language understanding capabilities of Mistral-7B while enhancing its performance on biomedical tasks. The fine-tuning process involves training the model on the PubMedQA dataset. Our experiments demonstrate that the fine-tuned Mistral-7B model achieves notable accuracy, 60%. This performance is particularly significant given the relatively modest size of the Mistral-7B model compared to other approaches that often require larger models to achieve comparable results. The results highlight the effectiveness of LoRA in fine-tuning large language models for domain-specific applications, particularly in the biomedical field, where precise and contextually accurate language understanding is crucial. This work contributes to the advancement of AI in healthcare by providing a robust and efficient method for adapting LLMs to biomedical applications, demonstrating that high precision can be achieved with a smaller model size.