Safe AI for Sensitive Data: Our NLI4CT SemEval 2024 Submission

Summary

Natural Language Models (NLMs) are rapidly being explored for their potential to analyse complex medical data. However, Clinical Trial Reports (CTRs) present a unique and high-stakes challenge. This data is not only complex—filled with dense medical terminology, acronyms, and numerical data, but it is also susceptible to errors. This necessitates a focus on safe and reliable NLMs.

Our Objective & Method

For the NLI4CT SemEval 2024 Task 2, our team tackled this challenge head-on. The goal was to build a system that could accurately perform entailment recognition on data from CTRs.

Our paper details an exploration of various advanced techniques to achieve this. We didn’t just pick one model; we investigated a full pipeline:

  • Data Pre-processing: We explored methods to help NLMs better understand and contextualise medical-specific terms and numerical values.

  • Model Evaluation: We compared the performance of multiple state-of-the-art models, including DeBERTa, BioLink BERT, GPT2, BioGPT, and Clinical BERT.

  • Training Techniques: We experimented with different training approaches, such as fine-tuning, prompt tuning, and contrastive learning, to find the most effective way to optimize performance.

What We Found

Our results demonstrate the effectiveness of this rigorous approach. The final proposed ensemble model, which combines the strengths of several different NLMs, achieved a highly competitive F1 score of 0.77. This performance secured our team the sixth rank on the official leaderboard.

Why This Matters

This paper contributes to the advancement of safe and effective NLMs for analyzing complex medical data. By focusing on robust training and evaluation, our work provides valuable insights into how these models can be safely deployed for real-world analysis of sensitive clinical trial data.

Click below to READ

You may also like

Scroll to Top