Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning

Guoxin Wang^1†, Jun Zhao¹, Xinyi Liu¹, Enyou Liu¹, Xuyang Cao¹, Chao Li¹, Zhuoyun Liu¹, Qintian Sun¹, Fangru Zhou¹, Haoqiang Xing¹, Zhenhong Yang^1
1JDH Algo, JD Health International Inc. ^†Project Leader

Abstract

Medical imaging provides critical evidence for clinical diagnosis, treatment planning, and surgical decisions, yet most existing imaging models are narrowly focused and require multiple specialized networks, limiting their generalization. Although large-scale language and multimodal models exhibit strong reasoning and multi-task capabilities, real-world clinical applications demand precise visual grounding, multimodal integration, and chain-of-thought reasoning. We introduce Citrus-V, a multimodal medical foundation model that combines image analysis with textual reasoning. The model integrates detection, segmentation, and multimodal chain-of-thought reasoning, enabling pixel-level lesion localization, structured report generation, and physician-like diagnostic inference in a single framework. We propose a novel multimodal training approach and release a curated open-source data suite covering reasoning, detection, segmentation, and document understanding tasks. Evaluations demonstrate that Citrus-V outperforms existing open-source medical models and expert-level imaging systems across multiple benchmarks, delivering a unified pipeline from visual grounding to clinical reasoning and supporting precise lesion quantification, automated reporting, and reliable second opinions.

Contribution

1. We construct a unified model that integrates detection, segmentation, and multimodal chain-of-thought reasoning, enabling pixel-level lesion localization, structured report generation, and physician-like diagnostic inference within a single model.

2. To facilitate reproducibility and support the research community, we release Citrus-V along with a curated open-source data suite, including a multimodal chain-of-thought reasoning dataset for report generation, a refined detection and segmentation benchmark with corrected labels, and a medical document understanding benchmark with graded difficulty levels.

3. We further design a novel multimodal training paradigm to accelerate convergence and enhance generalization across diverse imaging and reasoning tasks.

Comparison with Other Methods

Model Access

Model	Size	Ability	Download
Citrus-V	8B	Med-VQA, Med-TQA, Med-Doc, Med-IRG and Multimodal Chain-of-Thought Reasoning (MM-CoT).	Huggingface
	33B	Med-VQA, Med-TQA, Med-Doc, and MM-CoT.	Huggingface
	73B	Med-VQA, Med-TQA, Med-Doc, and MM-CoT.	Huggingface

Dataset Access

Dataset	Usage	Description	Download
MeCoVQA-G-Plus	Benchmark	MeCOVQA-G+ is mainly used to evaluate the medical image modality segmentation capability of medical multimodal language models, and it is built on the basis of the open-source dataset MeCOVQA-G through methods such as expert modification and review.	Huggingface
MedXray-CoT	Train Data	The MedXray-CoT dataset is primarily used to train large medical multimodal models to develop the ability, similar to that of physicians, to reason based on different anatomical structure regions under the X-ray modality.	Huggingface

Training Strategy

Concept Alignment: In this stage, most model parameters are frozen, and only the MLLM projector together with the vision encoder are updated. Training primarily relies on image-caption pairs, which establish a stable mapping from visual features into the language space without disrupting the pretrained reasoning ability of the LLM. This step provides a lightweight but essential initialization for subsequent multimodal alignment.

Comprehension Enhancement: All MLLM parameters, including the projector, vision encoder, and LLM, are trainable, while the segmentation projector, encoder, and decoder remain frozen. Training incorporates a broader range of tasks, reporting detailed and structured, interpreting of medical image are preserved to establish correlations between visual features, medical concept, radiological findings and imaging diagnostic. Additionally, scientific document comprehension data such as chart, diagram, are taken considering that explication and interpretation of scientific illustrations and graphs is necessary facing with clinical documentation, laboratory reports, diagnostic image annotations and radiographic markers. This stage strengthens the MLLM’s multimodal comprehension capacity, while freezing the segmentation modules prevents premature interference from segmentation supervision.

Instruction Fine-Tuning: In the instruction fine-tuning stage, the MLLM is trained on the most diverse instruction-following data, including report generation, text-only instructions, OCR, grounding, medical VQA, reasoning-based chain-of-thought tasks, and segmentation instructions. All MLLM parameters are updated, whereas segmentation modules remain frozen. It is worth noting that our empirical results show that directly combining segmentation loss with other tasks can damage VQA performance significantly. To this end, for segmentation samples, the supervision is restricted to textual outputs containing the special "[SEG]" token, without applying mask-level losses. This design allows the MLLM to acquire the discourse patterns needed for segmentation queries and encode segmentation intent into the hidden state of the "[SEG]" token.

Segmentation Fine-tuning: At this stage, all MLLM parameters are frozen, and optimization focuses exclusively on segmentation components, including the segmentation projector, the segmentation encoder, and the segmentation decoder. Unlike prior methods that freeze most of the segmentation backbone, the full SAM2 architecture, including the vision encoder, prompt encoder, and mask decoder, is fine-tuned to adapt effectively to medical imaging. With the MLLM frozen, training is computationally efficient, while full adaptation of SAM2 ensures precise and domain-specific pixel-level segmentation.

Case Study