You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

MedHemo EARCP (Multimodal Medical Assistant)

This is a custom unified PreTrainedModel that acts as a single endpoint for a multimodal medical assistant.

Under the hood, it uses the EARCP (Ensemble Auto-Régulé par Cohérence et Performance) architecture to dynamically orchestrate three separate expert models:

Text Expert: google/medgemma-1.5-4b-it
Vision Expert: llava-hf/llava-1.5-7b-hf
Audio Expert: openai/whisper-large-v3

Usage (Requires `trust_remote_code=True`)

from transformers import AutoModel, AutoConfig

# Load the custom configuration and model
model = AutoModel.from_pretrained("amewebstudio/medhemo-earcp", trust_remote_code=True)

# Generate from text, audio, and/or image
result = model.forward(
    text="Quelles sont les complications de la drépanocytose?",
    image_b64="<base64_string>",  # Optional
    audio_b64="<base64_string>",  # Optional
    history=[]
)

print(result["response"])
print("EARCP Weights used:", result["earcp_weights"])

Architecture

This model does not contain a massive monolithic set of weights. Instead, it is a smart routing and ensembling wrapper. When forward() is called, it processes audio via the Whisper expert, vision via the LLaVA expert, and fuses the context into the primary medical LLM expert (MedGemma) using EARCP dynamic weighting.

Downloads last month: -