FALCON

Functional Assembly and Language for Compositional Reasoning in X-ray.

Paper · Project page · Code

This repository contains the completed Stage-3 model, including Vicuna-7B-v1.5, DINOv2-L/14, the trained RF-DETR segmentation detector, multimodal adapters, and unmerged LoRA weights.

Usage

Install the pinned dependencies from the code repository:

pip install 'falcon-x[train] @ git+https://github.com/yonathan-kiflom/FALCON.git'

Use JonathanJMK/FALCON or a local downloaded model directory. Authenticate with huggingface-cli login when accessing the private repository. Review the custom code before trusting it; pin a Hub commit with revision= for reproducible use.

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "JonathanJMK/FALCON", trust_remote_code=True,
).to("cuda").eval()

result, masks = model.predict(
    image="image.png", prompt="Describe the image.", max_new_tokens=256,
)
print(result["answer"])

Use predict_segmentation for grounding prompts and predict_panoptic for panoptic prompts. Tokenization, image preprocessing, and prompt formatting are included. The model supports one device; automatic multi-device dispatch and quantized loading are not supported. The saved per-component precision is preserved; do not cast the entire model to half precision.

Evaluate using the same model:

python -m falcon evaluate run --model JonathanJMK/FALCON \
  --dataset /path/to/falcon-x --split test --tasks all \
  --run-dir runs/falcon-evaluation --device cuda

Scope and limitations

Trained on falcon-x for X-ray descriptions, questions, component presence/completeness, instance grounding, and segmentation. Generated answers and masks can be wrong; this is a research model, not a certified screening system. Available structured heads are declared in config.json; untrained risk and physical-link heads are not presented as predictions. Counterfactual completeness does not establish real-world danger or physical connectivity.

The package preserves the trained architecture and LoRA for further research. It is not an optimizer-state checkpoint, and loading with Transformers alone does not make the repository's stage-training CLI a general fine-tuning tool. Task metrics and evaluation limitations are documented in the code repository.

Licenses

Vicuna is derived from Llama 2 and retains the Llama 2 Community License and Acceptable Use Policy. Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.

DINOv2 and RF-DETR segmentation 1.5.2 use Apache-2.0. FALCON code is Apache-2.0 (LICENSE-code); this does not relicense upstream model weights. Retain LICENSE-Llama-2, LICENSE-code and Notice when redistributing the full package.