README.md

---
license: mit
library_name: open_clip
pipeline_tag: feature-extraction
base_model_relation: finetune
language:
- en
tags:
- clip
- open-clip
- vision-language
- image-text-retrieval
- cross-modal-retrieval
- long-context
- hyperbolic-learning
- hyperbolic-geometry
- lorentz-model
- eccv-2026
---

<h1 align="center">HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding</h1>

<div align="center">

**ECCV 2026**

</div>

• [Project Page](https://janeyeon.github.io/hyflclip/)  
• [Paper](https://arxiv.org/abs/2607.00428)  
• [Code](https://github.com/janeyeon/hyfl-clip)


## Overview

HyFL-CLIP improves the **robustness of long-context vision-language understanding** by transferring OpenCLIP’s pretrained Euclidean alignment into a **hyperbolic representation space**.
The model reports improvements of up to **19.5%** in long-text cross-modal retrieval under perturbations including:

- caption reordering;
- caption summarization;
- caption condensation; and
- partial text deletion.

## Model Description

**HyFL-CLIP** is a hyperbolic fine-tuning framework that improves the robustness of CLIP to long and compositional text descriptions.

Standard CLIP models are primarily trained using short captions and rely on Euclidean contrastive learning. Their image–text alignment can therefore degrade when long descriptions are reordered, summarized, condensed, or partially deleted.

HyFL-CLIP addresses this limitation by projecting OpenCLIP representations into **hyperbolic space**, specifically the Lorentz model. Hyperbolic geometry allows the model to represent hierarchical semantic relationships among token-level information, partial descriptions, short captions, long-form descriptions, and images.

This checkpoint is fine-tuned from an **OpenCLIP** pretrained model.

## Method

HyFL-CLIP incorporates three main components.

### Cross-Manifold Similarity Distillation

Similarity knowledge from a pretrained Euclidean OpenCLIP model is distilled into a hyperbolic student model. This preserves the alignment knowledge of the pretrained model while changing the underlying representation geometry.

### Hierarchical Semantic Modeling

Summarized tokens, short captions, long-form descriptions, and images are embedded in a shared hyperbolic space. This enables the model to represent part–whole and summary–detail relationships.

### Einstein Midpoint Aggregation

Token- and phrase-level representations are aggregated using the Lorentzian centroid, also known as the Einstein midpoint. This provides a geometry-consistent method for combining partial representations into whole-level representations.


## Results

The evaluation covers:

- long-context cross-modal retrieval;
- retrieval under caption perturbations;
- intra-modality retrieval;
- short-text cross-modal retrieval; and
- integration with downstream generators such as Stable Diffusion XL.

Please refer to the [paper](https://arxiv.org/abs/2607.00428) for the complete quantitative results and ablation studies.

## Evaluation Datasets

HyFL-CLIP is evaluated on the following datasets:

| Dataset    | Evaluation setting                        |
| ---------- | ----------------------------------------- |
| Urban1K    | Long-context image–text retrieval         |
| DOCCI      | Dense and compositional caption retrieval |
| Long-DCI   | Long-description image–text retrieval     |
| ShareGPT4V | Detailed vision-language descriptions     |
| COCO       | Short-text image–text retrieval           |
| Flickr30K  | Short-text image–text retrieval           |

The datasets are not distributed as part of this model repository. Users must obtain them from their respective sources and comply with their individual licenses and terms of use.

## Installation

Clone the official HyFL-CLIP repository and prepare the required environment.

```bash
git clone https://github.com/janeyeon/hyfl-clip.git
cd hyfl-clip
```

Install the required dependencies:

```bash
pip install torch torchvision
pip install open_clip_torch
pip install huggingface_hub
```

Please refer to the official code repository for the complete environment configuration.

## Download

Two HyFL-CLIP checkpoints are available:

- `HyFL-CLIP_ViTB.pt`: HyFL-CLIP with a ViT-B backbone
- `HyFL-CLIP_ViTL.pt`: HyFL-CLIP with a ViT-L backbone

The checkpoints can be downloaded using `huggingface_hub`:

```python
from huggingface_hub import hf_hub_download

vit_b_checkpoint = hf_hub_download(
    repo_id="jeeit17/HyFL-CLIP",
    filename="HyFL-CLIP_ViTB.pt",
)

vit_l_checkpoint = hf_hub_download(
    repo_id="jeeit17/HyFL-CLIP",
    filename="HyFL-CLIP_ViTL.pt",
)

print("ViT-B checkpoint:", vit_b_checkpoint)
print("ViT-L checkpoint:", vit_l_checkpoint)
```

## Usage

HyFL-CLIP checkpoints should be loaded using the model architecture and Lorentz-geometry implementation provided in the official HyFL-CLIP codebase.

A general checkpoint-loading pattern is shown below:

```python
import torch

checkpoint = torch.load(
    "hyfl_clip.pt",
    map_location="cpu",
)

model = build_hyfl_clip_model()

if "state_dict" in checkpoint:
    model.load_state_dict(checkpoint["state_dict"])
else:
    model.load_state_dict(checkpoint)

model.eval()
```

The model must be initialized using the same configuration used during training, including:

- OpenCLIP backbone architecture;
- pretrained OpenCLIP checkpoint;
- text context length;
- embedding dimension;
- hyperbolic curvature;
- checkpoint state-dictionary format; and
- image and text preprocessing configuration.

Please refer to the official HyFL-CLIP repository for the exact model initialization and inference code.

## Evaluation

Cross-modal retrieval evaluation is implemented in `eval/retrieval/`.

Run the provided evaluation script:

```bash
bash eval.sh
```

The underlying command follows this format:

```bash
python eval/retrieval/unified_eval.py \
    --ckpt_path /path/to/hyfl_clip.pt \
    --datasets urban1k docci dci sharegpt4v
```

Supported long-context evaluation datasets include:

```
urban1k
docci
dci
sharegpt4v
```

Before running the evaluation, update the dataset `image_root` and `json_path` fields in `eval/retrieval/unified_eval.py`.

Dataset preparation instructions are available from:

1. [Urban1K](https://github.com/beichenzbc/Long-CLIP)
2. [TULIP](https://github.com/ivonajdenkoska/tulip) for Long-DCI
3. [DOCCI](https://google.github.io/docci/)

## Training

Training is performed using `train/train.py` with distributed PyTorch execution.

Run the provided launcher:

```bash
bash train.sh
```

The launcher internally uses `torchrun`. Update `CUDA_VISIBLE_DEVICES` and `--nproc_per_node` according to the target machine.

Checkpoints are saved in the following format:

```
output/ckpts/<experiment_name>/ep=<epoch>_hyfl.pt
```

## Intended Uses

HyFL-CLIP is intended for research involving:

- long-context image–text retrieval;
- robust cross-modal retrieval;
- compositional vision-language understanding;
- hierarchical multimodal representation learning;
- hyperbolic representation learning;
- retrieval under textual perturbations;
- long-caption image search; and
- multimodal representation analysis.

## Out-of-Scope Uses

HyFL-CLIP is not designed or validated for:

- safety-critical decision-making;
- medical diagnosis;
- legal or financial decision-making;
- biometric identification;
- surveillance;
- autonomous content moderation; or
- applications requiring guaranteed robustness or fairness.

Retrieval performance should not be interpreted as general-purpose long-context reasoning ability.

## Authors

- [Ji Ha Jang](https://jeeit17.github.io)<sup>1*</sup>
- [Hayeon Kim](https://janeyeon.github.io)<sup>1*</sup>
- [Chulwon Lee](https://github.com/signalee95)<sup>2</sup>
- [Junghun James Kim](https://github.com/jongheean11)<sup>2</sup>
- [Se Young Chun](https://icl.snu.ac.kr/pi)<sup>1,2,3</sup>

<sup>1</sup> Department of Electrical and Computer Engineering, Seoul National University
<sup>2</sup> Interdisciplinary Program in Artificial Intelligence, Seoul National University
<sup>3</sup> INMC and AIIS, Seoul National University

\* Equal contribution

## Acknowledgements

HyFL-CLIP builds on:

- [OpenCLIP](https://github.com/mlfoundations/open_clip);
- [LongCLIP](https://github.com/beichenzbc/Long-CLIP);
- [HiMo-CLIP](https://github.com/UnicomAI/HiMo-CLIP) and
- prior work on hyperbolic representation learning using the Lorentz model.

We thank the authors and maintainers of these projects for making their research, models, and implementations publicly available.

## License

HyFL-CLIP is released under the **MIT License**. See the [LICENSE](./LICENSE) file for the full license text.

This model is a fine-tuned version of an OpenCLIP pretrained model. Users are responsible for complying with the license terms applicable to:

- the specific pretrained OpenCLIP checkpoint;
- incorporated third-party source code;
- training and evaluation datasets; and
- other third-party components used with the model.

The relevant upstream copyright and license notices must be preserved in accordance with their respective license terms.

The MIT License applied to HyFL-CLIP does not replace or override separate licenses, attribution requirements, or usage restrictions applicable to third-party checkpoints, code, or datasets.

## Citation

Please cite the following paper when using HyFL-CLIP:

```bibtex
@misc{jang2026hyflclip,
  title         = {HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding},
  author        = {Ji Ha Jang and Hayeon Kim and Chulwon Lee and Junghun James Kim and Se Young Chun},
  year          = {2026},
  eprint        = {2607.00428},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2607.00428}
}
```