Fluffy-LoRA πΆπΆπΆ
One body, three heads. Text, image, audio β one embedding space.
Named for Fluffy, Hagrid's three-headed dog (Cerberus's better-tempered nephew). Each head is an input modality; the body is a single shared embedding space.
No bolted-on encoders β no CLIP/CLAP/Whisper glued together with projectors.
google/gemma-4-12b-it natively ingests all three modalities into one backbone;
we QLoRA the language tower (r=8, Ξ±=16, ~32.8M params), freeze the native towers,
and pool the last token into one L2-normed embedding. Trained with symmetric
InfoNCE (Ο=0.02) against frozen, ratcheted retrieval evals.
The honest scoreboard and all code live in the GitHub repo: β https://github.com/SEBK4C/Fluffy-LoRA
Published adapters
fluffy-text-v0/ β alpha artifact from the abandoned v1 text-only run
This adapter is published as a research artifact, not because it passed the ratchet β the ratchet rejected every checkpoint of the run it came from. Full disclosure:
- What it is: LoRA (r=8, Ξ±=16, q/k/v/o/gate/up/down projections, ~32.8M
params) on the language tower of
gemma-4-12b-it, trained with symmetric InfoNCE (Ο=0.02, in-batch negatives, last-token pooling, L2 norm, max_length 512) on 40,941 fully self-synthetic text cards (no scraped or third-party data β rights-clean). - The run was abandoned: planned as 14 days, stopped by operator decision
at
0.4 days wall time, step 1449 of a schedule sized for 200,000 steps (56k were actually achievable in the window). The learning rate never left warmup (2.5e-5 of the 1e-4 peak at stop); the cosine schedule never annealed. - Training loss fell ~5.5Γ (9.36 β ~1.7) β¦
- β¦ but retrieval on our frozen eval did not move: R@1 0.010 vs 0.008 base-model baseline (β random) on the frozen G0 card-retrieval eval, n_pool = 1500. Loss-eval decoupling is the run's headline empirical result.
- Post-mortem found the root cause: gemma's tokenizer pads left, and
the training/eval code pooled at
attention_mask.sum(1)-1(a right-padding assumption) β so every sequence that wasn't the longest in its batch was pooled at a padding position. Two unrelated texts pooled this way sit at cos 0.96 (their true last-token embeddings: cos 0.75). The adapter was trained on a mostly-collapsed embedding function.
Benchmarks (mteb 2.18.0, identical harness for all contenders)
nDCG@10 for retrieval, Spearman for STS. "fixed pool" = corrected last-token
pooling (h[:, -1] under left padding). Reference = Qwen3-Embedding-8B with
its own card protocol β it reproduces its published MTEB scores on this
harness, which validates the pipeline.
| Task | base gemma | base + this adapter | base (fixed pool) | + adapter (fixed pool) | reference |
|---|---|---|---|---|---|
| SciFact | 0.000 | 0.004 | 0.002 | 0.002 | 0.788 |
| NFCorpus | 0.013 | 0.010 | 0.009 | β | 0.414 |
| FiQA2018 | 0.000 | 0.000 | β | β | 0.612 |
| STSBenchmark | 0.021 | 0.034 | 0.035 | β | 0.935 |
| STS17 (en-en) | 0.357 | 0.295 | 0.315 | β | 0.957 |
Two honest take-aways: (1) the adapter moved no external metric β train it as we did and you learn the training data's domain, not transferable embedding geometry; (2) raw decoder last-token (or mean) embeddings of gemma-4-12b-it are near-unusable for retrieval without contrastive adaptation β the reference model's scores on the same harness show what a properly-trained embedder does. Full analysis and raw results: LEARNINGS-V1.md.
Treat it as a checkpoint of scientific interest (what ~1.4k steps of contrastive warmup on a broken pooling fn does to a decoder LM's embedding geometry), not as a useful embedding model.
Usage
import torch, torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
from peft import PeftModel
base = AutoModel.from_pretrained("google/gemma-4-12b-it", torch_dtype=torch.bfloat16)
if hasattr(base, "language_model"):
base = base.language_model # text tower only, as trained
model = PeftModel.from_pretrained(base, "SEBK4C/Fluffy-LoRA", subfolder="fluffy-text-v0").eval()
tok = AutoTokenizer.from_pretrained("google/gemma-4-12b-it")
enc = tok(["a query"], padding=True, truncation=True, max_length=512, return_tensors="pt")
h = model(**enc).last_hidden_state
idx = enc["attention_mask"].sum(1) - 1 # last real token
emb = F.normalize(h[torch.arange(h.shape[0]), idx].float(), dim=-1)
Status: raw alpha β building in public
v1 (text-only) is over; v2 (three-lane multimodal) is being built in the open in the GitHub repo. Future checkpoints land here only when they beat the frozen evals by more than Ξ΅ β the ratchet rejects everything else.
- Downloads last month
- -