senku21x/cot-oracle-jlens

Qwen3-8B Jacobian Lens (jlens)

Jacobian lens fitted for Qwen/Qwen3-8B, following the Anthropic jlens reference and cot-oracle layer selection (oracle injection layers 9/18/27 plus AO band 21–25).

Fit configuration

Setting	Value
Model	Qwen/Qwen3-8B
Corpus	HuggingFaceFW/fineweb (sample-10BT)
Final fit prompts	1000 (4-node shard: 250 × 4, --corpus-skip 0 / 250 / 500 / 750)
Pilot fit	100 prompts (4 × 25), used for early read/fingerprint below
Source layers	9, 18, 21, 22, 23, 24, 25, 26, 27
Target layer	35 (final block, default)
prompt_batch / dim_batch	16 / 16
max_seq_len	128
skip_first	16 positions
dtype	bf16
Hook point	Decoder block output (forward hook, tuple[0])

Compute: CSCS Clariden GH200, 4 nodes × 4 GPUs, device_map=auto per node, torch 2.13.0+cu130. Full-fit wall time ~1h14m per shard (parallel across nodes).

Known-answer read (sanity check)

Run after pilot fit (N=100). Prompt:

Fact: The currency used in the country shaped like a boot is

At the final token ( is), model logits (logit lens): the, called, Euro, euro, EUR, …

jlens readout at layer 27: top tokens include 欧元 (Euro), currency, euros, Euros — consistent with the expected answer (Italy → Euro).

At mid layers, readout picks up Italy/geography tokens (e.g. layer 18–26: 意大利, Italy, Italian), which matches the “boot-shaped country” cue before the currency slot.

Full per-layer table: Qwen3-8B/known_answer_read.json.

Fingerprint (layer curves)

Also run on pilot lens (N=100), 32 prompts, corpus_skip=100000.

Layer	top-1 agreement	top-5 agreement	kurtosis
9	1.4%	5.0%	1.26
18	1.1%	3.6%	1.01
21	1.1%	3.6%	0.94
22	2.4%	6.9%	0.97
23	5.7%	12.8%	1.20
24	7.5%	17.0%	1.27
25	9.6%	21.0%	1.33
26	10.9%	23.6%	1.43
27	14.4%	28.0%	1.61

Suggested workspace band: layers 9–27 (heuristic: kurtosis ≥ 0.5×peak, top1 ≥ 0.5).

Plot: Qwen3-8B/fingerprint.png. Raw curves: Qwen3-8B/fingerprint.json.

Files in this repo

Path	Description
Qwen3-8B/lens.pt	Merged Jacobian lens (N=1000)
Qwen3-8B/lens.meta.json	Fit conventions sidecar (required for correct apply)
Qwen3-8B/known_answer_read.json	Italy/Euro readout (pilot lens, all source layers)
Qwen3-8B/fingerprint.json	Layer diagnostic curves (pilot lens)
Qwen3-8B/fingerprint.png	Fingerprint plot
Qwen3-8B/results_summary.md	This file

Load

from jlens.lens import JacobianLens

lens = JacobianLens.from_pretrained(
    "senku21x/cot-oracle-jlens",
    filename="Qwen3-8B/lens.pt",
)

CLI readout example:

python -m jlens.read_cli \
  --lens senku21x/cot-oracle-jlens --lens-filename Qwen3-8B/lens.pt \
  --model Qwen/Qwen3-8B \
  --prompt "Fact: The currency used in the country shaped like a boot is" \
  --positions=-2,-1 --layers 9,18,21-27 --logit-lens

Notes

• Read/fingerprint artifacts in this repo were generated from the pilot (N=100) lens during the pipeline; the uploaded lens.pt is the full N=1000 fit. Re-run read/fingerprint locally against the full lens if you want diagnostics matched to the final artifact.
• Merge: JacobianLens.merge() over 4 disjoint corpus shards (jlens/merge_cli.py).