ANLP Assignment 1 — Configuration C3
attention: MHA -> grouped-query (8 query heads, 2 KV groups)
Encoder-decoder Transformer built from basic PyTorch operations (no nn.Transformer or
nn.MultiheadAttention), trained to decrypt binary cipher text into English plaintext.
One of five configurations in a controlled ablation. Each of C2-C5 changes exactly one component from the C1 base; every other hyperparameter is identical across all five runs.
This configuration
| Axis | Setting |
|---|---|
| Positional encoding | sinusoidal |
| Attention | gqa |
| Normalization | layernorm |
| Tokenization | subword |
Shared hyperparameters
d_model 256 | 4 encoder layers | 4 decoder layers
8 heads | d_ff 1024 | dropout 0.1 | pre-norm
AdamW lr 0.0003 | warmup 1000 | 15000 steps | batch 64 | seed 42
fp32 (no AMP)
Results
| Metric | Value |
|---|---|
| Bit-level accuracy | 0.9900 |
| Sequence accuracy | 0.8902 |
| Levenshtein distance | 0.19 |
| BLEU | 97.65 |
| ROUGE-L | 0.9883 |
| Parameters | 8,345,344 |
| Peak GPU memory (MB) | 1263.1 |
| Seconds per training step | 0.0742 |
Usage
Checkpoints are plain PyTorch state dicts. See the accompanying code repository for the
model definition; torch.load(...)["model"] loads into build_model(get_config("c3"), ...).
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support