MEBestawy/apollo-11-transcript-unmasker

Apollo 11 Transcript Unmasker

Character-level transformer trained to reconstruct masked Apollo 11 air-to-ground dialogue. Input: masked text (e.g. Ho**ton, *pollo 11.)
Output: reconstructed text in packed form (e.g. {Ho**ton, *pollo 11.|Houston, Apollo 11.})

Dataset

Trained on masked variants of the cleaned Apollo 11 transcript:

• MEBestawy/apollo-11-air-to-ground-transcript Each example uses the prompt format {masked|text}. Loss is applied only to the response (characters after |).

Architecture

Small causal character-level transformer (~3.7M parameters):

Setting	Value
Vocab	256 (ASCII)
Model dim (d_model)	256
Attention heads	8
Transformer blocks	4
Max sequence length	2048
Positional encoding	Learned absolute
Activation	GELU
Normalization	LayerNorm (pre-norm blocks)
Output	Tied embedding projection → next-character logits

Block layout:
x → LayerNorm → Multi-Head Self-Attention → residual → LayerNorm → FeedForward (4× expand) → residual Attention is causal (each position only sees past tokens), so the model can be trained in parallel and used autoregressively at inference.

Training results

Best checkpoint after 8 epochs / 10,000 steps:

Metric	Value
Train loss	0.69
Test loss	0.98
Test char accuracy (response tokens)	~72%

Loss is computed on response tokens only (after |). The curve shows train and test loss over training iterations.

Files

File	Description
model.pt	Best checkpoint (model state dict + training metrics)
config.json	Architecture and training metadata
loss_curve.png	Iteration vs train/test loss

Usage

This is a custom PyTorch model (not a Hugging Face transformers checkpoint). You need the model code from the training repo to load it.

import torch
from model import build_model
model = build_model()
state = torch.load("model.pt", map_location="cpu")
model.load_state_dict(state["model"])
model.eval()

Prompt format: wrap masked input as {masked| and generate until }.

Example:

Input: Hoton, *pollo 11. Prompt: {Ho**ton, *pollo 11.| Output: {Hoton, *pollo 11.|Houston, Apollo 11.}