Odyssey

Odyssey is a small language model (SLM) with a LLaMA-style decoder-only architecture, built for primary purposes like lightweight text generation / experimentation / on-device inference.

Model Details

• Developed by: Rakesh44
• Model type: Decoder-only transformer (LlamaForCausalLM)
• Parameters: ~471M (24 layers × 1280 hidden, tied embeddings)
• Context length: 2048 tokens
• Vocabulary size: 50,257 (GPT-2 tokenizer)
• Precision: float32
• Language: English
• License: Apache 2.0

Architecture

Component	Value
Hidden size	1280
Layers	24
Attention heads	16
KV heads	16 (no GQA — full multi-head attention)
Head dim	80
Intermediate (MLP) size	3456
Activation	SiLU
Norm	RMSNorm (eps 1e-5)
Positional encoding	RoPE (theta 10000)
Attention bias	None
Tied embeddings	Yes
Max position embeddings	2048

Scope

Odyssey is a base language model pretrained on OpenWebText, so its native capability is fluent English text continuation rather than instruction following. Out of the box it is suited to simple language tasks: text completion, short-form generation, style and tone continuation, and serving as a foundation for further training.

Its main value is as a starting point for fine-tuning. Because it has already learned general English structure and web-scale language patterns, it can be adapted to focused downstream tasks with a relatively small labeled dataset. Suitable fine-tuning targets include:

• Financial sentiment analysis — classifying the sentiment of financial text such as earnings reports, filings, or analyst notes.
• Market sentiment analysis — gauging bullish, bearish, or neutral tone in news headlines, social posts, or commentary.
• General text classification — topic tagging, intent detection, and similar sentence- or document-level labeling tasks.
• Domain-specific text generation — adapting the model's output style to a narrow domain after task-specific tuning.

Out of scope

Not suitable for high-stakes, factual-authority, or safety-critical use. As a small model, factual reliability and reasoning depth are limited.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Rakesh44/odyssey")
model = AutoModelForCausalLM.from_pretrained("Rakesh44/odyssey")

inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training

• Data: OpenWebText (~8M documents, ~40GB)
• Tokenizer: GPT-2 BPE (50,257 tokens)
• Hardware: 1× RTX 4090
• Hyperparameters:
• Training time: 100 hours

Limitations and Bias

Odyssey is a small model and may produce inaccurate, repetitive, or biased text reflecting its training data. Verify outputs before use. The 2048-token context and float32 weights mean memory use is higher than a quantized deployment would require; consider converting to bf16/fp16 for inference.