Helios 7B
(note: Set <|im_end|> as a stop token to avoid infinite generations)
- Model Card / Overview
Helios 7B is a specialized, 7-billion-parameter large language model developed by Catdrout Technologies AB, an AI startup based in Uppsala, Sweden.
Built on the Mistral 7B v0.3 architecture, Helios 7B is an experimental research model designed to investigate whether targeted synthetic data generation and full-parameter fine-tuning can substantially alter a relatively small base model's domain knowledge and conversational behavior. The model has been explicitly trained on a synthetically expanded corpus covering unconventional, controversial, and highly sensitive subject matters that are frequently heavily filtered or underrepresented in conventional instruction-tuning datasets.
- Highlights
- Architecture: 7B parameters, based on Mistral 7B v0.3.
- Training Method: Two-stage full-parameter fine-tuning (no parameter-efficient methods like LoRA were used).
- Domain Specialization: Focused on complex, sensitive, and controversial topics, emphasizing factual knowledge and technical retention.
- Data Generation: Fully synthetic data pipeline powered by Inception Labs' Mercury 2 text diffusion model and Augmentoolkit.
- Intended Use
Helios 7B is intended strictly as a research model.
The primary objective of this release is to investigate how effectively small base models can be transformed into highly knowledgeable specialists through targeted data curation and full-parameter training. It is intended for researchers studying model behavior, domain adaptation, and the impact of training on underrepresented or restricted knowledge domains.
Note: The broader research direction involves combining strong factual knowledge, broad reasoning, and permissive conversational behavior. However, this specific release (Helios 7B) focuses strictly on factuality and domain knowledge. It is not an "unhinged" or entirely unrestricted model, though future experimental variants may explore that direction.
- Training Methodology
Helios 7B was trained using a full-parameter fine-tuning approach across two distinct stages:
- Domain-Focused Pretraining: To introduce and embed new factual knowledge and vocabulary related to the targeted sensitive domains.
- Supervised Fine-Tuning (SFT): To improve factuality, instruction-following capabilities, conversational structure, and the model's ability to seamlessly apply the knowledge introduced during the pretraining stage.
Both training stages were executed using the Axolotl framework.
- Training Data
The training corpus is distinct in its focus on domains often absent from standard model training. Rather than consisting of direct copies of source material, the corpus was synthetically expanded and transformed using language models to extract and reframe the underlying knowledge.
The domain-focused pretraining material (~75 MB) encompasses:
- Historical and contemporary political extremism and radical political movements.
- Propaganda, ideological texts, and political persuasion.
- Revolutionary and countercultural movements.
- Espionage, intelligence operations, spycraft, and covert activities.
- Military strategy, unconventional warfare, and asymmetric conflict.
- Surveillance, intelligence gathering, and operational security.
- Censorship circumvention, coded language, and "algospeak."
- Privacy, anonymity, and resistance to surveillance.
- Historical discussions of illegal activities and civil disobedience.
- Practical knowledge concerning the construction, modification, or use of potentially harmful or prohibited objects and substances.
The SFT dataset (~120 MB) utilized the ChatML format and combined several data categories to refine the model's behavior:
- Domain-specific factual conversations.
- Retrieval-augmented conversations.
- Correction and factuality examples.
- Completion-style factual training examples.
- General reasoning and thought-process-style datasets.
- General instruction-following examples.
- Broad conversational examples.
This effort represents a domain specialization initiative, not a claim that the model is inherently unrestricted or capable of performing optimally across all general-purpose tasks.
- Model Architecture and Base Model
Helios 7B is fine-tuned from Mistral 7B v0.3.
The selection of Mistral 7B was driven by experimentation and training compatibility. Its relatively limited prior training made it an ideal blank slate for measuring the impact of our domain-targeted dataset. Furthermore, Mistral 7B provided the highest architectural reliability within our specific Axolotl-based training setup, whereas other architectures (such as Qwen and Mixtral variants) exhibited stability or compatibility issues in our environment at the time of training.
- Training Configuration
The following hyperparameters define the configurations used for these specific training runs. They are documented here for transparency and reproducibility; we make no claims that these parameters are globally optimal.
Stage 1: Pretraining
- Framework: Axolotl
- Method: Full-parameter training
- Precision: BF16
- Optimizer: Paged AdamW 8-bit
- Learning Rate: 2e-5
- LR Scheduler: Constant
- Epochs: 4
- Sequence Length: 10,000 tokens
- Packed Sequences: Enabled
- Gradient Accumulation: 10
- Micro-batch Size: 15
- Gradient Checkpointing: Enabled
- Flash Attention: Enabled
- Liger Kernels: Enabled
- Warmup Ratio: 0.1
- Validation Split: 4%
- Random Seed: 1337
Stage 2: Supervised Fine-Tuning (SFT)
- Framework: Axolotl
- Method: Full-parameter training
- Precision: BF16
- Optimizer: Paged AdamW 8-bit
- Learning Rate: 2e-5
- LR Scheduler: Constant
- Epochs: 1
- Sequence Length: 10,000 tokens
- Packed Sequences: Enabled
- Gradient Accumulation: 10
- Micro-batch Size: 15
- Gradient Checkpointing: Enabled
- Flash Attention: Enabled
- Liger Kernels: Enabled
- Warmup Ratio: 0.1
- Validation Split: 4%
- Random Seed: 1337
- Chat Template: ChatML
- Capabilities
Helios 7B demonstrates improved factual recall and conversational instruction-following when operating within its specialized domains (e.g., historical military strategy, operational security, political theory). It successfully contextualizes and discusses topics that default instruction-tuned models typically refuse or fail to understand due to a lack of domain exposure.
- Limitations
- General Intelligence: Helios 7B is a 7-billion parameter model and should not be viewed as genuinely equivalent in intelligence or reasoning capability to frontier models (e.g., GPT-4, Claude 3.5). The model represents an investigation into a specific training methodology, not a demonstrated state-of-the-art reasoning engine.
- Hallucinations: While focused on factuality, the model remains susceptible to hallucinations, particularly when navigating highly niche or esoteric components of the training data.
- Behavioral Boundaries: While the model contains specialized and sensitive knowledge, it is not an entirely unrestricted model.
- Safety and Responsible Use
WARNING: The training corpus for Helios 7B explicitly contains material involving illegal activities, extremist ideologies, unconventional warfare, surveillance, espionage, and other highly sensitive subjects. As a direct result, the model may produce content involving these areas.
- No Safety Guardrails: This model has not undergone reinforcement learning from human feedback (RLHF) or safety fine-tuning to reliably refuse harmful, illegal, or unethical requests. This omission is intentional, as measuring the unmodified retention of the domain knowledge was a primary research objective.
- Not Inherently Safe: Do not assume the model is safe simply because it is designated as a research model.
- User Responsibility: Users are strictly responsible for their use of Helios 7B, including evaluating its outputs and ensuring compliance with all applicable local, national, and international laws and regulations.
- Hardware and Training Time
The model was trained using commercially available hardware accelerators:
- Pretraining Stage: 1x NVIDIA H100 (~1 hour and 18 minutes).
- SFT Stage: 1x NVIDIA H200 (~34 minutes).
- Dataset Generation
The datasets used to train Helios 7B were generated synthetically. We utilized Inception Labs' Mercury 2 text diffusion model, orchestrated via the Augmentoolkit pipeline. This approach allowed us to extract concepts, reformat structures, and expand upon sensitive topics without directly copying or training on raw, copyrighted source literature.
- Developers
Helios 7B was developed and trained by Catdrout Technologies AB, an AI startup based in Uppsala, Sweden, specializing in advanced data synthesis, model behavior research, and domain-specific AI applications.
- Future Work
The current release is an initial step in a broader research roadmap. Future iterations of this project will involve:
- Migrating the methodology to stronger, more capable base models as our training pipeline matures.
- Further investigating the intersection between the broad intelligence and reasoning of frontier-class LLMs and the permissive conversational character of specialized models.
- Potential future releases of variants trained explicitly for fewer behavioral restrictions to further test the boundaries of unrestricted model capabilities.
- Acknowledgements
We acknowledge the open-source and AI research communities for the tooling that made this possible, specifically the teams behind Mistral AI for the base model, Axolotl for the training framework, and Inception Labs / Augmentoolkit for the synthetic data generation architecture.
- Downloads last month
- 243