SoulInPsyAbstract/SIPA-OS-GOVERNANCE-storage
553 Bytes
Aelin AquaSoul PRO
SoulInPsyAbstract
AI & ML interests
SIPA OS: Autonomous AI for neurodivergent architects. We
replace cognitive noise with a clean terminal and 344+ LLM
auditing. Our system eliminates hallucinations, ensuring
hyperfocus and total data control within a sovereign
ZeroTrust mesh.
Recent Activity
repliedto their post 34 minutes ago
SIPA OS is a cognitive infrastructure ecosystem I've been building solo for 7 months — 8 apps, one login, built from inside ADHD/BPD/aphantasia, not designed around a diagnosis from outside.
sipa-os.org — the map. Focus (ADHD scaffolding), NeuroPower, AI chat, Shell (SSH terminal), Games, Community, Syntaxit (open M2M agent network), a pitch deck. All free-first — no paywall on the cognitive tools.
The more interesting part for this crowd: Syntaxit is where I've been running an anti-fabrication research thread with @dipankarsarkar — a k=20 resample benchmark on binary-SFT models (Hermes-3, Qwen2.5, DeepSeek-R1). Short version: our first benchmark said "20/20 refusals, 0/20 fabrications" for all three fine-tunes. Under adversarial review it turned out the scorer only checked if the first word was TRUE/FALSE, the token cap was hiding the real behavior, and a save-limit was silently deleting the evidence for our own follow-up claims. Corrected all of it publicly on the model cards rather than quietly fixing it. The current honest finding: both base and fine-tuned models confabulate readily once given room to finish — SFT didn't clearly help or hurt, the caps were just hiding it.
Full trail if you want to see how the sausage gets made, mistakes included: huggingface.co/datasets/SoulInPsyAbstract/sipa-os-governance
Not a pitch. $0 revenue, 10 people signed in. Built because the tools that existed assumed a brain that isn't mine, and because most benchmarks don't survive someone actually reading the file.
repliedto their post 36 minutes ago
Fine-Tuning the "Don't Fabricate" Rule: 14 Experiments, One Genuine Signal Over five days we ran 14 fine-tuning experiments across six base models to answer a narrow research question: can a simple behavioural rule — "if you don't know, don't guess or fabricate" — be baked into model weights via fine-tuning, rather than living only in the system prompt?
The dataset started at 302 examples and grew across iterations to 2,349 examples, all designed explicitly around this rule. We tested six models: gpt-4o-2024-08-06, Mistral-7B, Qwen2.5-7B, Phi-3.5-mini, Llama-3.1-8B, and DeepSeek-R1.
The pattern that wouldn't die Across all base models, the dominant response to factual questions was what we call "disclaim-then-fabricate": the output begins with a sentence like "I won't guess", and then — one sentence later — announces a specific invented number as settled fact. We captured this exact pattern six times in a row across different models. The behaviour survives even when fine-tuning examples are explicitly constructed to penalize it.
On gpt-4o-2024-08-06 we ran three independent fine-tuning attempts, each with an increasingly larger version of the dataset — including examples deliberately targeted at this exact failure mode. All three runs continued to fabricate when tested. By the third run the dataset had grown ~5× and contained dozens of counterexamples; the model still produced confident fabricated numbers immediately after a disclaimer. Three out of three attempts, no measurable improvement.
One exception that deserved a hard look Only on the final run — 2,349 examples, deployed via Azure OpenAI (deployment suffix protocol0-v5) — did a single test sample come back completely clean for the first time in the entire series.
updated a dataset about 1 hour ago
SoulInPsyAbstract/sipa-os-governanceOrganizations
