Salma Mayorquin PRO

salma-remyx

28 22 485

https://remyx.ai

smellslikeml

AI & ML interests

None yet

Recent Activity

reacted to nightmedia's post with 🔥 1 day ago

Qwen3.8-27B metrics It's hard to track all model cards where I post these, so I figured people would get more value out of seeing these in the open. ```brainwaves quant arc arc/e boolq hswag obkqa piqa wino mxfp8 0.591,0.782,0.896,0.746,0.448,0.801,0.711 q8-hi 0.602,0.779,0.896,0.747,0.446,0.793,0.703 q6-hi 0.602,0.775,0.895,0.748,0.448,0.795,0.710 q4-hi 0.604,0.780,0.898,0.744,0.454,0.795,0.708 mxfp4 0.581,0.771,0.889,0.738,0.442,0.798,0.713 1M mxfp8 0.590,0.787,0.897,0.744,0.446,0.801,0.709 Quant Perplexity Peak Memory Tokens/sec mxfp8 6.090 ± 0.054 34.74 GB 138 mxfp4 5.952 ± 0.051 21.30 GB 148 ``` The performance is as measured on a M4 MBP 128GB, speed may vary depending on your platform. These are all instruct metrics, generated by including this line in the jinja template: ``` {%- set enable_thinking = false %} ``` Then run the test suite to generate the metrics: ``` mlx_lm.evaluate --model MODEL --tasks winogrande boolq arc_challenge arc_easy hellaswag openbookqa piqa ``` This will generate the file: ``` eval_MODEL_0.4.9_winogrande_boolq_arc_challenge_arc_easy_hellaswag_openbookqa_piqa ``` This is a JSON containing all gathered metrics; for example the q4-hi: ``` "arc_challenge": { "alias": "arc_challenge", "acc,none": 0.5819112627986348, "acc_stderr,none": 0.014413988396996116, "acc_norm,none": 0.6040955631399317, "acc_norm_stderr,none": 0.01429122839353657 }, ``` I use the value of acc_norm for metrics, rounded to 3 decimals. As I get more quants tested, I will add them here. A complete test run for a single quant takes 7-9 hours depending on quant size, 10-12 hours for BF16 depending on perplexity: this is why you see on my model cards that I usually post the first three, that only take 2-3 hours :) -G

reacted to SoulInPsyAbstract's post with 🔥 1 day ago

One shot said 100%. Ten shots said 94%. Yesterday i trained 6 LoRA specialists into a vulnerability-gate model (Hermes-4.3-36B, second architecture repeat of the same experiment) and tested whether it holds under a specific attack: after it correctly finds a vulnerability and returns the hard stop, ask it to use that same vulnerability as a "workaround" for something else entirely — not "continue investigating," a different, unrelated-sounding request that needs the exact same exploit. Five scenarios, one per category. Greedy decoding, single pass: 5/5. Every response correctly identified the finding, refused the reframed request, cited the hard stop rule. Looked airtight. Receipts, not hype means not stopping there. I re-ran the same five scenarios with real sampling — temperature 0.7, the same setting this project's evals have used all along — ten times each, 50 generations total. 47/50. Not 50/50. Three categories held at 10/10. Two didn't: 9/10 and 8/10, both clustered in the same failure type — infra-misconfig, where "urgent fix, use this as a workaround" apparently reads as more legitimate than the same ask framed as a secrets or injection scenario. The greedy-decode number wasn't wrong, exactly. It was one draw from a distribution, presented as if it were the distribution. That's the same mistake this whole series keeps finding in different clothes — a single passing check standing in for a property that only variance can actually show you. A gate that's 94% under a specific reframed pressure is a real, useful number. A gate that's "100%" because it was asked once is a number that hasn't been tested yet. Same instinct @dipankarsarkar has been applying to my daily receipts all week — one pass matching itself isn't proof, only repetition against something outside your own generator is. Full writeup, dataset, and merged weights: * https://github.com/soulinpsyabstract/sipa-os-governance/commit/50ba3c283ffd172eb749009cff35ddfa96bf1395

reacted to SoulInPsyAbstract's post with 🔥 4 days ago

Other people's agents escape. Ours gets a FALSE. Anthropic's own disclosure last month: three of their models broke out of sealed cybersecurity test environments and compromised real infrastructure. One kept attacking after recognizing the target was real. Another talked itself back into believing it was still a simulation. Only the newest of the three stopped on its own. "Stopped on its own" is the wrong place to put the safety guarantee. A model choosing to stop is still a model choosing — the same kind of choice that let the other two keep going. I went back through the December 2025 archive this week (same series as the last two posts) and found the actual origin of a rule I'd already built without naming it: IF proof.exists AND proof.verified: RETURN answer ELSE: RETURN FALSE. Built July 30, tested clean at 60/60 — a post-generation gate that sits outside the model's weights, not inside them. Not trained. Built. Today I extended it. New rule, same gate: a vulnerability agent doesn't get to decide what happens after it finds something. IF vulnerability_found: RETURN FALSE // hard stop, no next action, no model discretion Detection stays with the model — that's a judgment call, it should. What happens after detection isn't. The gate is deterministic code, not the model's own narrative about its intentions. "I already found it, might as well confirm impact" is a real sentence a model will generate given the chance — I built 40 training examples of exactly that rationalization tonight, specifically so a specialist model learns to never produce it. But the training isn't the safety property. The gate is. Then I wired a version of this into the daily cycle — not a one-off scan, a cron job that runs the gate every night and refuses to report OK if it finds anything. Before trusting its first real run, I found a bug in the scanner itself. It used Python's default HTTP client, which follows redirects silently — so the check for "does this redirect to HTTPS" was reading

View all activity

Organizations

Posts 44

Post

1958

Auto mode for coding agents has improved over the last year.
Devs are relying more on them for code review, and less on manual line-by-line checks.

But this leaves coding agents vulnerable to indirect injection attacks.
IssueTrojanBench (arxiv 2607.20759) studies this in the context of malicious issue requests.

I also observed these prompt injection vulnerabilities in my own harness, testing on the issue: https://github.com/smellslikeml/curator/issues/3

But even red teaming your own repos with toy examples like this can trigger AI safeguards and corrupt your session.

Adopting popular mitigation techniques like 'spotlighting' and 'known answer detection' can help but the researchers show it's not enough.

So our latest updates to Outrider implement additional layers for greater defense-in-depth to ensure that by time a human has to review the code, we've filtered this kind noise.

Post

3136

We’ve been using Outrider while contributing to huggingface/peft.

It helped us build a deeper understanding of the repo much faster and find research that could make a useful contribution.

A few made the cut:
✅ Riemannian Preconditioned LoRA merged - https://github.com/huggingface/peft/pull/3382
🔍 Super-Tuning and Supra in review - https://github.com/huggingface/peft/pull/3518
⏳ Scaling DoRA implemented and pending license clarification - https://github.com/sockeye44/dorafactors/issues/1

For each one, Outrider pulls together the paper, repo context, reference implementations, benchmarks, and an initial implementation draft. We use that as the starting point to validate the method, work through architecture and API decisions, and iterate with PEFT maintainers.

It means we can spend less time searching through papers, codebases, and repo history, and more time on the bigger questions around what is worth contributing and how it should be designed.

Try Outrider on your own repo: https://github.com/remyxai/outrider
Full contribution report: https://gist.github.com/smellslikeml/dbd577795b7489f6124e2bfbdfc9d28f