m15dg/local-ai-toolkit

⚠️ Note: This README was generated and is maintained by AI.

🧠 Local AI Toolkit

A Curated Collection of Locally-Runnable AI Models

Optimized for RTX 3060 12GB VRAM + 32GB RAM

AI-assisted model selection · Quantized for consumer hardware · A suggested starting point

---

📖 About This Toolkit

This repository is a suggested toolkit of open-source AI models assembled to run entirely locally on consumer-grade hardware — specifically an NVIDIA RTX 3060 (12GB VRAM) with 32GB system RAM. Each model was chosen to fit within these constraints while offering strong output quality for the VRAM budget.

The selection process was AI-assisted — I consulted several AI assistants to weigh benchmarks, quantization trade-offs, and model capabilities, then settled on the common recommendations you see here. This is not a definitively optimal combination: better models may already exist, and stronger ones are released regularly. Treat it as a well-reasoned starting point rather than a final answer — your own testing on your own hardware is the real benchmark.

🔄 Updates & Longevity

This toolkit is periodically refreshed as stronger or more efficient models are released. When a model gets clearly outclassed, it gets swapped out — so the lineup you see today may differ from a few months down the line.

🎯 Design Philosophy

Each category follows a dual-tier architecture. Both tiers target the same 12GB VRAM + 32GB RAM machine — the difference is how that budget is used:

Tier	Purpose	What It Means
🏆 Quality	Best achievable quality	The highest-quality model that can still run within the 12GB VRAM + 32GB RAM budget, using GPU+CPU offload (some layers spill into system RAM). Slower, but maximizes output quality
⚡ Speed	Fastest, no offload	A capable model that fits entirely within 12GB VRAM — leaving room for context too — so it runs fully on-GPU with no CPU offload overhead. Prioritizes throughput and low latency

Note on Coding models: The Coding Agents category includes two quality-tier models (Architect & Executor + Detective & Debugger). The "speed" variants in that category, while labeled for speed, are still powerful enough to serve as competent rapid-response code assistants — they simply trade some reasoning depth for significantly faster token generation.

⚠️ Work In Progress

This toolkit is not yet complete. Notably, Image Generation and Video Generation models are still missing and will be added in future updates. The current collection focuses on text, speech, and multimodal understanding capabilities. Stay tuned for expansions.

---

📂 Repository Structure & Category Guide

local-ai-toolkit/
├── 1-Universal_Foundation_LLMs/      🧠 General-purpose language models
├── 2-Coding_Agents/                  💻 Specialized code generation & debugging
├── 3-Agent_Orchestration_and_Routing/ 🔗 Multi-agent system coordination
├── 4-Document_Analysis_and_RAG/      📄 Embeddings, rerankers, and retrieval
├── 5-TTS/                            🎙️ Text-to-speech synthesis
├── 6-STT/                            🎤 Speech-to-text recognition
├── 7-Multimodal/                     👁️ Vision + language understanding
└── 8-Uncensored_Models/              🔓 Unfiltered model variants

---

1️⃣ Universal Foundation LLMs

The backbone of any local AI setup — general-purpose models for conversation, analysis, writing, and reasoning.

These are your go-to models for everyday tasks: answering questions, drafting emails, summarizing documents, brainstorming ideas, creative writing, and general knowledge retrieval. They serve as the foundation upon which more specialized workflows are built.

Tier	Model	Size	Quantization	Key Strengths
🏆 Quality	Qwen 3.6-35B-A3B	35B (MoE, 3B active)	Q4_K_XL	Mixture-of-Experts with only 3B active params — exceptional quality-to-VRAM ratio. Strong multilingual support, excellent reasoning
⚡ Speed	Qwen 3.5-9B	9B	Q5_K_M	Blazing fast inference, still remarkably capable for its size. Perfect for quick Q&A, chat, and real-time interactions

Why these models? The Qwen 3.x series represents a significant leap in efficient architecture. The 35B-A3B MoE variant is particularly special — it gives you the reasoning depth of a much larger model while only activating 3 billion parameters at a time, making it one of the most VRAM-efficient high-quality models available. The 9B speed variant offers near-instant responses for situations where latency matters more than depth.

📁 Folder Structure

1-Universal_Foundation_LLMs/
├── Quality/
│   └── Qwen3.6-35B-A3B-UD-Q4_K_XL/
│       ├── Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
│       └── mmproj-BF16.gguf
└── Speed/
    └── Qwen3.5-9B-Q5_K_M/
        ├── Qwen3.5-9B-Q5_K_M.gguf
        └── mmproj-BF16.gguf

---

2️⃣ Coding Agents

Specialized models for writing, debugging, and architecting code — the core of any AI-assisted development workflow.

This category contains models specifically fine-tuned for software engineering tasks. They are organized into five distinct roles that mirror a professional development team structure, from high-level architecture down to real-time inline suggestions.

Role	Tier	Model	Size	Quantization	Purpose
🏗️ Architect & Executor	🏆 Quality	Qwen3-Coder-30B-A3B	30B (MoE, 3B active)	Q5_K_M	Plans architecture, writes full implementations, handles complex codebases
🔍 Detective & Debugger	🏆 Quality	DeepSeek-R1-Distill-Qwen-32B	32B	Q4_K_M	Deep chain-of-thought reasoning for finding and fixing bugs
⚡ Autocomplete	—	Qwen2.5-Coder-7B	7B	Q5_K_M	Ultra-fast inline code completion for IDE integration
🏗️ Architect (Fast)	⚡ Speed	Qwen2.5-Coder-14B	14B	Q4_K_M	Quick code generation and architectural guidance
🔍 Debugger (Fast)	⚡ Speed	DeepSeek-R1-Distill-Qwen-14B	14B	Q4_K_M	Rapid debugging with reasoning capabilities

Why this structure? Modern AI-assisted coding works best when you use specialized models for specialized tasks. The Qwen3-Coder excels at writing new code and understanding project structure, while the DeepSeek-R1 distill variant uses its chain-of-thought training to methodically trace through bugs. The 7B autocomplete model is lightweight enough to run alongside your IDE for real-time suggestions without competing for VRAM with your main coding model.

Pro tip: Use the "speed" variants for quick prototyping and test-driven development cycles where you need rapid iteration. They're still high-quality coders — they just generate tokens faster and use less memory, leaving headroom for other tools in your pipeline.

📁 Folder Structure

2-Coding_Agents/
├── 1-Architect_and_Executor/
│   └── Qwen3-Coder-30B-A3B-Instruct-Q5_K_M/
│       └── Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf
├── 2-Detective_and_Debugger/
│   └── DeepSeek-R1-Distill-Qwen-32B-Q4_K_M/
│       └── DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf
├── 3-Inline_Autocomplete/
│   └── qwen2.5-coder-7b-instruct-q5_k_m/
│       └── qwen2.5-coder-7b-instruct-q5_k_m.gguf
├── 4-Architect_and_Executor_speed/
│   └── Qwen2.5_Coder_14B_Instruct_Q4_K_M/
│       └── Qwen2.5_Coder_14B_Instruct_Q4_K_M.gguf
└── 5-Detective_and_Debugger_speed/
    └── DeepSeek_R1_Distill_Qwen_14B_Q4_K_M/
        └── DeepSeek_R1_Distill_Qwen_14B_Q4_K_M.gguf

---

3️⃣ Agent Orchestration and Routing

The nervous system of multi-agent AI systems — models that coordinate, route, guard, and execute complex agent workflows.

When you move beyond single-model interactions into multi-agent architectures (where different specialized models collaborate on complex tasks), you need orchestrators that can plan, delegate, monitor, and validate. This category provides the complete infrastructure for building sophisticated agentic systems locally.

🎯 Orchestrators & Managers

The "brains" of your agent network. These models decide which worker to call, when to call it, and how to synthesize results.

Model	Size	Quantization	Role
Agents-A1 35B MoE	35B (MoE)	IQ4_XS	CEO Agent — high-level autonomous planning and task decomposition. Calibrated with imatrix for superior quantization fidelity
GLM-4.7-Flash	Flash	Q4_K_XL	Router — ultra-fast model for routing decisions, intent classification, and dispatching tasks to appropriate workers
Hermes 4.3 36B	36B	Q4_K_M	Quality Orchestrator — a capable model for complex multi-step reasoning, tool use, and agent coordination. Excellent function calling

⚙️ Agentic Workers

Models optimized to be called by orchestrators — they receive specific tasks, execute them, and return structured results.

Model	Size	Quantization	Role
Gemma 4 12B Agentic Worker v2	12B	Q4_K_M	Worker Agent — fine-tuned specifically for agentic task execution with Fable5 and Composer2.5 training. Designed to follow orchestrator instructions reliably

🛡️ Guardrails & Safety

Models that monitor and filter agent inputs/outputs to prevent harmful, off-topic, or undesired behavior.

Model	Size	Quantization	Role
Hermes 4 14B	14B	Q4_K_M	Speed Guardrail — fast content moderation and output validation for real-time agent pipelines
Llama Guard 3 8B	8B	i1-Q4_K_M	Firewall — Meta's safety-focused model for input/output classification and content policy enforcement

📁 Folder Structure

3-Agent_Orchestration_and_Routing/
├── 1-Orchestrators_and_Managers/
│   ├── Agents_A1_35B_MoE_Calibrated_CEO/
│   │   └── Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated.gguf
│   ├── GLM-4.7-Flash-UD-Q4_K_XL/
│   │   └── GLM-4.7-Flash-UD-Q4_K_XL.gguf
│   └── Hermes_4_3_36B_Quality_Orchestrator/
│       └── hermes-4_3_36b-Q4_K_M.gguf
├── 2-Agentic_Workers/
│   └── Gemma4_12B_Agentic_Worker_v2/
│       └── Gemma4_12B_Agentic_Worker_v2.gguf
└── 3-Guardrails_and_Safety/
    ├── Hermes_4_14B_Speed_Guardrail/
    │   └── Hermes-4-14B-Q4_K_M.gguf
    └── Llama_Guard_3_8B_i1_Firewall/
        └── Llama-Guard-3-8B.i1-Q4_K_M.gguf

---

4️⃣ Document Analysis and RAG

The complete pipeline for building local Retrieval-Augmented Generation systems — from embedding to generation.

RAG (Retrieval-Augmented Generation) is how you give your AI access to your own documents, knowledge bases, and data. This category provides every component needed to build a production-quality local RAG pipeline — no cloud APIs required.

Pipeline Overview

📄 Documents → 🔤 Embedding → 📊 Vector Store → 🔄 Reranking → 🧠 LLM Generation
                  BGE-M3           (your DB)        BGE-Reranker     Gemma 4 12B
                                                              ↕
                                              🕸️ GraphRAG Extractor (NuExtract)
                                                              ↕
                                              🌐 Web Distiller (Jina Reader)

🔤 1. Embeddings — BGE-M3

The foundation of any RAG system. BGE-M3 is a state-of-the-art multilingual embedding model that supports three retrieval modes simultaneously:

• Dense retrieval — traditional semantic similarity search
• Sparse retrieval — BM25-style keyword matching (great for exact term matches)
• ColBERT retrieval — late interaction for fine-grained token-level relevance scoring

This multi-modal retrieval approach means you combine semantic understanding and keyword precision in a single model. It supports 100+ languages out of the box.

🔄 2. Reranker — BGE-Reranker-V2-M3

After initial retrieval returns candidate documents, the reranker re-scores and re-orders them for maximum relevance. This two-stage approach (retrieve broadly, then rerank precisely) dramatically improves RAG accuracy compared to single-stage retrieval. The M3 variant is specifically designed to complement BGE-M3 embeddings.

🕸️ 3. GraphRAG Extractor — NuExtract v1.5

Takes unstructured text and extracts structured knowledge graphs — entities, relationships, and attributes. This enables GraphRAG pipelines that can reason over the connections between facts, not just individual document chunks. Particularly powerful for research, legal, and technical domains where relationships matter as much as the facts themselves.

🌐 4. Web Distiller — Jina Reader LM 1.5B

A tiny but remarkably effective model that cleans and structures raw web content into clean, readable text suitable for embedding. Instead of embedding HTML noise, navigation bars, and ads, this model extracts the actual article content. At only 1.5B parameters, it runs almost instantly.

🧠 5. LLM Generator — Gemma 4 12B

The final stage of the RAG pipeline — a vision-capable LLM that synthesizes retrieved context into coherent, accurate answers. Includes speculative decoding via MTP (Multi-Token Prediction) for faster generation, and a multimodal projector for handling documents with images and diagrams.

📁 Folder Structure

4-Document_Analysis_and_RAG/
├── 1-Embeddings/
│   └── BGE_M3_Core_Embedding/
│       ├── config.json, tokenizer files, model weights...
│       └── colbert_linear.pt, sparse_linear.pt
├── 2-Rerankers/
│   └── BGE_Reranker_V2_M3_Core/
│       ├── config.json, model.safetensors
│       └── tokenizer files
├── 3-GraphRAG_Extractors/
│   └── NuExtract_1_5_GraphRAG_Extractor/
│       └── NuExtract-v1.5-Q8_0.gguf
├── 4-Web_Distillers/
│   └── Jina_Reader_LM_1_5B_Web_Distiller/
│       └── reader-lm-1.5b-Q8_0.gguf
└── 5-LLM_Generators/
    └── gemma-4-12b-it-UD-Q4_K_XL/
        ├── gemma-4-12b-it-UD-Q4_K_XL.gguf
        ├── mmproj-BF16.gguf
        └── MTP/
            └── mtp-gemma-4-12b-it-Q8_0.gguf

---

5️⃣ Text-to-Speech (TTS)

Convert text to natural-sounding speech — from multi-speaker dialogue to voice cloning and custom voice design.

This category covers four distinct TTS paradigms, each suited to different use cases. Whether you need a single consistent narrator, a full cast of characters, or the ability to clone any voice from a short sample, there's a model here for it.

Paradigm	Model	Size	Use Case
🎭 Multi-Speaker	Dia 2.5 2B	2B	Generate dialogue with multiple distinct speakers in a single pass — perfect for audiobooks, podcasts, and conversational AI
🇮🇷 Persian TTS	Chatterbox TTS Persian	—	High-quality Farsi text-to-speech with natural Persian prosody and pronunciation
🇮🇷 Persian TTS	F5-TTS Persian	—	Alternative Persian TTS model — good for A/B testing voice quality on Farsi content
🎨 Voice Design	Qwen3-TTS 12Hz	1.7B	Design entirely new voices from text descriptions — describe the voice you want ("warm grandmotherly voice") and it generates it
🎤 Zero-Shot Cloning	F5-TTS v1	—	Clone any voice from a short audio reference sample — no fine-tuning required

Why so many TTS models? Different tasks demand different approaches. Voice Design is perfect for creating brand voices or fictional characters. Zero-shot cloning excels when you need to match an existing voice. Multi-speaker Dia handles scripts with dialogue natively. And the Persian-specific models ensure high-quality Farsi output that general multilingual TTS models often struggle with.

📁 Folder Structure

5-TTS/
├── MULTI_SPEAKER/
│   └── Dia2_2B_Dialogue/
├── TTS_PERSIAN/
│   ├── Chatterbox_TTS_Persian/
│   │   └── t3_fa.safetensors
│   └── F5_TTS_Persian/
├── VOICE_DESIGN/
│   └── Qwen3_TTS_12Hz_VoiceDesign/
│       ├── model.safetensors
│       └── speech_tokenizer/
└── ZERO_SHOT_CLONING/
    └── F5_TTS_v1_Base/
        └── F5TTS_v1_Base/

---

6️⃣ Speech-to-Text (STT)

Transcribe, translate, and preprocess audio — from studio-quality multi-language recognition to specialized Persian ASR.

A complete speech understanding pipeline that starts with audio preprocessing (cleaning and segmentation) and ends with either transcription or translation. Like other categories, it follows the Quality vs. Speed dual-tier design.

🔧 Audio Preprocessing Filters

Filter	Model	Purpose
🎙️ Vocal Isolation	Demucs V4 (ONNX)	Studio-quality vocal separation — removes background music, noise, and other speakers. Essential for noisy recordings
🔇 Silence/Noise Cutter	Silero VAD (ONNX)	Voice Activity Detection — automatically cuts silence and non-speech segments, reducing processing time and improving transcription accuracy

🏆 Quality Tier — High Accuracy

Scope	Model	Size	Highlights
🌍 Multi-Language	Whisper Large V3	Large	OpenAI's flagship multilingual STT model. Supports 99 languages with state-of-the-art accuracy. The gold standard for transcription
🇮🇷 Persian	FastConformer Fa Large	Large	Anti-hallucination Persian ASR by NVIDIA. Specifically designed to avoid the common Whisper problem of fabricating text during silence — critical for Persian content
🇮🇷 Persian	Whisper Persian V4	Large	Whisper fine-tuned specifically on Persian data for improved Farsi recognition accuracy
🌐 Translation	Seamless M4T V2 Large	Large	Meta's universal translation model — translates speech-to-text across 100+ languages. Not just transcription, but actual cross-lingual translation

⚡ Speed Tier — Fastest Processing

Scope	Model	Size	Highlights
🌍 Multi-Language	Whisper Large V3 Turbo	Turbo	8x faster than Whisper Large with only minimal accuracy loss. The sweet spot for real-time or batch processing
🇮🇷 Persian	Whisper Turbo Persian	Turbo	Turbo-speed Whisper variant for Persian language. Ideal for transcribing long Persian audio files quickly

📁 Folder Structure

6-STT/
├── Audio_Preprocessing_Filters/
│   ├── Studio_Vocal_Isolation_DemucsV4/
│   │   └── Studio_Vocal_Isolation_DemucsV4.onnx
│   └── Universal_Silence_Noise_Cutter_Silero/
│       └── Universal_Silence_Noise_Cutter_Silero.onnx
├── Quality/
│   ├── Multi_Language/
│   │   └── Whisper_Large_V3_Global_FP16/
│   ├── Persian_ASR/
│   │   ├── Anti_Hallucination_FastConformer_Fa/
│   │   │   └── stt_fa_fastconformer_hybrid_large.nemo
│   │   └── Whisper_Persian_V4_FP16/
│   └── Translation/
│       └── Seamless_M4T_V2_Large_Translation/
└── Speed/
    ├── Multi_Language/
    │   └── Whisper_Turbo_Global_Speed/
    └── Persian_ASR/
        └── Whisper_Turbo_Persian_Speed/

---

7️⃣ Multimodal

Models that see and hear — process images, audio, and text together for rich understanding.

Multimodal models are the frontier of local AI — they can analyze images, describe screenshots, read documents with visual layouts, and process audio alongside text. This enables workflows like "take a photo of this whiteboard and summarize it" or "analyze this chart and explain the trends."

Tier	Model	Size	Quantization	Key Capabilities
🏆 Quality	Qwen3-Omni 30B-A3B	30B (MoE, 3B active)	Q4_K_M	Full omni-model — understands text, images, AND audio natively. The most capable multimodal model in this collection. MoE architecture keeps VRAM usage manageable
⚡ Speed	Gemma 4 E4B	~4B (effective)	Q4_K_XL	Speculative decoding with MTP draft model for faster generation. Excellent for quick image analysis, screenshot understanding, and visual Q&A. Despite its small effective size, Gemma 4 punches well above its weight

⚠️ CRITICAL — Read before relying on "multimodal": The models shipped here are GGUF quantized files. In their current state, tooling only supports the vision (image) modality from these quantized files — audio input does not work on the quantized versions.

True text + image + audio multimodality is currently only available from the original, uncompressed (FP16/BF16) models — not from the quantized GGUF files in this toolkit. The quantized variants effectively behave as vision-language models (VLM): they can see images and read text, but they cannot process audio.

This is a tooling limitation, not a model defect — current inference engines (llama.cpp, Ollama, LM Studio, etc.) cannot yet route audio through quantized GGUF multimodal pipelines. If you need genuine audio understanding, download and run the original uncompressed weights from the source repos linked below.

I learned this the hard way after downloading — the quantized files simply won't run as full multimodal. Hence this warning.

Why Qwen3-Omni for quality? It's one of the very few models that genuinely handles text + image + audio in a unified architecture. Most "multimodal" models only handle text and images — Qwen3-Omni adds audio understanding, making it a true all-in-one model for multimedia tasks.

📁 Folder Structure

7-Multimodal/
├── Quality/
│   └── Qwen3_Omni_30B_A3B_Quality/
│       ├── Qwen3-Omni-30B-A3B-Instruct-Q4_K_M.gguf
│       └── mmproj-Qwen3-Omni-30B-A3B-Instruct-Q8_0.gguf
└── Speed/
    └── Gemma4_E4B_Speed_Speculative/
        ├── gemma-4-E4B-it-UD-Q4_K_XL.gguf
        ├── mmproj-F16.gguf
        └── MTP/
            └── gemma-4-E4B-it-F16-MTP.gguf

---

8️⃣ Uncensored Models

Unfiltered variants for research, creative writing, and use cases requiring unrestricted model outputs.

These are abliterated (uncensored) versions of the models found in other categories. They have had their refusal mechanisms and output filters removed through fine-tuning. They produce the same quality outputs as their censored counterparts but without refusing requests.

⚠️ Disclaimer: These models are provided for legitimate research, creative, and educational purposes. Users are responsible for ensuring their use complies with applicable laws and ethical guidelines.

Category	Tier	Model	Size	Quantization	Notes
💻 Code	🏆 Quality	Huihui Qwen3-VL 30B (Uncensored)	30B (MoE)	Q4_K_M	Uncensored vision-language model — can analyze code screenshots, diagrams, and visual content without restrictions
🧠 General	🏆 Quality	Huihui Qwen3.6-35B Claude 4.7 Opus (Uncensored)	35B (MoE)	Q4_K	Uncensored general-purpose model with MTP support. Tuned with Claude 4.7 Opus data for enhanced instruction following
💻 Code	⚡ Speed	Huihui Qwen3-Coder 30B (Uncensored)	30B (MoE)	i1-Q3_K_M	Uncensored coding specialist — importance-matrix quantized for better quality at lower bitrates
🧠 General	⚡ Speed	Huihui Qwen3-VL 8B (Uncensored)	8B	Q8_0	Small, fast uncensored VLM — great for quick visual analysis tasks where you need unrestricted output

📁 Folder Structure

8-Uncensored_Models/
├── Quality/
│   ├── Code/
│   │   └── Huihui_Qwen3_VL_30B_Uncensored_Q4_K_M/
│   │       └── Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated.Q4_K_M.gguf
│   └── General/
│       └── Huihui_Qwen3.6_35B_Claude4.7_Opus_MTP_Uncensored_Q4_K/
│           ├── Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-ggml-model-Q4_K.gguf
│           └── mmproj-model-f16.gguf
└── Speed/
    ├── Code/
    │   └── Huihui_Qwen3_Coder_30B_Uncensored_i1_Q3_K_M/
    │       └── Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated.i1-Q3_K_M.gguf
    └── General/
        └── Huihui_Qwen3_VL_8B_Uncensored_Q8_0/
            ├── Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf
            └── Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf

---

🔗 Model Source Links

All models in this toolkit are sourced from Hugging Face. During the initial download process, direct URLs were not recorded. The original sources were recovered by matching SHA-256 hashes against Hugging Face repositories. While some links may point to slightly different file versions within the same quantization family, the hash-verified content is identical to what is hosted in this toolkit.

1 — Universal Foundation LLMs

File	Source
Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf	unsloth/Qwen3.6-35B-A3B-GGUF
mmproj-BF16.gguf (Qwen3.6)	unsloth/Qwen3.6-35B-A3B-GGUF
Qwen3.5-9B-Q5_K_M.gguf	unsloth/Qwen3.5-9B-GGUF
mmproj-BF16.gguf (Qwen3.5)	unsloth/Qwen3.5-9B-GGUF

2 — Coding Agents

File	Source
Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf	unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF
DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf	bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF
qwen2.5-coder-7b-instruct-q5_k_m.gguf	EasierAI/Qwen-2.5-Coder-7B
Qwen2.5-Coder-14B-Q4_K_M.gguf	bartowski/Qwen2.5-Coder-14B-GGUF
DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf	bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF

3 — Agent Orchestration and Routing

File	Source
Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated.gguf	Chungulus/Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated
GLM-4.7-Flash-UD-Q4_K_XL.gguf	unsloth/GLM-4.7-Flash-GGUF
hermes-4_3_36b-Q4_K_M.gguf	NousResearch/Hermes-4.3-36B-GGUF
Gemma4_12B_Agentic_Worker_v2.gguf	yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF
Hermes-4-14B-Q4_K_M.gguf	gabriellarson/Hermes-4-14B-GGUF
Llama-Guard-3-8B.i1-Q4_K_M.gguf	mradermacher/Llama-Guard-3-8B-i1-GGUF

4 — Document Analysis and RAG

File	Source
BGE-M3 Embedding	BAAI/bge-m3
BGE-Reranker-V2-M3	Heisenberg0314/bge-reranker-v2-m3
NuExtract-v1.5-Q8_0.gguf	bartowski/NuExtract-v1.5-GGUF
reader-lm-1.5b-Q8_0.gguf	bartowski/reader-lm-1.5b-GGUF
gemma-4-12b-it-UD-Q4_K_XL.gguf	unsloth/gemma-4-12b-it-GGUF
mmproj-BF16.gguf (Gemma 4 12B)	unsloth/gemma-4-12b-it-GGUF
mtp-gemma-4-12b-it-Q8_0.gguf	unsloth/gemma-4-12b-it-GGUF

5 — Text-to-Speech

File	Source
Dia2 2B Dialogue	nari-labs/Dia2-2B
Chatterbox TTS Persian	Thomcles/Chatterbox-TTS-Persian-Farsi
F5-TTS Persian	Lumos675/F5_TTS_Persian
Qwen3-TTS 12Hz VoiceDesign	Jinstudio/Qwen3-TTS-12Hz-1.7B-VoiceDesign
F5-TTS v1 Base	TopherAU/F5-TTS-v1

6 — Speech-to-Text

File	Source
Demucs V4 ONNX	MrCitron/demucs-v4-onnx
Silero VAD ONNX	istupakov/silero-vad-onnx
Whisper Large V3	openai/whisper-large-v3
FastConformer Fa Large	nvidia/stt_fa_fastconformer_hybrid_large
Whisper Persian V4	nezamisafa/whisper-persian-v4
Seamless M4T V2 Large	facebook/seamless-m4t-v2-large
Whisper Large V3 Turbo	openai/whisper-large-v3-turbo
Whisper Turbo Persian	SadeghK/whisper-large-v3-turbo

7 — Multimodal

File	Source
Qwen3-Omni-30B-A3B-Instruct-Q4_K_M.gguf	ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF
mmproj-Qwen3-Omni-30B-A3B-Instruct-Q8_0.gguf	ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF
gemma-4-E4B-it-UD-Q4_K_XL.gguf	unsloth/gemma-4-E4B-it-GGUF
mmproj-F16.gguf (Gemma 4 E4B)	unsloth/gemma-4-E4B-it-GGUF
gemma-4-E4B-it-F16-MTP.gguf	unsloth/gemma-4-E4B-it-GGUF

8 — Uncensored Models

File	Source
Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated.Q4_K_M.gguf	mradermacher/Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated-GGUF
Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-Q4_K.gguf	huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-MTP-GGUF
mmproj-model-f16.gguf (Huihui 35B)	huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-MTP-GGUF
Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated.i1-Q3_K_M.gguf	mradermacher/Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated-i1-GGUF
Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf	mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF
Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf	mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF

---

🛠️ Recommended Tools for Running These Models

Tool	Best For	Link
Ollama	Easiest setup, CLI-first, model management	ollama.com
LM Studio	GUI-based, beginner-friendly, model browsing	lmstudio.ai
llama.cpp	Maximum control, server mode, API compatible	github.com/ggml-org/llama.cpp
vLLM	High-throughput serving, OpenAI-compatible API	github.com/vllm-project/vllm
Koboldcpp	One-click launcher, built-in GPU support	github.com/lostruins/koboldcpp

---

📋 Hardware Requirements

Component	Minimum	Recommended	This Toolkit's Target
GPU VRAM	8 GB	12 GB	NVIDIA RTX 3060 12GB
System RAM	16 GB	32 GB	32 GB DDR4
Storage	100 GB SSD	500 GB NVMe	SSD strongly recommended
OS	Windows/Linux/macOS	Linux (Ubuntu)	Any with CUDA support

💡 MoE Advantage: Many models in this toolkit use Mixture-of-Experts (MoE) architecture, which activates only a fraction of total parameters during inference. This means you get the quality of much larger models while staying within 12GB VRAM constraints.

---

⭐ Acknowledgments

All models in this toolkit are the work of their respective creators and are used in accordance with their original licenses. Special thanks to:

Model Families: Qwen · DeepSeek · Gemma · Hermes/NousResearch · Whisper/OpenAI · Llama/Meta · GLM · BGE/BAAI

Quantization: unsloth · bartowski · mradermacher · ggml-org · Chungulus

Uncensored Variants: huihui-ai · mradermacher

---

Assembled with AI-assisted model selection · Curated for RTX 3060 12GB · Updated as stronger models arrive