Instructions to use vanta-research/atom-27b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use vanta-research/atom-27b with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="vanta-research/atom-27b", filename="atom-27b-f16.gguf", )
llm.create_chat_completion( messages = [ { "role": "user", "content": "What is the capital of France?" } ] ) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use vanta-research/atom-27b with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf vanta-research/atom-27b:F16 # Run inference directly in the terminal: llama-cli -hf vanta-research/atom-27b:F16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf vanta-research/atom-27b:F16 # Run inference directly in the terminal: llama-cli -hf vanta-research/atom-27b:F16
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf vanta-research/atom-27b:F16 # Run inference directly in the terminal: ./llama-cli -hf vanta-research/atom-27b:F16
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf vanta-research/atom-27b:F16 # Run inference directly in the terminal: ./build/bin/llama-cli -hf vanta-research/atom-27b:F16
Use Docker
docker model run hf.co/vanta-research/atom-27b:F16
- LM Studio
- Jan
- vLLM
How to use vanta-research/atom-27b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "vanta-research/atom-27b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vanta-research/atom-27b", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/vanta-research/atom-27b:F16
- Ollama
How to use vanta-research/atom-27b with Ollama:
ollama run hf.co/vanta-research/atom-27b:F16
- Unsloth Studio
How to use vanta-research/atom-27b with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for vanta-research/atom-27b to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for vanta-research/atom-27b to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for vanta-research/atom-27b to start chatting
- Docker Model Runner
How to use vanta-research/atom-27b with Docker Model Runner:
docker model run hf.co/vanta-research/atom-27b:F16
- Lemonade
How to use vanta-research/atom-27b with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull vanta-research/atom-27b:F16
Run and chat with the model
lemonade run user.atom-27b-F16
List all available models
lemonade list
Well done!
Well done. I've been following your work since Wraith; your fine-tunes are quite unique and well-executed. While at times they feel a bit too leaning towards safety, though this is simply from vibe testing. Realistically, your models are likely on par with base Gemma in terms of safety, with the heretic being an easy solution if that ever becomes an actual hindrance.
Scout, in its own way, was a charming tune; it was challenging to assess its practical utility due to the size constraints of the models, but an intriguing, novel approach nonetheless.
In an era where fine-tunes are tapering off as base models become increasingly overfit and benchmaxxed, your work has been a refreshing change of pace.
Keep up the excellent work, and thank you for sharing your innovations with the community!
Also, I am unsure if you are affiliated with this account. But, figured I'd at least bring it to your attention: https://huggingface.co/Cheeeeeeeeky/affine-homonculus
Wow, what a nice post! Definitely the highlight of my day, thank you so much! Your observations on my fine-tunes are actually really sharp - I'm not sure anyone has made those connections before (at least not to me π)
I'm really glad you like the Entities family - it's a fun project that's overdue for an update. I have a few ideas floating around for new Entities, so I'll have to jump on that! (If you ever have ideas for entities, feel free to send them my way - I'll cite you in the model card!)
Thank you for the note on safety! I've been playing around with safety/refusals a bit so that feedback is super helpful! I'll run some tests over here and see if I can make any improvements in that area. I've seen a few heretic iterations of my models floating around the hub, so I've kinda just left those to the community. I run everything myself over here so I just haven't had time to explore it.
Thanks for the heads up on the duplicate repo - this is the first I've heard of it. Not affiliated with me, but they're using the original model card and weights, so it's technically within open source norms I suppose. Annoying, but part of the territory π€·ββοΈ Thanks for bringing it to my attention!