Instructions to use Sweaterdog/Andy-4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Sweaterdog/Andy-4 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Sweaterdog/Andy-4",
	filename="Andy-4.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use Sweaterdog/Andy-4 with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Sweaterdog/Andy-4:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Sweaterdog/Andy-4:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Sweaterdog/Andy-4:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Sweaterdog/Andy-4:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Sweaterdog/Andy-4:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Sweaterdog/Andy-4:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Sweaterdog/Andy-4:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Sweaterdog/Andy-4:Q4_K_M

Use Docker

docker model run hf.co/Sweaterdog/Andy-4:Q4_K_M

LM Studio
Jan
Ollama
How to use Sweaterdog/Andy-4 with Ollama:
```
ollama run hf.co/Sweaterdog/Andy-4:Q4_K_M
```

Unsloth Studio new

How to use Sweaterdog/Andy-4 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Sweaterdog/Andy-4 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Sweaterdog/Andy-4 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Sweaterdog/Andy-4 to start chatting

Docker Model Runner
How to use Sweaterdog/Andy-4 with Docker Model Runner:
```
docker model run hf.co/Sweaterdog/Andy-4:Q4_K_M
```

Lemonade

How to use Sweaterdog/Andy-4 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Sweaterdog/Andy-4:Q4_K_M

Run and chat with the model

lemonade run user.Andy-4-Q4_K_M

List all available models

lemonade list

Using Andy with Macos

by PhilNan - opened Mar 4

Discussion

PhilNan

Mar 4

•

edited Mar 4

Hello every one,
I'm the happy owner of a Mac Studio 4M with 36Gb shared ram, and Tahoe 26.x software plus the latest ollama version.
I was eager to try mindcraft with a local only solution. And Sweater/Andy-4 models seemed to be a good solution.
It was with andy-4:micro-q8_0. With that little nuance that it is hallucinating a lot despite a strict prompt, and doing whatever random, unexpected, undesired things to ruin your minecraft world.
So, having plenty of ram, I tried to give andy-4:latest a chance.
Unfortunately, it ended with a code 500. It couldn't be memory. So I dug into the log :

Assertion failed: (found), function llama_sampler_dist_apply, file llama-sampling.cpp, line 660. SIGABRT: abort
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect

As I'm not specialized in ollama model making, whether on MAc or any other system, here's my question :

Is there a possible fix for this problem, or is it too deep and costly to investigate and correct? (i.e. Mac architecture is Arm64 based and doesn't have Cuda's instruction set.)
If, hopefully, there is a fix, would you be kind enough to drive me through the process of applying the correction, with as much detail as possible.
Should you need more informations, just tell me what. I didn't dare copy/pasting the 50k log here not to pollute the server with no need.
If it is necessary to train the model on MacOs Tahoe 26.x, I understand you don't want to do it for the few people using that configuration. So, if I'm guided to do the job, I'll be pleased to participate.
Thanks a lot.

Sweaterdog

Owner Mar 4

•

edited Mar 4

Hi there!

I am not sure if you know but Andy-4.1 just released, I would recommend using that model. Among other things, I would highly recommend using LM Studio instead of Ollama, it is much easier to configure, and set up.

EDIT: Also, Ollama's engine has recently blocked all models that are not using their strict specification, which means most models won't work on Ollama anymore. MacOS isn't the issue.

PhilNan

Mar 5

Hi!
Thanks a lot for your quick and pertinent answer.
I switched to LM Studio
I experienced odd issues with Minecraft-CE/Andy-4-GGUF Q4_K_M : Since I'm not familiar with LM Studio, It would block me requiring 59GbRam! After a some exploration and guessing game, I finally managed to get it working with proper settings.
Thanks again.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment