output = llm(
"Once upon a time,",
max_tokens=512,
echo=True
)
print(output)Phi-3.5 Mini Instruct - Medical LoRA (GGUF Quantized)
This repository contains quantized GGUF versions of a LoRA fine-tuned model based on the microsoft/Phi-3.5-mini-instruct base model.
The fine-tuning was performed using the LoRA adapter:
segestic/phi3.5-mini-4k-qlora-medical-seg-vall_med
🔧 Description
The .gguf files in this repository are merged and quantized versions of the above base model + LoRA, suitable for use with llama.cpp or other GGUF-compatible inference engines.
Each file corresponds to a different quantization level.
🧠 Intended Use
These models are intended for offline inference in medical question-answering or document understanding tasks using the phi-3.5 base and a custom LoRA.
⚠️ Disclaimer
This model is fine-tuned on a medical dataset, but it is not a certified clinical tool. Use in production or decision-making requires proper validation.
- Downloads last month
- 43
4-bit
5-bit
8-bit
32-bit
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="segestic/DrSeg_Phi3.5_GGUF", filename="", )