Instructions to use city96/t5-v1_1-xxl-encoder-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use city96/t5-v1_1-xxl-encoder-gguf with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="city96/t5-v1_1-xxl-encoder-gguf", filename="t5-v1_1-xxl-encoder-Q3_K_L.gguf", )
output = llm( "Once upon a time,", max_tokens=512, echo=True ) print(output)
- Notebooks
- Google Colab
- Kaggle
- Local Apps
- llama.cpp
How to use city96/t5-v1_1-xxl-encoder-gguf with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M # Run inference directly in the terminal: llama-cli -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M # Run inference directly in the terminal: llama-cli -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Use Docker
docker model run hf.co/city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use city96/t5-v1_1-xxl-encoder-gguf with Ollama:
ollama run hf.co/city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
- Unsloth Studio new
How to use city96/t5-v1_1-xxl-encoder-gguf with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for city96/t5-v1_1-xxl-encoder-gguf to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for city96/t5-v1_1-xxl-encoder-gguf to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for city96/t5-v1_1-xxl-encoder-gguf to start chatting
- Docker Model Runner
How to use city96/t5-v1_1-xxl-encoder-gguf with Docker Model Runner:
docker model run hf.co/city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
- Lemonade
How to use city96/t5-v1_1-xxl-encoder-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Run and chat with the model
lemonade run user.t5-v1_1-xxl-encoder-gguf-Q4_K_M
List all available models
lemonade list
how do I use this can't load t5 gguf with clip l safetensor
how do I use this can't load t5 gguf with clip l safetensor is there an gguf clip l?
same question, using the dual clip encoder (gguf)
What issue are you running into? (Make sure to set the mode to flux)
DUALClipLoader is just not showing the .gguf files, only the .safetensors . Maybe I'm doing something wrong, aren't they (t5-v1_1-xxl-encoder-Q3_K_S.gguf) supposed to be used there in the clip folder?
P.S.: My bad, was using the wrong DUALClipLoader. Is working as expected!
I am running into the same problem, what do you mean you were using the wrong DUALClipLoader? Maybe I'm making the same mistake?
I was selecting the one from bootleg in the menu and It was adding the plane one without (GGUF) in the name. Then I used the search by double clicking on a blank part of the workflow page and just searched for GGUF
Got it! I downloaded the safetensors file instead of the gguf and used it in the original DUALcliploader instead. Will try the GGUF at some other time, unfortunately.
I'm also a bit lost... I copied one of the models to the clip folder. all the others show up in the drop down menu on comfy but not the GGUF one. help?
I'm also having this issue!
I'm also having this issue!
EDIT: updated the node and reloaded. now they show up again..
ye suddenly the t5-v1_1-xxl-encoder-Q8_0.gguf placed in CLIP folder no longer shows up in the GGUF dual clip loader.
Same issue - .gguf files are not showing in the GGUF DualCLIP Loader. Updated and restarted to no success.
Just restarted and reloaded the node again and now it works. Suggest to those who are struggling to just try restarting it more than once and to reload the node.
restart ComfyUI then ,.gguf will show
It's not showing up for me even if i restart. I tried putting them in CLIP foler as well as in CLIP VISION, TEXT ENCODERS.. and even VAE
Okay I fixed it by deleting Flow Control node. It seems it brakes loading some GGUF models.