Instructions to use alpindale/c4ai-command-r-plus-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use alpindale/c4ai-command-r-plus-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="alpindale/c4ai-command-r-plus-GPTQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("alpindale/c4ai-command-r-plus-GPTQ")
model = AutoModelForCausalLM.from_pretrained("alpindale/c4ai-command-r-plus-GPTQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use alpindale/c4ai-command-r-plus-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "alpindale/c4ai-command-r-plus-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alpindale/c4ai-command-r-plus-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/alpindale/c4ai-command-r-plus-GPTQ

SGLang

How to use alpindale/c4ai-command-r-plus-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "alpindale/c4ai-command-r-plus-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alpindale/c4ai-command-r-plus-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "alpindale/c4ai-command-r-plus-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alpindale/c4ai-command-r-plus-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use alpindale/c4ai-command-r-plus-GPTQ with Docker Model Runner:
```
docker model run hf.co/alpindale/c4ai-command-r-plus-GPTQ
```

How to load command r+ in text-generation-webui?

by MLDataScientist - opened Apr 5, 2024

Discussion

MLDataScientist

Apr 5, 2024

Thank you for converting this model into GPTQ.
If text-generation-webui does not support this model yet, Can you please share a script on how to load the model for inferencing with partial offloading into CPU RAM? I have 36GB VRAM and 96 GB RAM.

Thanks!

MLDataScientist

Apr 5, 2024

By the way, these are the errors I get with different loaders:

AutoGPTQ_loader: TypeError: cohere isn't supported yet.

Transformers (with auto-devices): CUDA out of memory. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

MLDataScientist

Apr 6, 2024

@alpindale let me know. Thanks!

alpindale

Owner Apr 6, 2024

•

edited Apr 7, 2024

Hi! Sorry for the delayed response. This was quantized with a custom script. Inference is currently possible with Aphrodite Engine's dev branch, but no CPU offloading is supported yet. You can find the quantization script here.

Light4Bear

Apr 6, 2024

To run it in text-generation-webui, you need to install the latest transformers package, and choose Transformers loader in webui.

davideuler

Apr 10, 2024

Hi! Sorry for the delayed response. This was quantized with a custom script. Inference is currently possible with Aphrodite Engine's dev branch, but no CPU offloading is supported yet. You can find the quantization script here.

Thanks for the great job, this revision on the dev branch of aphrodite works for Command R Plus: 95faf27d2b39eb34ed59edadcfe24121412decaa

I run it on A100.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment