Instructions to use aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2

SGLang

How to use aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2 with Docker Model Runner:
```
docker model run hf.co/aws-neuron/Mistral-7B-Instruct-v0.2-seqlen-2048-bs-1-cores-2
```

Neuronx model for mistralai/Mistral-7B-Instruct-v0.2

This repository contains AWS Inferentia2 and neuronx compatible checkpoints for meta-llama/Llama-2-7b-hf. You can find detailed information about the base model on its Model Card.

This model has been exported to the neuron format using specific input_shapes and compiler parameters detailed in the paragraphs below.

Please refer to the 🤗 optimum-neuron documentation for an explanation of these parameters.

Usage with 🤗 `optimum-neuron`

>>> from optimum.neuron import pipeline

>>> p = pipeline('text-generation', 'aws-neuron/Mistral-7B-Instruct-v0.2-Neuron-inf2.8xlarge')
>>> p("<s>[INST] Tell me something interesting about AWS. [/INST]", max_new_tokens=64, do_sample=True, top_k=50)
[{'generated_text': "<s>[INST] Tell me something interesting about AWS. [/INST] I'd be happy to tell you something interesting about Amazon Web Services (AWS). AWS is the world's most extensive and rapidly expanding cloud computing platform, offering over 200 fully featured services from data centers globally. It is used by millions of customers, including the largest enterprises and the h"}]

Compilation of your own version

Deploy an AWS inf2.8xlarge or larger instance. Deploy using the Hugging Face Deep Learning AMI so you have all the software installed. This model was compiled and tested on version 20240123 Download a copy locally so that you can edit the config.json file to set the sliding_window value to 4096 (instead of null)

(See https://github.com/aws-neuron/transformers-neuronx/issues/71 for a reason why)

git clone https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2

Then edit config.json to change sliding_window to 4096

Then the standard compilation process will work. You can change your arguments.

model_to_test = "Mistral-7B-Instruct-v0.2"

from optimum.neuron import NeuronModelForCausalLM
#num_cores should be changed based on the instance.  inf2.24xlarge has 6 neuron processors (they have two cores each) so 12 total
#larger models will need more cores.  You can make your model smaller by changing fp16 to f8.  Some models may requre num_cores to be a power of 2 
compiler_args = {"num_cores": 2, "auto_cast_type": 'fp16'}
input_shapes = {"batch_size": 1, "sequence_length": 2048}


model = NeuronModelForCausalLM.from_pretrained(model_to_test, export=True, **compiler_args, **input_shapes) 

from optimum.neuron import pipeline
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_to_test)

p = pipeline('text-generation', model, tokenizer)
p("<s>[INST] Tell me something interesting about AWS. [/INST]", max_new_tokens=64, do_sample=True, top_k=50)

model.save_pretrained("Mistral-7B-Instruct-v0.2-Neuron-inf2.8xlarge")

Arguments passed during export

input_shapes

{
  "batch_size": 1,
  "sequence_length": 2048,
}

compiler_args

{
  "auto_cast_type": "bf16",
  "num_cores": 2,
}

Downloads last month: 2

Neuronx model for mistralai/Mistral-7B-Instruct-v0.2

Usage with 🤗 optimum-neuron

Compilation of your own version

Arguments passed during export

Usage with 🤗 `optimum-neuron`