Phi-2-ORPO

Phi-2-ORPO is a fine-tuned version of microsoft/phi-2 on argilla/dpo-mix-7k preference dataset using Odds Ratio Preference Optimization (ORPO). The model has been trained for 1 epoch.

LazyORPO

This model has been trained using LazyORPO. A colab notebook that makes the training process much easier. Based on ORPO paper. This notebook has been created by Zain Ul Abideen

What is ORPO?

Odds Ratio Preference Optimization (ORPO) proposes a new method to train LLMs by combining SFT and Alignment into a new objective (loss function), achieving state of the art results. Some highlights of this techniques are:

• 🧠 Reference model-free → memory friendly
• 🔄 Replaces SFT+DPO/PPO with 1 single method (ORPO)
• 🏆 ORPO Outperforms SFT, SFT+DPO on PHI-2, Llama 2, and Mistral
• 📊 Mistral ORPO achieves 12.20% on AlpacaEval2.0, 66.19% on IFEval, and 7.32 on MT-Bench out Hugging Face Zephyr Beta

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.set_default_device("cuda")

model = AutoModelForCausalLM.from_pretrained("abideen/phi2-pro", torch_dtype="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("abideen/phi2-pro", trust_remote_code=True)

inputs = tokenizer('''
   """
   Write a detailed analogy between mathematics and a lighthouse.
   """''', return_tensors="pt", return_attention_mask=False)

outputs = model.generate(**inputs, max_length=200)
text = tokenizer.batch_decode(outputs)[0]
print(text)

Evaluation

COMING SOON