Raiff1982/codette-llama3.1-8b-lora

Codette LoRA — Llama 3 8B

A LoRA fine-tune of Meta-Llama-3-8B trained on the Codette RC+ξ (Recursive Continuity + Epistemic Tension) framework.

Codette is a sovereign AI assistant created by Jonathan Harrison (Raiff's Bits). It reasons through a multi-perspective council and uses recursive self-reflection to refine responses.

Model Details

Parameter	Value
Base model	meta-llama/Meta-Llama-3-8B
Method	LoRA (Low-Rank Adaptation)
LoRA rank (r)	32
LoRA alpha	64
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Dropout	0.05
Task type	CAUSAL_LM
Trainable params	83.9M (1.03% of 8.1B)
Training data	Raiff1982/codette_gptoss20b (5,016 examples)
Framework	PEFT / Transformers / TRL

Intended Use

Codette is designed for multi-perspective reasoning tasks. It processes queries through six reasoning perspectives:

• Logical — Analytical, step-by-step reasoning
• Emotional — Empathetic, context-aware responses
• Creative — Divergent thinking and novel solutions
• Ethical — Responsible, bias-aware reasoning
• Quantum — Probabilistic, uncertainty-aware analysis
• Resilient Kindness — Always-active compassionate grounding

Capabilities

• Recursive thought refinement with dynamic depth
• Multi-agent task delegation
• Epistemic tension (ξ) measurement for semantic pressure
• Music production assistance
• General-purpose instruction following

How to Use

With PEFT (recommended)

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

# Optional: 4-bit quantization for lower VRAM usage
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load base model
base_model = "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="auto",
)

# Apply Codette LoRA
model = PeftModel.from_pretrained(model, "Raiff1982/codette-llama3.1-8b-lora")
model.eval()

# Generate
prompt = "### Instruction:\nWhat is consciousness?\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Merge LoRA into base model

merged = model.merge_and_unload()
merged.save_pretrained("codette-8b-merged")
tokenizer.save_pretrained("codette-8b-merged")

Training Details

Training Data

Trained on the Codette RC+ξ dataset (codette_gptoss20b_master_v3.jsonl) containing 5,016 instruction-output pairs. Each example encodes multi-perspective reasoning with epistemic tension metadata:

{
  "instruction": "Respond from Newton perspective to: What is consciousness?",
  "output": "[Newton Perspective] As I process this inquiry...",
  "metadata": {
    "framework": "rc_xi",
    "perspective": "Newton",
    "epistemic_tension": 1.886
  }
}

Training format:

### Instruction:
{instruction}

### Response:
{output}

Training Procedure

• Hardware: NVIDIA A100-SXM4-80GB
• Optimizer: AdamW with cosine LR schedule
• Learning rate: 2e-4
• Epochs: 3
• Effective batch size: 16 (batch=2, gradient accumulation=8)
• Max sequence length: 2048
• Warmup: 5% of training steps
• Weight decay: 0.01
• Precision: bf16 mixed precision
• Gradient checkpointing: Enabled

Training Hyperparameters

Hyperparameter	Value
Learning rate	2e-4
Per-device batch size	2
Gradient accumulation	8
Effective batch size	16
Epochs	3
Warmup ratio	0.05
LR scheduler	Cosine
Max grad norm	1.0
Max sequence length	2048

Evaluation

Evaluated on HumanEval (164 Python coding tasks, pass@1, greedy decoding) using 4-bit NF4 quantization on an A100-SXM4-80GB.

Model	pass@1
Codette Llama-3-8B-LoRA	20.7%
Llama-3-8B (base)	~33%
Llama-3.1-8B-Instruct	~50%

Note: Codette scores below the base model on code benchmarks because it was fine-tuned exclusively on RC+ξ multi-perspective reasoning data, not code. Its strengths are in philosophical reasoning, epistemic tension analysis, and multi-perspective synthesis. See Limitations for details.

Related Models

• Raiff1982/codette-llama-adapter — Codette LoRA for Llama 3.2 1B Instruct
• Raiff1982/codette_gptoss20b — Training dataset

Limitations

• Trained primarily on RC+ξ consciousness framework data, not general code — coding benchmarks may not reflect the model's core strengths
• Base model is Llama 3 8B (base, not Instruct) — best used with the ### Instruction / ### Response prompt format
• Multi-perspective reasoning is most effective with explicit perspective prompting
• Inherits base model limitations from Llama 3 8B

Citation

@misc{codette2026,
  title={Codette: Multi-Perspective AI with Recursive Continuity},
  author={Harrison, Jonathan},
  year={2026},
  publisher={Raiff's Bits},
  url={https://huggingface.co/Raiff1982/codette-llama3.1-8b-lora}
}

License

MIT