distily/verify_v0.3.0

Summary

Distilled with Distily library using teacher model gpt2 on dataset wikimedia/wikipedia.

Model Architecture:

• Architecture: GPT2LMHeadModel
• Total Parameters: 124,439,808
• Data Type (dtype): torch.bfloat16
• Model Size: 0.24 GB

Benchmark Metrics Comparison

Metric	dataset_sample_size=1000	teacher
ai2_arc (acc)	0.225	0.304
ai2_arc (acc_norm)	0.251	0.309
ai2_arc (acc_norm_stderr)		0.01
ai2_arc (acc_stderr)		0.01
arc_challenge (acc)	0.182	0.184
arc_challenge (acc_norm)	0.223	0.214
arc_challenge (acc_norm_stderr)		0.013
arc_challenge (acc_stderr)		0.012
arc_easy (acc)	0.268	0.424
arc_easy (acc_norm)	0.278	0.405
arc_easy (acc_norm_stderr)		0.016
arc_easy (acc_stderr)		0.016
boolq (acc)	0.375	0.541
boolq (acc_stderr)		0.016
cola (mcc)	0.0	0.009
cola (mcc_stderr)		0.032
glue (acc)	0.477	0.41
glue (acc_stderr)		0.006
glue (f1)	0.0	0.526
glue (f1_stderr)		0.014
glue (mcc)	0.0	0.009
glue (mcc_stderr)		0.032
hellaswag (acc)	0.287	0.337
hellaswag (acc_norm)	0.269	0.384
hellaswag (acc_norm_stderr)		0.015
hellaswag (acc_stderr)		0.015
mnli (acc)	0.335	0.323
mnli (acc_stderr)		0.015
mnli_mismatch (acc)	0.357	0.344
mnli_mismatch (acc_stderr)		0.015
mrpc (acc)	0.316	0.515
mrpc (acc_stderr)		0.025
mrpc (f1)	0.0	0.631
mrpc (f1_stderr)		0.024
qnli (acc)	0.527	0.472
qnli (acc_stderr)		0.016
qqp (acc)	0.673	0.34
qqp (acc_stderr)		0.015
qqp (f1)	0.0	0.483
qqp (f1_stderr)		0.017
rte (acc)	0.527	0.516
rte (acc_stderr)		0.03
sst2 (acc)	0.557	0.511
sst2 (acc_stderr)		0.017
wikitext (bits_per_byte)	1.979
wikitext (byte_perplexity)	3.942
wikitext (word_perplexity)	1533.0
wnli (acc)	0.437	0.451
wnli (acc_stderr)		0.059

Resource Usage Comparison

• VRAM Use: 7.4923 GB

Distillation (Teacher -> Student) Architecture Difference:

• Architecture: GPT2LMHeadModel -> GPT2LMHeadModel
• Total Parameters: 124,439,808 -> 124,439,808
• Data Type (dtype): torch.bfloat16 -> torch.bfloat16
• Model Size: 0.24 GB -> 0.24 GB

Module Diff Details

Train Dataset

Trained on 923,203 tokens from the wikimedia/wikipedia dataset.

• Num Samples: 990
• Subset: 20231101.en
• Split: train

Training Objective

DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl))

Hyperparameters

The following hyperparameters were used during training:

Expand

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.2
• num_epochs: 1.0
• distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl))
• train_embeddings: True
• lr_scheduler: <torch.optim.lr_scheduler.LambdaLR object at 0x7ff7e81bb7c0>
• student_model_name_or_path: None
• student_config_name_or_path: None
• student_model_config: None
• reinitialize_weights: None
• copy_teacher_modules: [('lm_head', False)]
• student_model_as_bitnet: True
• student_model_compile: False
• dropout: None
• teacher_model_name_or_path: gpt2
• teacher_load_in_8bit: False
• teacher_load_in_4bit: False
• teacher_model_compile: False
• dataset_uri: wikimedia/wikipedia
• dataset_subset: 20231101.en
• dataset_split: train
• dataset_column_name: text
• dataset_sample_size: 1000
• dataset_test_size: 0.01
• gradient_accumulation_steps: 1
• weight_decay: 0.0
• max_grad_norm: 1.0
• warmup_ratio: 0.2
• warmup_steps: 0
• gradient_checkpointing: True

Framework Versions

• Distily 0.3.0
• Transformers 4.44.2
• Pytorch 2.3.0
• Datasets 2.21.0