Accelerate documentation
示例动物园
入门指南
教程
操作指南
Accelerate
训练
梯度累积本地 SGD低精度(FP8)训练DeepSpeed使用 DeepSpeed 搭配多个模型DDP 通信钩子完全分片数据并行Megatron-LMAmazon SageMakerApple M1 GPUIntel CPUIntel Gaudi编译
推理
概念与基础
Accelerate 的内部机制将大型模型加载到内存中比较不同分布式配置下的性能执行和延迟任务梯度同步FSDP 与 DeepSpeedFSDP1 与 FSDP2上下文并行序列并行低精度训练方法在 TPU 上进行训练
参考
示例动物园
以下列出了部分展示 Accelerate 的教程和脚本,列表并不完整。
官方 Accelerate 示例:
基础示例
这些示例展示了 Accelerate 的基础功能,是绝佳的入门起点
- 最简 NLP 示例
- 在 Jupyter Notebook 中的极简分布式 NLP 示例
- 最简计算机视觉示例
- 在 Jupyter Notebook 中运行的极简分布式计算机视觉示例
- 在 Kaggle 中使用 Accelerate
特定功能示例
这些示例展示了 Accelerate 框架提供的特定功能
完整示例
这些示例一次性展示了在“特定功能示例”中介绍的 Accelerate 的所有功能
- 完整的 NLP 示例
- 完整的计算机视觉示例
- 一个非常完整且可扩展的视觉示例,展示了 SLURM、hydra 以及该框架非常灵活的用法
- 因果语言模型微调示例
- 掩码语言模型微调示例
- 语音预训练示例
- 翻译微调示例
- 文本分类微调示例
- 语义分割微调示例
- 问答微调示例
- 束搜索问答微调示例
- 多项选择题问答微调示例
- 命名实体识别微调示例
- 图像分类微调示例
- 摘要微调示例
- 如何使用 Accelerate 的 AWS SageMaker 集成的端到端示例
- 用于各种 NLP 任务的 Megatron-LM 示例
集成示例
以下是与 Accelerate 集成的库提供的教程:
在这里找不到您的集成?请提交一个 PR 将其包含进来!
Amphion
Catalyst
DALLE2-pytorch
Diffusers
fastai
GradsFlow
imagen-pytorch
Kornia
PyTorch 加速
PyTorch3D
Stable-Dreamfusion
Tez
trlx
Comfy-UI
在科学领域
以下列出了使用 Accelerate 的部分论文(非完整列表)。
在这里找不到您的论文?请提交一个 PR 将其包含进来!
- Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, Omer Levy:“Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation”,2023 年;arXiv:2305.01569。
- Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, Ee-Peng Lim:“Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models”,2023 年;arXiv:2305.04091。
- Arthur Câmara, Claudia Hauff:“Moving Stuff Around: A study on efficiency of moving documents into memory for Neural IR models”,2022 年;arXiv:2205.08343。
- Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang:“High-throughput Generative Inference of Large Language Models with a Single GPU”,2023 年;arXiv:2303.06865。
- Peter Melchior, Yan Liang, ChangHoon Hahn, Andy Goulding:“Autoencoding Galaxy Spectra I: Architecture”,2022 年;arXiv:2211.07890。
- Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, Diyi Yang:“A Cheaper and Better Diffusion Language Model with Soft-Masked Noise”,2023 年;arXiv:2304.04746。
- Ayaan Haque、Matthew Tancik、Alexei A. Efros、Aleksander Holynski、Angjoo Kanazawa:“Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions”,2023 年;arXiv:2303.12789。
- Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, Andrea Vedaldi:“RealFusion: 360° Reconstruction of Any Object from a Single Image”,2023 年;arXiv:2302.10663。
- Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, Hongsheng Li:“Better Aligning Text-to-Image Models with Human Preference”,2023 年;arXiv:2303.14420。
- Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, Yueting Zhuang:“HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace”,2023 年;arXiv:2303.17580。
- Yue Yang, Wenlin Yao, Hongming Zhang, Xiaoyang Wang, Dong Yu, Jianshu Chen:“Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination”,2022 年;arXiv:2210.12261。
- Sheng-Yen Chou, Pin-Yu Chen, Tsung-Yi Ho:“How to Backdoor Diffusion Models?”,2022 年;arXiv:2212.05400。
- Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, Seungryong Kim:“Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation”,2023 年;arXiv:2303.07937。
- Or Patashnik、Daniel Garibi、Idan Azuri、Hadar Averbuch-Elor、Daniel Cohen-Or:“Localizing Object-level Shape Variations with Text-to-Image Diffusion Models”,2023 年;arXiv:2303.11306。
- Dídac Surís、Sachit Menon、Carl Vondrick:“ViperGPT: Visual Inference via Python Execution for Reasoning”,2023 年;arXiv:2303.08128。
- Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, Qifeng Chen:“FateZero: Fusing Attentions for Zero-shot Text-based Video Editing”,2023 年;arXiv:2303.09535。
- Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, Yejin Choi:“NaturalProver: Grounded Mathematical Proof Generation with Language Models”,2022 年;arXiv:2205.12910。
- Elad Richardson、Gal Metzer、Yuval Alaluf、Raja Giryes、Daniel Cohen-Or:“TEXTure: Text-Guided Texturing of 3D Shapes”,2023 年;arXiv:2302.01721。
- Puijin Cheng, Li Lin, Yijin Huang, Huaqing He, Wenhan Luo, Xiaoying Tang:“Learning Enhancement From Degradation: A Diffusion Model For Fundus Image Enhancement”,2023 年;arXiv:2303.04603。
- Shun Shao, Yftah Ziser, Shay Cohen:“Erasure of Unaligned Attributes from Neural Representations”,2023 年;arXiv:2302.02997。
- Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim, Minjoon Seo:“In-Context Instruction Learning”,2023 年;arXiv:2302.14691。
- Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, Anima Anandkumar:“Prismer: A Vision-Language Model with An Ensemble of Experts”,2023 年;arXiv:2303.02506。
- Haoyu Chen, Zhihua Wang, Yang Yang, Qilin Sun, Kede Ma:“Learning a Deep Color Difference Metric for Photographic Images”,2023 年;arXiv:2303.14964。
- Van-Hoang Le, Hongyu Zhang:“基于提示的少样本学习日志解析”,2023 年;arXiv:2302.07435。
- Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi, Kentaro Inui:“Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?”,2023 年;arXiv:2302.07866。
- Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, Prithviraj Ammanabrolu:“Behavior Cloned Transformers are Neurosymbolic Reasoners”,2022 年;arXiv:2210.07382。
- Martin Wessel、Tomáš Horych、Terry Ruas、Akiko Aizawa、Bela Gipp、Timo Spinde:“Introducing MBIB — the first Media Bias Identification Benchmark Task and Dataset Collection”,2023 年;arXiv:2304.13148。DOI:[https://dx.doi.org/10.1145/3539618.3591882 10.1145/3539618.3591882]。
- Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, Daniel Cohen-Or:“Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models”,2023 年;arXiv:2301.13826。
- Marcio Fonseca、Yftah Ziser、Shay B. Cohen:“长文档摘要生成中内容与预算决策的分解”,2022 年;arXiv:2205.12486。
- Elad Richardson、Gal Metzer、Yuval Alaluf、Raja Giryes、Daniel Cohen-Or:“TEXTure: Text-Guided Texturing of 3D Shapes”,2023 年;arXiv:2302.01721。
- Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, Yulia Tsvetkov:“On the Blind Spots of Model-Based Evaluation Metrics for Text Generation”,2022 年;arXiv:2212.10020。
- Ori Ram、Yoav Levine、Itay Dalmedigos、Dor Muhlgay、Amnon Shashua、Kevin Leyton-Brown、Yoav Shoham:“In-Context Retrieval-Augmented Language Models”,2023 年;arXiv:2302.00083。
- Dacheng Li, Rulin Shao, Hongyi Wang, Han Guo, Eric P. Xing, Hao Zhang:“MPCFormer: fast, performant and private Transformer inference with MPC”,2022 年;arXiv:2211.01452。
- Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, Jianfeng Gao:“GODEL: Large-Scale Pre-Training for Goal-Directed Dialog”,2022 年;arXiv:2206.11309。
- Egil Rønningstad、Erik Velldal、Lilja Øvrelid:“实体级情感分析(ELSA):一项探索性任务调查”,2023,《第29届国际计算语言学会议论文集》,2022,第6773-6783页;arXiv:2304.14241。
- Charlie Snell、Ilya Kostrikov、Yi Su、Mengjiao Yang、Sergey Levine:“Offline RL for Natural Language Generation with Implicit Language Q Learning”,2022 年;arXiv:2206.11871。
- Zhiruo Wang, Shuyan Zhou, Daniel Fried, Graham Neubig:“Execution-Based Evaluation for Open-Domain Code Generation”,2022 年;arXiv:2212.10481。
- Minh-Long Luu, Zeyi Huang, Eric P. Xing, Yong Jae Lee, Haohan Wang:“Expeditious Saliency-guided Mix-up through Random Gradient Thresholding”,2022 年;arXiv:2212.04875。
- Jun Hao Liew, Hanshu Yan, Daquan Zhou, Jiashi Feng:“MagicMix: Semantic Mixing with Diffusion Models”,2022 年;arXiv:2210.16056。
- Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao:“LiST: Lite Prompted Self-training Makes Parameter-Efficient Few-shot Learners”,2021 年;arXiv:2110.06274。