Transformers documentation

MOSS-Transcribe-Diarize

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

该模型于 2026-10-14 贡献至 Hugging Face Transformers。

MOSS-Transcribe-Diarize

概述

MOSS-Transcribe-Diarize 0.9B 是一款面向长音频多说话人转写、说话人分离和时间戳标注的端到端音频理解模型。它集成了 Whisper 风格的编码器、4× 帧合并步骤、多模态投影器以及 Qwen3 语言模型。

模型检查点可在以下位置获取：itazap/MOSS-Transcribe-Diarize-HF

主要功能包括：

在单次处理中完成联合转录与说话人分离，片段格式为 [start][S01]text[end]。
通过处理器上游的 Whisper-window 分块处理长音频；模型会按样本重新组装这些分块。
通过 apply_transcription_request 或聊天模板实现可提示转录。
支持 50 多种语言的多语言支持。

该模型由 Hugging Face 团队的 Ita Zaporozhets 贡献。有关更多详细信息，请参阅模型卡片和 OpenMOSS 仓库。

用法

以下大多数示例使用了 apply_transcription_request，这是一个为您构建聊天模板的便捷函数。有关等效的 apply_chat_template 调用，请参阅使用聊天模板的高级用法。

基础转录与说话人分离

每个说话人轮次都被解码为 [start][S0N]text[end] 片段。processor.decode 可以返回原始解码字符串、以字典列表形式表示的解析后片段（return_format="parsed"），或仅返回拼接后的转录文本（return_format="transcription_only"）。

from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/podcast_sample.wav"
inputs = processor.apply_transcription_request(audio_url)
inputs = inputs.to(model.device, dtype=model.dtype)

outputs = model.generate(**inputs, do_sample=False, max_new_tokens=512)
generated_ids = outputs[:, inputs.input_ids.shape[1] :]

decoded = processor.decode(generated_ids, skip_special_tokens=True)
print("RAW OUTPUT:", decoded)

parsed = processor.decode(generated_ids, return_format="parsed")[0]
print("\nPARSED (list of dicts):")
for segment in parsed:
    print(segment)

transcription = processor.decode(generated_ids, return_format="transcription_only")
print("\nTRANSCRIPTION ONLY:", transcription[0])

"""
RAW OUTPUT: ["[0.48][S01] Hey everyone, welcome back to the show.[2.43][2.46][S01] Today, we're talking about the best way to learn a new skill.[5.22][5.43][S01] I'm here with Jordan to dig into it.[7.14][8.40][S02] Thanks for having me.[9.42][9.63][S02] My take is that consistency beats intensity every single time.[12.87][13.14][S01] That makes sense.[13.92][13.92][S01] Even 20 minutes a day adds up over a few months.[16.17][16.32][S02] Exactly, and writing down small wins keeps you motivated along the way.[19.62]"]

PARSED (list of dicts):
{'Start': 0.48, 'End': 2.43, 'Speaker': 1, 'Content': 'Hey everyone, welcome back to the show.'}
{'Start': 2.46, 'End': 5.22, 'Speaker': 1, 'Content': "Today, we're talking about the best way to learn a new skill."}
{'Start': 5.43, 'End': 7.14, 'Speaker': 1, 'Content': "I'm here with Jordan to dig into it."}
{'Start': 8.4, 'End': 9.42, 'Speaker': 2, 'Content': 'Thanks for having me.'}
{'Start': 9.63, 'End': 12.87, 'Speaker': 2, 'Content': 'My take is that consistency beats intensity every single time.'}
{'Start': 13.14, 'End': 13.92, 'Speaker': 1, 'Content': 'That makes sense.'}
{'Start': 13.92, 'End': 16.17, 'Speaker': 1, 'Content': 'Even 20 minutes a day adds up over a few months.'}
{'Start': 16.32, 'End': 19.62, 'Speaker': 2, 'Content': 'Exactly, and writing down small wins keeps you motivated along the way.'}

TRANSCRIPTION ONLY: Hey everyone, welcome back to the show. Today, we're talking about the best way to learn a new skill. I'm here with Jordan to dig into it. Thanks for having me. My take is that consistency beats intensity every single time. That makes sense. Even 20 minutes a day adds up over a few months. Exactly, and writing down small wins keeps you motivated along the way.
"""

自定义提示词和热词

apply_transcription_request 接受 keywords 参数，用于将转录结果偏向特定术语（例如专有名词或领域特定词汇），这些术语在聊天模板中呈现为热词列表：（“hotword list”）。还可以同时传入自定义的 prompt 以提供额外的上下文。

下方，我们转录了一段提及“Yorùbá”的音频片段，并对比了传入该关键词与未传入该关键词时的结果。

from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/languages_sample.wav"

# Without keywords
inputs = processor.apply_transcription_request(audio_url)
inputs = inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs, do_sample=False, max_new_tokens=128)
transcription = processor.decode(outputs[:, inputs.input_ids.shape[1] :], return_format="transcription_only")
print(f"WITHOUT keywords: {transcription[0]}")

# With keywords
inputs = processor.apply_transcription_request(audio_url, keywords=["Mandarin", "Yorùbá"])
inputs = inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs, do_sample=False, max_new_tokens=128)
transcription = processor.decode(outputs[:, inputs.input_ids.shape[1] :], return_format="transcription_only")
print(f"WITH keywords   : {transcription[0]}")

"""
WITHOUT keywords: Let's talk about the most spoken languages in the world. Mandarin Chinese has the most native speakers, followed by Spanish, English, and Hindi. In West Africa, one of the most widely spoken languages is Yoruba, with over 50 million speakers.
WITH keywords   : Let's talk about the most spoken languages in the world. Mandarin Chinese has the most native speakers, followed by Spanish, English, and Hindi. In West Africa, one of the most widely spoken languages is Yorùbá, with over 50 million speakers.
"""

使用聊天模板的高级用法

apply_transcription_request 等效于一个仅包含音频的用户轮次的聊天模板：

from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav"
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "url": audio_url},
        ],
    },
]

inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
).to(model.device, dtype=model.dtype)

outputs = model.generate(**inputs, do_sample=False, max_new_tokens=128)
decoded = processor.decode(
    outputs[:, inputs.input_ids.shape[1] :],
    skip_special_tokens=True,
)
print(decoded)

批量推理

传入一个音频列表，并可选地传入与之匹配的提示词和/或关键词列表，以便在一次调用中批量转录。将某个条目设置为 None 即可在对应样本中跳过该条目。

from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
# `from_pretrained` is used here for convenience, the model can also be built from a config directly.
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

audio = [
    "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav",
    "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/languages_sample.wav",
]
keywords = [None, ["Mandarin", "Yorùbá"]]

inputs = processor.apply_transcription_request(audio, keywords=keywords)
inputs = inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs, do_sample=False, max_new_tokens=128)
transcription = processor.decode(outputs[:, inputs.input_ids.shape[1] :], return_format="transcription_only")
print(transcription)

训练

将目标转录文本以模型带时间戳和说话人标签的输出格式，通过 transcription 传递给 apply_transcription_request。返回的 labels 用于计算损失。

from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
model.train()

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav"
transcription = (
    "[0.00][S01] Welcome to this demonstration of automatic speech recognition and speaker diarization using an "
    "end-to-end transformer model.[7.47]"
)

inputs = processor.apply_transcription_request(audio_url, transcription=transcription)
inputs.to(model.device, dtype=model.dtype)

loss = model(**inputs).loss
print("Loss:", loss.item())
loss.backward()

Torch 编译

该模型可以编译以加快推理/训练速度。device_map="auto" 会使用 accelerate 钩子包装前向传播，从而导致反复重新编译，因此建议直接将模型加载到单个设备上。

在 B200 上，我们观察到当批处理大小为 4 时，速度提升了约 2.1 倍。

import time

import torch

from transformers import AutoProcessor, AutoModelForCausalLM


model_id = "itazap/MOSS-Transcribe-Diarize-HF"

num_warmup = 5
num_runs = 20

# Load processor + model
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to("cuda")

# Prepare static inputs
audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav"
inputs = processor.apply_transcription_request([audio_url] * 4).to(model.device, torch.bfloat16)

# Benchmark without compile
print("Warming up without compile...")
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**inputs)

torch.accelerator.synchronize()

print("\nBenchmarking without torch.compile...")
torch.accelerator.synchronize()
start = time.time()
with torch.no_grad():
    for _ in range(num_runs):
        _ = model(**inputs)
torch.accelerator.synchronize()
no_compile_time = (time.time() - start) / num_runs
print(f"Average time without compile: {no_compile_time:.4f}s")

# Benchmark with compile
print("\nCompiling model...")
model = torch.compile(model)

print("Warming up with compile (includes graph capture)...")
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**inputs)

torch.accelerator.synchronize()

print("\nBenchmarking with torch.compile...")
torch.accelerator.synchronize()
start = time.time()
with torch.no_grad():
    for _ in range(num_runs):
        _ = model(**inputs)
torch.accelerator.synchronize()
compile_time = (time.time() - start) / num_runs
print(f"Average time with compile: {compile_time:.4f}s")

speedup = no_compile_time / compile_time
print(f"\nSpeedup: {speedup:.2f}x")

对于自回归转录，torch.compile 通过提供一个 CompileConfig 对象，而不是直接编译 model.forward，来加速 generate 内部逐 token 的前向传播。

import torch
from transformers import AutoProcessor, AutoModelForCausalLM, CompileConfig

model_id = "itazap/MOSS-Transcribe-Diarize-HF"
num_warmup = 3

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to("cuda")

audio_url = "https://huggingface.co/datasets/itazap/audio_samples/resolve/main/intro_sample.wav"
inputs = processor.apply_transcription_request([audio_url] * 4).to(model.device, torch.bfloat16)

compile_config = CompileConfig()

# Warmup
for _ in range(num_warmup):
    _ = model.generate(**inputs, max_new_tokens=128, cache_implementation="static", compile_config=compile_config)

# Apply model
generated_ids = model.generate(
    **inputs, max_new_tokens=128, cache_implementation="static", compile_config=compile_config
)
transcription = processor.decode(generated_ids[:, inputs.input_ids.shape[1] :], return_format="transcription_only")
print(transcription)

Transformers

MOSS-Transcribe-Diarize

概述

用法

基础转录与说话人分离

自定义提示词和热词

使用聊天模板的高级用法

批量推理

训练

Torch 编译

MossTranscribeDiarizeConfig

class transformers.MossTranscribeDiarizeConfig

MossTranscribeDiarizeAudioConfig

类 transformers.MossTranscribeDiarizeAudioConfig

MossTranscribeDiarizeAudioModel

class transformers.MossTranscribeDiarizeAudioModel

forward

MossTranscribeDiarizeFeatureExtractor

class transformers.MossTranscribeDiarizeFeatureExtractor

__call__

MossTranscribeDiarizeProcessor

class transformers.MossTranscribeDiarizeProcessor

__call__

apply_chat_template

apply_transcription_request

MossTranscribeDiarizeModel

类 transformers.MossTranscribeDiarizeModel

forward

get_audio_features

MossTranscribeDiarizeForConditionalGeneration

class transformers.MossTranscribeDiarizeForConditionalGeneration

forward

get_audio_features

call

call