Running 13 Multilingual Tokenizer Leaderboard π 13 Evaluating and comparing tokenizers of language models
Snowflake/snowflake-arctic-embed-xs Sentence Similarity β’ 22.6M β’ Updated Dec 13, 2024 β’ 375k β’ β’ 43
mlx-community/translategemma-4b-it-4bit_immersive-translate Text Generation β’ 0.6B β’ Updated Apr 19 β’ 1.22k β’ 10