Hub documentation

Embedding Atlas

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Embedding Atlas

Embedding Atlas์€ ๋Œ€๊ทœ๋ชจ ์ž„๋ฒ ๋”ฉ ๊ณต๊ฐ„์„ ํƒ์ƒ‰ํ•˜๊ธฐ ์œ„ํ•œ ์ธํ„ฐ๋ž™ํ‹ฐ๋ธŒ ์‹œ๊ฐํ™” ๋„๊ตฌ์ž…๋‹ˆ๋‹ค. ์ž„๋ฒ ๋”ฉ๊ณผ ๊ด€๋ จ ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ๋ฅผ ํ•จ๊ป˜ ์‹œ๊ฐํ™”ํ•˜๊ณ  ํฌ๋กœ์Šค ํ•„ํ„ฐ๋ง ๋ฐ ๊ฒ€์ƒ‰ํ•  ์ˆ˜ ์žˆ์–ด, ๊ณ ์ฐจ์› ๋ฐ์ดํ„ฐ์˜ ํŒจํ„ด๊ณผ ๊ด€๊ณ„๋ฅผ ์ดํ•ดํ•˜๋Š” ๋ฐ ๋„์›€์ด ๋ฉ๋‹ˆ๋‹ค. ๋ชจ๋“  ๊ณ„์‚ฐ์€ ์‚ฌ์šฉ์ž์˜ ์ปดํ“จํ„ฐ์—์„œ ์ˆ˜ํ–‰๋˜๋ฏ€๋กœ ๋ฐ์ดํ„ฐ์˜ ํ”„๋ผ์ด๋ฒ„์‹œ์™€ ๋ณด์•ˆ์„ ๋ณด์žฅํ•ฉ๋‹ˆ๋‹ค.

Static Space๋กœ ํ˜ธ์ŠคํŒ…๋œ MegaScience ๋ฐ์ดํ„ฐ์…‹์— ๋Œ€ํ•œ ์˜ˆ์‹œ ์•„ํ‹€๋ผ์Šค๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค:

์ฃผ์š” ๊ธฐ๋Šฅ

  • ์ธํ„ฐ๋ž™ํ‹ฐ๋ธŒ ํƒ์ƒ‰: ์ˆ˜๋ฐฑ๋งŒ ๊ฐœ์˜ ์ž„๋ฒ ๋”ฉ์„ ๋งค๋„๋Ÿฝ๊ณ  ๋ฐ˜์‘์„ฑ์ด ์ข‹์€ ์‹œ๊ฐํ™”๋กœ ํƒ์ƒ‰
  • ๋ธŒ๋ผ์šฐ์ € ๊ธฐ๋ฐ˜ ๊ณ„์‚ฐ: ๋ฐ์ดํ„ฐ๋ฅผ ์™ธ๋ถ€ ์„œ๋ฒ„๋กœ ์ „์†กํ•˜์ง€ ์•Š๊ณ  ๋กœ์ปฌ์—์„œ ์ž„๋ฒ ๋”ฉ๊ณผ ํ”„๋กœ์ ์…˜์„ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค
  • ํฌ๋กœ์Šค ํ•„ํ„ฐ๋ง: ์—ฌ๋Ÿฌ ๋ฉ”ํƒ€๋ฐ์ดํ„ฐ ์—ด์— ๊ฑธ์ณ ๋ฐ์ดํ„ฐ๋ฅผ ์—ฐ๊ฒฐํ•˜๊ณ  ํ•„ํ„ฐ๋ง
  • ๊ฒ€์ƒ‰ ๊ธฐ๋Šฅ: ์ฃผ์–ด์ง„ ์ฟผ๋ฆฌ ๋˜๋Š” ๊ธฐ์กด ํ•ญ๋ชฉ๊ณผ ์œ ์‚ฌํ•œ ๋ฐ์ดํ„ฐ ํฌ์ธํŠธ๋ฅผ ์ฐพ์Šต๋‹ˆ๋‹ค
  • ๋‹ค์–‘ํ•œ ํ†ตํ•ฉ ์˜ต์…˜: ๋ช…๋ น์ค„, Jupyter ์œ„์ ฏ ๋˜๋Š” ์›น ์ธํ„ฐํŽ˜์ด์Šค๋ฅผ ํ†ตํ•ด ์‚ฌ์šฉ

ํ•„์ˆ˜ ์กฐ๊ฑด

๋จผ์ €, Embedding Atlas๋ฅผ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค:

pip install embedding-atlas

Hugging Face Hub์—์„œ ๋น„๊ณต๊ฐœ ๋ฐ์ดํ„ฐ์…‹์„ ๋กœ๋“œํ•  ๊ณ„ํš์ด๋ผ๋ฉด Hugging Face ๊ณ„์ •์œผ๋กœ ๋กœ๊ทธ์ธํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค:

hf auth login

Hub์—์„œ ๋ฐ์ดํ„ฐ์…‹ ๋กœ๋“œ

Embedding Atlas์€ Hugging Face Hub์™€ ์›ํ™œํ•˜๊ฒŒ ํ†ตํ•ฉ๋˜์–ด, ๋ชจ๋“  ๋ฐ์ดํ„ฐ์…‹์˜ ์ž„๋ฒ ๋”ฉ์„ ์ง์ ‘ ์‹œ๊ฐํ™”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋ช…๋ น์ค„ ์‚ฌ์šฉ

Hugging Face ๋ฐ์ดํ„ฐ์…‹์„ ์‹œ๊ฐํ™”ํ•˜๋Š” ๊ฐ€์žฅ ๊ฐ„๋‹จํ•œ ๋ฐฉ๋ฒ•์€ ์ปค๋งจ๋“œ ๋ผ์ธ ์ธํ„ฐํŽ˜์ด์Šค๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. IMDB ๋ฐ์ดํ„ฐ์…‹์œผ๋กœ ์‹œ๋„ํ•ด ๋ณด์„ธ์š”:

# Load the IMDB dataset from the Hub
embedding-atlas stanfordnlp/imdb

# Specify the text column for embedding computation
embedding-atlas stanfordnlp/imdb --text "text"

# Load only a sample for faster exploration
embedding-atlas stanfordnlp/imdb --text "text" --sample 5000

์ž์ฒด ๋ฐ์ดํ„ฐ์…‹์˜ ๊ฒฝ์šฐ ๋™์ผํ•œ ํŒจํ„ด์„ ์‚ฌ์šฉํ•˜์„ธ์š”:

# Load your dataset from the Hub
embedding-atlas username/dataset-name

# Load multiple splits
embedding-atlas username/dataset-name --split train --split test

# Specify custom text column
embedding-atlas username/dataset-name --text "content"

Python๊ณผ Jupyter ์‚ฌ์šฉ

Jupyter ๋…ธํŠธ๋ถ์—์„œ Embedding Atlas๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์ธํ„ฐ๋ž™ํ‹ฐ๋ธŒ ํƒ์ƒ‰์„ ์ˆ˜ํ–‰ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค:

from embedding_atlas.widget import EmbeddingAtlasWidget
from datasets import load_dataset
import pandas as pd

# Load the IMDB dataset from Hugging Face Hub
dataset = load_dataset("stanfordnlp/imdb", split="train[:5000]")

# Convert to pandas DataFrame
df = dataset.to_pandas()

# Create interactive widget
widget = EmbeddingAtlasWidget(df)
widget

์ž์ฒด ๋ฐ์ดํ„ฐ์…‹์˜ ๊ฒฝ์šฐ:

from embedding_atlas.widget import EmbeddingAtlasWidget
from datasets import load_dataset
import pandas as pd

# Load your dataset from the Hub
dataset = load_dataset("username/dataset-name", split="train")
df = dataset.to_pandas()

# Create interactive widget
widget = EmbeddingAtlasWidget(df)
widget

์‚ฌ์ „ ๊ณ„์‚ฐ๋œ ์ž„๋ฒ ๋”ฉ ์ž‘์—…ํ•˜๊ธฐ

pre-computed ์ž„๋ฒ ๋”ฉ์ด ํฌํ•จ๋œ ๋ฐ์ดํ„ฐ์…‹์ด ์žˆ๋Š” ๊ฒฝ์šฐ, ์ด๋ฅผ ์ง์ ‘ ๋กœ๋“œํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

# Load dataset with pre-computed coordinates
embedding-atlas username/dataset-name \
    --x "embedding_x" \
    --y "embedding_y"

# Load with pre-computed nearest neighbors
embedding-atlas username/dataset-name \
    --neighbors "neighbors_column"

์ž„๋ฒ ๋”ฉ ์‚ฌ์šฉ์ž ์ •์˜

Embedding Atlas์€ ๊ธฐ๋ณธ์ ์œผ๋กœ SentenceTransformers๋ฅผ ์‚ฌ์šฉํ•˜์ง€๋งŒ ์‚ฌ์šฉ์ž ์ •์˜ ์ž„๋ฒ ๋”ฉ ๋ชจ๋ธ์„ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค:

# Use a specific embedding model
embedding-atlas stanfordnlp/imdb \
    --text "text" \
    --model "sentence-transformers/all-MiniLM-L6-v2"

# For models requiring remote code execution
embedding-atlas username/dataset-name \
    --model "custom/model" \
    --trust-remote-code

UMAP ํˆฌ์˜ ๋งค๊ฐœ๋ณ€์ˆ˜

ํŠน์ • ์‚ฌ์šฉ ์‚ฌ๋ก€์— ๋งž๊ฒŒ ์ฐจ์› ์ถ•์†Œ๋ฅผ ๋ฏธ์„ธ ์กฐ์ •ํ•˜์„ธ์š”:

embedding-atlas stanfordnlp/imdb \
    --text "text" \
    --umap-n-neighbors 30 \
    --umap-min-dist 0.1 \
    --umap-metric "cosine"

์‚ฌ์šฉ ์‚ฌ๋ก€

ํ…์ŠคํŠธ ๋ฐ์ดํ„ฐ์…‹ ํƒ์ƒ‰

ํด๋Ÿฌ์Šคํ„ฐ, ์ด์ƒ์น˜, ํŒจํ„ด์„ ์‹๋ณ„ํ•˜๊ธฐ ์œ„ํ•ด ํ…์ŠคํŠธ ์ฝ”ํผ์Šค๋ฅผ ์‹œ๊ฐํ™”ํ•˜๊ณ  ํƒ์ƒ‰ํ•ฉ๋‹ˆ๋‹ค:

from embedding_atlas.widget import EmbeddingAtlasWidget
from datasets import load_dataset
import pandas as pd

# Load a text classification dataset
dataset = load_dataset("stanfordnlp/imdb", split="train[:5000]")
df = dataset.to_pandas()

# Visualize with metadata
widget = EmbeddingAtlasWidget(df)
widget

์ถ”๊ฐ€ ์ž๋ฃŒ

Update on GitHub