| | --- |
| | tags: |
| | - mteb |
| | - feature-extraction |
| | - sentence-similarity |
| | model-index: |
| | - name: v2 |
| | results: |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_counterfactual |
| | name: MTEB AmazonCounterfactualClassification (en) |
| | config: en |
| | split: test |
| | revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
| | metrics: |
| | - type: accuracy |
| | value: 76.56716417910448 |
| | - type: ap |
| | value: 39.864746145463656 |
| | - type: f1 |
| | value: 70.60275403114987 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_polarity |
| | name: MTEB AmazonPolarityClassification |
| | config: default |
| | split: test |
| | revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
| | metrics: |
| | - type: accuracy |
| | value: 93.46427500000001 |
| | - type: ap |
| | value: 90.36283359936121 |
| | - type: f1 |
| | value: 93.45329322673612 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_reviews_multi |
| | name: MTEB AmazonReviewsClassification (en) |
| | config: en |
| | split: test |
| | revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
| | metrics: |
| | - type: accuracy |
| | value: 48.77199999999999 |
| | - type: f1 |
| | value: 48.16695258838576 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: arguana |
| | name: MTEB ArguAna |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 40.184999999999995 |
| | - type: map_at_10 |
| | value: 56.114 |
| | - type: map_at_100 |
| | value: 56.676 |
| | - type: map_at_1000 |
| | value: 56.68 |
| | - type: map_at_3 |
| | value: 51.968 |
| | - type: map_at_5 |
| | value: 54.642 |
| | - type: mrr_at_1 |
| | value: 40.896 |
| | - type: mrr_at_10 |
| | value: 56.388000000000005 |
| | - type: mrr_at_100 |
| | value: 56.95099999999999 |
| | - type: mrr_at_1000 |
| | value: 56.95400000000001 |
| | - type: mrr_at_3 |
| | value: 52.251999999999995 |
| | - type: mrr_at_5 |
| | value: 54.879999999999995 |
| | - type: ndcg_at_1 |
| | value: 40.184999999999995 |
| | - type: ndcg_at_10 |
| | value: 64.253 |
| | - type: ndcg_at_100 |
| | value: 66.47 |
| | - type: ndcg_at_1000 |
| | value: 66.549 |
| | - type: ndcg_at_3 |
| | value: 55.945 |
| | - type: ndcg_at_5 |
| | value: 60.742 |
| | - type: precision_at_1 |
| | value: 40.184999999999995 |
| | - type: precision_at_10 |
| | value: 8.982999999999999 |
| | - type: precision_at_100 |
| | value: 0.991 |
| | - type: precision_at_1000 |
| | value: 0.1 |
| | - type: precision_at_3 |
| | value: 22.499 |
| | - type: precision_at_5 |
| | value: 15.817999999999998 |
| | - type: recall_at_1 |
| | value: 40.184999999999995 |
| | - type: recall_at_10 |
| | value: 89.82900000000001 |
| | - type: recall_at_100 |
| | value: 99.075 |
| | - type: recall_at_1000 |
| | value: 99.644 |
| | - type: recall_at_3 |
| | value: 67.496 |
| | - type: recall_at_5 |
| | value: 79.09 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/arxiv-clustering-p2p |
| | name: MTEB ArxivClusteringP2P |
| | config: default |
| | split: test |
| | revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
| | metrics: |
| | - type: v_measure |
| | value: 49.64684811204023 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/arxiv-clustering-s2s |
| | name: MTEB ArxivClusteringS2S |
| | config: default |
| | split: test |
| | revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
| | metrics: |
| | - type: v_measure |
| | value: 43.6640710523389 |
| | - task: |
| | type: Reranking |
| | dataset: |
| | type: mteb/askubuntudupquestions-reranking |
| | name: MTEB AskUbuntuDupQuestions |
| | config: default |
| | split: test |
| | revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
| | metrics: |
| | - type: map |
| | value: 63.71316367624821 |
| | - type: mrr |
| | value: 77.02534845886647 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/biosses-sts |
| | name: MTEB BIOSSES |
| | config: default |
| | split: test |
| | revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 88.96786300506704 |
| | - type: cos_sim_spearman |
| | value: 88.08212749295554 |
| | - type: euclidean_pearson |
| | value: 87.1561534920524 |
| | - type: euclidean_spearman |
| | value: 88.016463346151 |
| | - type: manhattan_pearson |
| | value: 87.19359910450564 |
| | - type: manhattan_spearman |
| | value: 88.10803169765825 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/banking77 |
| | name: MTEB Banking77Classification |
| | config: default |
| | split: test |
| | revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
| | metrics: |
| | - type: accuracy |
| | value: 87.46103896103897 |
| | - type: f1 |
| | value: 87.4315144014101 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/biorxiv-clustering-p2p |
| | name: MTEB BiorxivClusteringP2P |
| | config: default |
| | split: test |
| | revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
| | metrics: |
| | - type: v_measure |
| | value: 41.03554871732576 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/biorxiv-clustering-s2s |
| | name: MTEB BiorxivClusteringS2S |
| | config: default |
| | split: test |
| | revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
| | metrics: |
| | - type: v_measure |
| | value: 37.974813344124264 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackAndroidRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 34.174 |
| | - type: map_at_10 |
| | value: 45.728 |
| | - type: map_at_100 |
| | value: 47.266999999999996 |
| | - type: map_at_1000 |
| | value: 47.39 |
| | - type: map_at_3 |
| | value: 41.667 |
| | - type: map_at_5 |
| | value: 44.028 |
| | - type: mrr_at_1 |
| | value: 41.202 |
| | - type: mrr_at_10 |
| | value: 51.49 |
| | - type: mrr_at_100 |
| | value: 52.159 |
| | - type: mrr_at_1000 |
| | value: 52.197 |
| | - type: mrr_at_3 |
| | value: 48.379 |
| | - type: mrr_at_5 |
| | value: 50.331 |
| | - type: ndcg_at_1 |
| | value: 41.202 |
| | - type: ndcg_at_10 |
| | value: 52.38699999999999 |
| | - type: ndcg_at_100 |
| | value: 57.611999999999995 |
| | - type: ndcg_at_1000 |
| | value: 59.318000000000005 |
| | - type: ndcg_at_3 |
| | value: 46.516000000000005 |
| | - type: ndcg_at_5 |
| | value: 49.519000000000005 |
| | - type: precision_at_1 |
| | value: 41.202 |
| | - type: precision_at_10 |
| | value: 9.971 |
| | - type: precision_at_100 |
| | value: 1.5879999999999999 |
| | - type: precision_at_1000 |
| | value: 0.20500000000000002 |
| | - type: precision_at_3 |
| | value: 22.031 |
| | - type: precision_at_5 |
| | value: 16.309 |
| | - type: recall_at_1 |
| | value: 34.174 |
| | - type: recall_at_10 |
| | value: 65.32900000000001 |
| | - type: recall_at_100 |
| | value: 86.64 |
| | - type: recall_at_1000 |
| | value: 97.069 |
| | - type: recall_at_3 |
| | value: 48.607 |
| | - type: recall_at_5 |
| | value: 56.615 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackEnglishRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 34.73 |
| | - type: map_at_10 |
| | value: 45.617999999999995 |
| | - type: map_at_100 |
| | value: 46.888000000000005 |
| | - type: map_at_1000 |
| | value: 47.016999999999996 |
| | - type: map_at_3 |
| | value: 42.425000000000004 |
| | - type: map_at_5 |
| | value: 44.214999999999996 |
| | - type: mrr_at_1 |
| | value: 43.631 |
| | - type: mrr_at_10 |
| | value: 52.014 |
| | - type: mrr_at_100 |
| | value: 52.6 |
| | - type: mrr_at_1000 |
| | value: 52.637 |
| | - type: mrr_at_3 |
| | value: 50.021 |
| | - type: mrr_at_5 |
| | value: 51.23799999999999 |
| | - type: ndcg_at_1 |
| | value: 43.631 |
| | - type: ndcg_at_10 |
| | value: 51.458000000000006 |
| | - type: ndcg_at_100 |
| | value: 55.61000000000001 |
| | - type: ndcg_at_1000 |
| | value: 57.462 |
| | - type: ndcg_at_3 |
| | value: 47.461 |
| | - type: ndcg_at_5 |
| | value: 49.312 |
| | - type: precision_at_1 |
| | value: 43.631 |
| | - type: precision_at_10 |
| | value: 9.661999999999999 |
| | - type: precision_at_100 |
| | value: 1.5270000000000001 |
| | - type: precision_at_1000 |
| | value: 0.198 |
| | - type: precision_at_3 |
| | value: 22.823999999999998 |
| | - type: precision_at_5 |
| | value: 16.075999999999997 |
| | - type: recall_at_1 |
| | value: 34.73 |
| | - type: recall_at_10 |
| | value: 61.041999999999994 |
| | - type: recall_at_100 |
| | value: 78.658 |
| | - type: recall_at_1000 |
| | value: 90.215 |
| | - type: recall_at_3 |
| | value: 48.952 |
| | - type: recall_at_5 |
| | value: 54.422000000000004 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackGamingRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 42.047000000000004 |
| | - type: map_at_10 |
| | value: 55.669999999999995 |
| | - type: map_at_100 |
| | value: 56.676 |
| | - type: map_at_1000 |
| | value: 56.728 |
| | - type: map_at_3 |
| | value: 52.275000000000006 |
| | - type: map_at_5 |
| | value: 54.254000000000005 |
| | - type: mrr_at_1 |
| | value: 48.15 |
| | - type: mrr_at_10 |
| | value: 59.036 |
| | - type: mrr_at_100 |
| | value: 59.650999999999996 |
| | - type: mrr_at_1000 |
| | value: 59.675 |
| | - type: mrr_at_3 |
| | value: 56.760999999999996 |
| | - type: mrr_at_5 |
| | value: 58.087 |
| | - type: ndcg_at_1 |
| | value: 48.15 |
| | - type: ndcg_at_10 |
| | value: 61.709 |
| | - type: ndcg_at_100 |
| | value: 65.446 |
| | - type: ndcg_at_1000 |
| | value: 66.388 |
| | - type: ndcg_at_3 |
| | value: 56.333 |
| | - type: ndcg_at_5 |
| | value: 59.028000000000006 |
| | - type: precision_at_1 |
| | value: 48.15 |
| | - type: precision_at_10 |
| | value: 9.893 |
| | - type: precision_at_100 |
| | value: 1.265 |
| | - type: precision_at_1000 |
| | value: 0.13799999999999998 |
| | - type: precision_at_3 |
| | value: 25.266 |
| | - type: precision_at_5 |
| | value: 17.204 |
| | - type: recall_at_1 |
| | value: 42.047000000000004 |
| | - type: recall_at_10 |
| | value: 76.004 |
| | - type: recall_at_100 |
| | value: 91.727 |
| | - type: recall_at_1000 |
| | value: 98.213 |
| | - type: recall_at_3 |
| | value: 61.82 |
| | - type: recall_at_5 |
| | value: 68.42200000000001 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackGisRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 29.985 |
| | - type: map_at_10 |
| | value: 38.763999999999996 |
| | - type: map_at_100 |
| | value: 39.835 |
| | - type: map_at_1000 |
| | value: 39.900000000000006 |
| | - type: map_at_3 |
| | value: 35.826 |
| | - type: map_at_5 |
| | value: 37.403 |
| | - type: mrr_at_1 |
| | value: 32.202999999999996 |
| | - type: mrr_at_10 |
| | value: 40.94 |
| | - type: mrr_at_100 |
| | value: 41.861 |
| | - type: mrr_at_1000 |
| | value: 41.909 |
| | - type: mrr_at_3 |
| | value: 38.267 |
| | - type: mrr_at_5 |
| | value: 39.748 |
| | - type: ndcg_at_1 |
| | value: 32.202999999999996 |
| | - type: ndcg_at_10 |
| | value: 43.909 |
| | - type: ndcg_at_100 |
| | value: 49.028 |
| | - type: ndcg_at_1000 |
| | value: 50.714999999999996 |
| | - type: ndcg_at_3 |
| | value: 38.239000000000004 |
| | - type: ndcg_at_5 |
| | value: 40.854 |
| | - type: precision_at_1 |
| | value: 32.202999999999996 |
| | - type: precision_at_10 |
| | value: 6.621 |
| | - type: precision_at_100 |
| | value: 0.964 |
| | - type: precision_at_1000 |
| | value: 0.11399999999999999 |
| | - type: precision_at_3 |
| | value: 15.781999999999998 |
| | - type: precision_at_5 |
| | value: 10.96 |
| | - type: recall_at_1 |
| | value: 29.985 |
| | - type: recall_at_10 |
| | value: 57.727 |
| | - type: recall_at_100 |
| | value: 80.833 |
| | - type: recall_at_1000 |
| | value: 93.625 |
| | - type: recall_at_3 |
| | value: 42.396 |
| | - type: recall_at_5 |
| | value: 48.624 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackMathematicaRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 19.249 |
| | - type: map_at_10 |
| | value: 28.565 |
| | - type: map_at_100 |
| | value: 29.753 |
| | - type: map_at_1000 |
| | value: 29.881 |
| | - type: map_at_3 |
| | value: 25.778000000000002 |
| | - type: map_at_5 |
| | value: 27.21 |
| | - type: mrr_at_1 |
| | value: 23.632 |
| | - type: mrr_at_10 |
| | value: 33.51 |
| | - type: mrr_at_100 |
| | value: 34.372 |
| | - type: mrr_at_1000 |
| | value: 34.443 |
| | - type: mrr_at_3 |
| | value: 30.784 |
| | - type: mrr_at_5 |
| | value: 32.301 |
| | - type: ndcg_at_1 |
| | value: 23.632 |
| | - type: ndcg_at_10 |
| | value: 34.42 |
| | - type: ndcg_at_100 |
| | value: 39.823 |
| | - type: ndcg_at_1000 |
| | value: 42.558 |
| | - type: ndcg_at_3 |
| | value: 29.237000000000002 |
| | - type: ndcg_at_5 |
| | value: 31.465 |
| | - type: precision_at_1 |
| | value: 23.632 |
| | - type: precision_at_10 |
| | value: 6.331 |
| | - type: precision_at_100 |
| | value: 1.042 |
| | - type: precision_at_1000 |
| | value: 0.14100000000000001 |
| | - type: precision_at_3 |
| | value: 14.179 |
| | - type: precision_at_5 |
| | value: 10.299 |
| | - type: recall_at_1 |
| | value: 19.249 |
| | - type: recall_at_10 |
| | value: 47.539 |
| | - type: recall_at_100 |
| | value: 70.612 |
| | - type: recall_at_1000 |
| | value: 89.633 |
| | - type: recall_at_3 |
| | value: 33.082 |
| | - type: recall_at_5 |
| | value: 38.622 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackPhysicsRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 31.599 |
| | - type: map_at_10 |
| | value: 42.948 |
| | - type: map_at_100 |
| | value: 44.244 |
| | - type: map_at_1000 |
| | value: 44.352000000000004 |
| | - type: map_at_3 |
| | value: 39.352 |
| | - type: map_at_5 |
| | value: 41.397 |
| | - type: mrr_at_1 |
| | value: 38.21 |
| | - type: mrr_at_10 |
| | value: 48.347 |
| | - type: mrr_at_100 |
| | value: 49.132999999999996 |
| | - type: mrr_at_1000 |
| | value: 49.171 |
| | - type: mrr_at_3 |
| | value: 45.653 |
| | - type: mrr_at_5 |
| | value: 47.323 |
| | - type: ndcg_at_1 |
| | value: 38.21 |
| | - type: ndcg_at_10 |
| | value: 49.225 |
| | - type: ndcg_at_100 |
| | value: 54.422000000000004 |
| | - type: ndcg_at_1000 |
| | value: 56.27799999999999 |
| | - type: ndcg_at_3 |
| | value: 43.482 |
| | - type: ndcg_at_5 |
| | value: 46.321 |
| | - type: precision_at_1 |
| | value: 38.21 |
| | - type: precision_at_10 |
| | value: 8.921999999999999 |
| | - type: precision_at_100 |
| | value: 1.333 |
| | - type: precision_at_1000 |
| | value: 0.169 |
| | - type: precision_at_3 |
| | value: 20.372 |
| | - type: precision_at_5 |
| | value: 14.629 |
| | - type: recall_at_1 |
| | value: 31.599 |
| | - type: recall_at_10 |
| | value: 62.364 |
| | - type: recall_at_100 |
| | value: 83.91199999999999 |
| | - type: recall_at_1000 |
| | value: 95.743 |
| | - type: recall_at_3 |
| | value: 46.671 |
| | - type: recall_at_5 |
| | value: 53.772 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackProgrammersRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 26.641 |
| | - type: map_at_10 |
| | value: 37.604 |
| | - type: map_at_100 |
| | value: 38.897 |
| | - type: map_at_1000 |
| | value: 39.001000000000005 |
| | - type: map_at_3 |
| | value: 34.04 |
| | - type: map_at_5 |
| | value: 35.684 |
| | - type: mrr_at_1 |
| | value: 32.991 |
| | - type: mrr_at_10 |
| | value: 43.029 |
| | - type: mrr_at_100 |
| | value: 43.782 |
| | - type: mrr_at_1000 |
| | value: 43.830999999999996 |
| | - type: mrr_at_3 |
| | value: 40.164 |
| | - type: mrr_at_5 |
| | value: 41.619 |
| | - type: ndcg_at_1 |
| | value: 32.991 |
| | - type: ndcg_at_10 |
| | value: 44.217 |
| | - type: ndcg_at_100 |
| | value: 49.497 |
| | - type: ndcg_at_1000 |
| | value: 51.598 |
| | - type: ndcg_at_3 |
| | value: 38.208999999999996 |
| | - type: ndcg_at_5 |
| | value: 40.444 |
| | - type: precision_at_1 |
| | value: 32.991 |
| | - type: precision_at_10 |
| | value: 8.436 |
| | - type: precision_at_100 |
| | value: 1.279 |
| | - type: precision_at_1000 |
| | value: 0.163 |
| | - type: precision_at_3 |
| | value: 18.379 |
| | - type: precision_at_5 |
| | value: 13.196 |
| | - type: recall_at_1 |
| | value: 26.641 |
| | - type: recall_at_10 |
| | value: 58.50300000000001 |
| | - type: recall_at_100 |
| | value: 81.228 |
| | - type: recall_at_1000 |
| | value: 95.345 |
| | - type: recall_at_3 |
| | value: 41.6 |
| | - type: recall_at_5 |
| | value: 47.425 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 28.678083333333333 |
| | - type: map_at_10 |
| | value: 38.63366666666666 |
| | - type: map_at_100 |
| | value: 39.84708333333333 |
| | - type: map_at_1000 |
| | value: 39.959583333333335 |
| | - type: map_at_3 |
| | value: 35.49 |
| | - type: map_at_5 |
| | value: 37.23125 |
| | - type: mrr_at_1 |
| | value: 33.813916666666664 |
| | - type: mrr_at_10 |
| | value: 42.955500000000015 |
| | - type: mrr_at_100 |
| | value: 43.75541666666667 |
| | - type: mrr_at_1000 |
| | value: 43.80616666666666 |
| | - type: mrr_at_3 |
| | value: 40.40191666666667 |
| | - type: mrr_at_5 |
| | value: 41.88358333333333 |
| | - type: ndcg_at_1 |
| | value: 33.813916666666664 |
| | - type: ndcg_at_10 |
| | value: 44.361666666666665 |
| | - type: ndcg_at_100 |
| | value: 49.37991666666667 |
| | - type: ndcg_at_1000 |
| | value: 51.432583333333326 |
| | - type: ndcg_at_3 |
| | value: 39.12949999999999 |
| | - type: ndcg_at_5 |
| | value: 41.60183333333333 |
| | - type: precision_at_1 |
| | value: 33.813916666666664 |
| | - type: precision_at_10 |
| | value: 7.759250000000002 |
| | - type: precision_at_100 |
| | value: 1.2108333333333332 |
| | - type: precision_at_1000 |
| | value: 0.158 |
| | - type: precision_at_3 |
| | value: 17.90716666666667 |
| | - type: precision_at_5 |
| | value: 12.765333333333334 |
| | - type: recall_at_1 |
| | value: 28.678083333333333 |
| | - type: recall_at_10 |
| | value: 56.92716666666667 |
| | - type: recall_at_100 |
| | value: 78.74991666666668 |
| | - type: recall_at_1000 |
| | value: 92.73875000000001 |
| | - type: recall_at_3 |
| | value: 42.459916666666665 |
| | - type: recall_at_5 |
| | value: 48.76258333333333 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackStatsRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 27.282 |
| | - type: map_at_10 |
| | value: 34.458 |
| | - type: map_at_100 |
| | value: 35.44 |
| | - type: map_at_1000 |
| | value: 35.536 |
| | - type: map_at_3 |
| | value: 31.912000000000003 |
| | - type: map_at_5 |
| | value: 33.495000000000005 |
| | - type: mrr_at_1 |
| | value: 30.675 |
| | - type: mrr_at_10 |
| | value: 37.563 |
| | - type: mrr_at_100 |
| | value: 38.374 |
| | - type: mrr_at_1000 |
| | value: 38.444 |
| | - type: mrr_at_3 |
| | value: 35.276 |
| | - type: mrr_at_5 |
| | value: 36.718 |
| | - type: ndcg_at_1 |
| | value: 30.675 |
| | - type: ndcg_at_10 |
| | value: 38.838 |
| | - type: ndcg_at_100 |
| | value: 43.527 |
| | - type: ndcg_at_1000 |
| | value: 45.891 |
| | - type: ndcg_at_3 |
| | value: 34.314 |
| | - type: ndcg_at_5 |
| | value: 36.789 |
| | - type: precision_at_1 |
| | value: 30.675 |
| | - type: precision_at_10 |
| | value: 6.012 |
| | - type: precision_at_100 |
| | value: 0.903 |
| | - type: precision_at_1000 |
| | value: 0.117 |
| | - type: precision_at_3 |
| | value: 14.571000000000002 |
| | - type: precision_at_5 |
| | value: 10.306999999999999 |
| | - type: recall_at_1 |
| | value: 27.282 |
| | - type: recall_at_10 |
| | value: 49.198 |
| | - type: recall_at_100 |
| | value: 70.489 |
| | - type: recall_at_1000 |
| | value: 87.902 |
| | - type: recall_at_3 |
| | value: 36.966 |
| | - type: recall_at_5 |
| | value: 43.079 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackTexRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 19.839000000000002 |
| | - type: map_at_10 |
| | value: 27.68 |
| | - type: map_at_100 |
| | value: 28.851 |
| | - type: map_at_1000 |
| | value: 28.977999999999998 |
| | - type: map_at_3 |
| | value: 25.062 |
| | - type: map_at_5 |
| | value: 26.389000000000003 |
| | - type: mrr_at_1 |
| | value: 23.813000000000002 |
| | - type: mrr_at_10 |
| | value: 31.628 |
| | - type: mrr_at_100 |
| | value: 32.58 |
| | - type: mrr_at_1000 |
| | value: 32.655 |
| | - type: mrr_at_3 |
| | value: 29.29 |
| | - type: mrr_at_5 |
| | value: 30.551000000000002 |
| | - type: ndcg_at_1 |
| | value: 23.813000000000002 |
| | - type: ndcg_at_10 |
| | value: 32.751000000000005 |
| | - type: ndcg_at_100 |
| | value: 38.218 |
| | - type: ndcg_at_1000 |
| | value: 40.979 |
| | - type: ndcg_at_3 |
| | value: 28.043000000000003 |
| | - type: ndcg_at_5 |
| | value: 30.043 |
| | - type: precision_at_1 |
| | value: 23.813000000000002 |
| | - type: precision_at_10 |
| | value: 5.936 |
| | - type: precision_at_100 |
| | value: 1.016 |
| | - type: precision_at_1000 |
| | value: 0.14400000000000002 |
| | - type: precision_at_3 |
| | value: 13.145000000000001 |
| | - type: precision_at_5 |
| | value: 9.443 |
| | - type: recall_at_1 |
| | value: 19.839000000000002 |
| | - type: recall_at_10 |
| | value: 44.072 |
| | - type: recall_at_100 |
| | value: 68.406 |
| | - type: recall_at_1000 |
| | value: 87.749 |
| | - type: recall_at_3 |
| | value: 30.906 |
| | - type: recall_at_5 |
| | value: 36.081 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackUnixRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 28.195999999999998 |
| | - type: map_at_10 |
| | value: 38.345 |
| | - type: map_at_100 |
| | value: 39.561 |
| | - type: map_at_1000 |
| | value: 39.65 |
| | - type: map_at_3 |
| | value: 35.382999999999996 |
| | - type: map_at_5 |
| | value: 37.023 |
| | - type: mrr_at_1 |
| | value: 33.022 |
| | - type: mrr_at_10 |
| | value: 42.504 |
| | - type: mrr_at_100 |
| | value: 43.376 |
| | - type: mrr_at_1000 |
| | value: 43.427 |
| | - type: mrr_at_3 |
| | value: 40.050000000000004 |
| | - type: mrr_at_5 |
| | value: 41.421 |
| | - type: ndcg_at_1 |
| | value: 33.022 |
| | - type: ndcg_at_10 |
| | value: 43.997 |
| | - type: ndcg_at_100 |
| | value: 49.370000000000005 |
| | - type: ndcg_at_1000 |
| | value: 51.38399999999999 |
| | - type: ndcg_at_3 |
| | value: 38.802 |
| | - type: ndcg_at_5 |
| | value: 41.209 |
| | - type: precision_at_1 |
| | value: 33.022 |
| | - type: precision_at_10 |
| | value: 7.351000000000001 |
| | - type: precision_at_100 |
| | value: 1.1440000000000001 |
| | - type: precision_at_1000 |
| | value: 0.14200000000000002 |
| | - type: precision_at_3 |
| | value: 17.724 |
| | - type: precision_at_5 |
| | value: 12.443999999999999 |
| | - type: recall_at_1 |
| | value: 28.195999999999998 |
| | - type: recall_at_10 |
| | value: 57.011 |
| | - type: recall_at_100 |
| | value: 79.922 |
| | - type: recall_at_1000 |
| | value: 93.952 |
| | - type: recall_at_3 |
| | value: 42.857 |
| | - type: recall_at_5 |
| | value: 48.916 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackWebmastersRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 25.768 |
| | - type: map_at_10 |
| | value: 35.118 |
| | - type: map_at_100 |
| | value: 36.817 |
| | - type: map_at_1000 |
| | value: 37.037 |
| | - type: map_at_3 |
| | value: 31.997999999999998 |
| | - type: map_at_5 |
| | value: 33.697 |
| | - type: mrr_at_1 |
| | value: 31.621 |
| | - type: mrr_at_10 |
| | value: 40.228 |
| | - type: mrr_at_100 |
| | value: 41.239 |
| | - type: mrr_at_1000 |
| | value: 41.277 |
| | - type: mrr_at_3 |
| | value: 37.614999999999995 |
| | - type: mrr_at_5 |
| | value: 39.058 |
| | - type: ndcg_at_1 |
| | value: 31.621 |
| | - type: ndcg_at_10 |
| | value: 41.347 |
| | - type: ndcg_at_100 |
| | value: 47.620000000000005 |
| | - type: ndcg_at_1000 |
| | value: 49.759 |
| | - type: ndcg_at_3 |
| | value: 36.361 |
| | - type: ndcg_at_5 |
| | value: 38.635000000000005 |
| | - type: precision_at_1 |
| | value: 31.621 |
| | - type: precision_at_10 |
| | value: 8.024000000000001 |
| | - type: precision_at_100 |
| | value: 1.595 |
| | - type: precision_at_1000 |
| | value: 0.244 |
| | - type: precision_at_3 |
| | value: 16.996 |
| | - type: precision_at_5 |
| | value: 12.372 |
| | - type: recall_at_1 |
| | value: 25.768 |
| | - type: recall_at_10 |
| | value: 53.02 |
| | - type: recall_at_100 |
| | value: 81.329 |
| | - type: recall_at_1000 |
| | value: 94.025 |
| | - type: recall_at_3 |
| | value: 38.884 |
| | - type: recall_at_5 |
| | value: 45.057 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: BeIR/cqadupstack |
| | name: MTEB CQADupstackWordpressRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 24.627 |
| | - type: map_at_10 |
| | value: 33.106 |
| | - type: map_at_100 |
| | value: 33.936 |
| | - type: map_at_1000 |
| | value: 34.044999999999995 |
| | - type: map_at_3 |
| | value: 30.162 |
| | - type: map_at_5 |
| | value: 31.979999999999997 |
| | - type: mrr_at_1 |
| | value: 26.617 |
| | - type: mrr_at_10 |
| | value: 35.177 |
| | - type: mrr_at_100 |
| | value: 35.937999999999995 |
| | - type: mrr_at_1000 |
| | value: 36.008 |
| | - type: mrr_at_3 |
| | value: 32.562999999999995 |
| | - type: mrr_at_5 |
| | value: 34.208 |
| | - type: ndcg_at_1 |
| | value: 26.617 |
| | - type: ndcg_at_10 |
| | value: 38.082 |
| | - type: ndcg_at_100 |
| | value: 42.386 |
| | - type: ndcg_at_1000 |
| | value: 44.861000000000004 |
| | - type: ndcg_at_3 |
| | value: 32.557 |
| | - type: ndcg_at_5 |
| | value: 35.603 |
| | - type: precision_at_1 |
| | value: 26.617 |
| | - type: precision_at_10 |
| | value: 5.952 |
| | - type: precision_at_100 |
| | value: 0.874 |
| | - type: precision_at_1000 |
| | value: 0.121 |
| | - type: precision_at_3 |
| | value: 13.617 |
| | - type: precision_at_5 |
| | value: 9.945 |
| | - type: recall_at_1 |
| | value: 24.627 |
| | - type: recall_at_10 |
| | value: 51.317 |
| | - type: recall_at_100 |
| | value: 71.243 |
| | - type: recall_at_1000 |
| | value: 89.39399999999999 |
| | - type: recall_at_3 |
| | value: 36.778 |
| | - type: recall_at_5 |
| | value: 44.116 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: climate-fever |
| | name: MTEB ClimateFEVER |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 16.631 |
| | - type: map_at_10 |
| | value: 28.069 |
| | - type: map_at_100 |
| | value: 30.130000000000003 |
| | - type: map_at_1000 |
| | value: 30.318 |
| | - type: map_at_3 |
| | value: 23.430999999999997 |
| | - type: map_at_5 |
| | value: 25.929000000000002 |
| | - type: mrr_at_1 |
| | value: 37.264 |
| | - type: mrr_at_10 |
| | value: 49.608999999999995 |
| | - type: mrr_at_100 |
| | value: 50.349 |
| | - type: mrr_at_1000 |
| | value: 50.373000000000005 |
| | - type: mrr_at_3 |
| | value: 46.515 |
| | - type: mrr_at_5 |
| | value: 48.41 |
| | - type: ndcg_at_1 |
| | value: 37.264 |
| | - type: ndcg_at_10 |
| | value: 37.688 |
| | - type: ndcg_at_100 |
| | value: 45.101 |
| | - type: ndcg_at_1000 |
| | value: 48.19 |
| | - type: ndcg_at_3 |
| | value: 31.471 |
| | - type: ndcg_at_5 |
| | value: 33.719 |
| | - type: precision_at_1 |
| | value: 37.264 |
| | - type: precision_at_10 |
| | value: 11.616 |
| | - type: precision_at_100 |
| | value: 1.9619999999999997 |
| | - type: precision_at_1000 |
| | value: 0.255 |
| | - type: precision_at_3 |
| | value: 23.214000000000002 |
| | - type: precision_at_5 |
| | value: 17.824 |
| | - type: recall_at_1 |
| | value: 16.631 |
| | - type: recall_at_10 |
| | value: 43.516 |
| | - type: recall_at_100 |
| | value: 68.681 |
| | - type: recall_at_1000 |
| | value: 85.751 |
| | - type: recall_at_3 |
| | value: 28.199 |
| | - type: recall_at_5 |
| | value: 34.826 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: dbpedia-entity |
| | name: MTEB DBPedia |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 9.971 |
| | - type: map_at_10 |
| | value: 22.274 |
| | - type: map_at_100 |
| | value: 32.61 |
| | - type: map_at_1000 |
| | value: 34.422000000000004 |
| | - type: map_at_3 |
| | value: 15.473999999999998 |
| | - type: map_at_5 |
| | value: 18.412 |
| | - type: mrr_at_1 |
| | value: 72.25 |
| | - type: mrr_at_10 |
| | value: 79.945 |
| | - type: mrr_at_100 |
| | value: 80.192 |
| | - type: mrr_at_1000 |
| | value: 80.199 |
| | - type: mrr_at_3 |
| | value: 78.667 |
| | - type: mrr_at_5 |
| | value: 79.49199999999999 |
| | - type: ndcg_at_1 |
| | value: 59.75 |
| | - type: ndcg_at_10 |
| | value: 45.689 |
| | - type: ndcg_at_100 |
| | value: 51.687000000000005 |
| | - type: ndcg_at_1000 |
| | value: 58.904999999999994 |
| | - type: ndcg_at_3 |
| | value: 49.675999999999995 |
| | - type: ndcg_at_5 |
| | value: 47.419 |
| | - type: precision_at_1 |
| | value: 72.25 |
| | - type: precision_at_10 |
| | value: 37.05 |
| | - type: precision_at_100 |
| | value: 12.183 |
| | - type: precision_at_1000 |
| | value: 2.2929999999999997 |
| | - type: precision_at_3 |
| | value: 53.417 |
| | - type: precision_at_5 |
| | value: 46.150000000000006 |
| | - type: recall_at_1 |
| | value: 9.971 |
| | - type: recall_at_10 |
| | value: 27.932000000000002 |
| | - type: recall_at_100 |
| | value: 58.85399999999999 |
| | - type: recall_at_1000 |
| | value: 81.728 |
| | - type: recall_at_3 |
| | value: 16.619999999999997 |
| | - type: recall_at_5 |
| | value: 21.082 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/emotion |
| | name: MTEB EmotionClassification |
| | config: default |
| | split: test |
| | revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
| | metrics: |
| | - type: accuracy |
| | value: 52.83499999999999 |
| | - type: f1 |
| | value: 47.754076079187044 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: fever |
| | name: MTEB FEVER |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 79.783 |
| | - type: map_at_10 |
| | value: 87.224 |
| | - type: map_at_100 |
| | value: 87.401 |
| | - type: map_at_1000 |
| | value: 87.413 |
| | - type: map_at_3 |
| | value: 86.29 |
| | - type: map_at_5 |
| | value: 86.896 |
| | - type: mrr_at_1 |
| | value: 86.09400000000001 |
| | - type: mrr_at_10 |
| | value: 91.789 |
| | - type: mrr_at_100 |
| | value: 91.814 |
| | - type: mrr_at_1000 |
| | value: 91.815 |
| | - type: mrr_at_3 |
| | value: 91.39399999999999 |
| | - type: mrr_at_5 |
| | value: 91.684 |
| | - type: ndcg_at_1 |
| | value: 86.09400000000001 |
| | - type: ndcg_at_10 |
| | value: 90.36999999999999 |
| | - type: ndcg_at_100 |
| | value: 90.95299999999999 |
| | - type: ndcg_at_1000 |
| | value: 91.13799999999999 |
| | - type: ndcg_at_3 |
| | value: 89.13799999999999 |
| | - type: ndcg_at_5 |
| | value: 89.845 |
| | - type: precision_at_1 |
| | value: 86.09400000000001 |
| | - type: precision_at_10 |
| | value: 10.671 |
| | - type: precision_at_100 |
| | value: 1.123 |
| | - type: precision_at_1000 |
| | value: 0.11499999999999999 |
| | - type: precision_at_3 |
| | value: 33.698 |
| | - type: precision_at_5 |
| | value: 20.788999999999998 |
| | - type: recall_at_1 |
| | value: 79.783 |
| | - type: recall_at_10 |
| | value: 95.50999999999999 |
| | - type: recall_at_100 |
| | value: 97.68900000000001 |
| | - type: recall_at_1000 |
| | value: 98.79400000000001 |
| | - type: recall_at_3 |
| | value: 92.14099999999999 |
| | - type: recall_at_5 |
| | value: 94.0 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: fiqa |
| | name: MTEB FiQA2018 |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 23.526 |
| | - type: map_at_10 |
| | value: 38.135999999999996 |
| | - type: map_at_100 |
| | value: 40.221000000000004 |
| | - type: map_at_1000 |
| | value: 40.394000000000005 |
| | - type: map_at_3 |
| | value: 33.548 |
| | - type: map_at_5 |
| | value: 35.975 |
| | - type: mrr_at_1 |
| | value: 47.068 |
| | - type: mrr_at_10 |
| | value: 55.224 |
| | - type: mrr_at_100 |
| | value: 56.038 |
| | - type: mrr_at_1000 |
| | value: 56.066 |
| | - type: mrr_at_3 |
| | value: 53.00899999999999 |
| | - type: mrr_at_5 |
| | value: 54.306 |
| | - type: ndcg_at_1 |
| | value: 47.068 |
| | - type: ndcg_at_10 |
| | value: 46.399 |
| | - type: ndcg_at_100 |
| | value: 53.312000000000005 |
| | - type: ndcg_at_1000 |
| | value: 55.946 |
| | - type: ndcg_at_3 |
| | value: 42.954 |
| | - type: ndcg_at_5 |
| | value: 43.765 |
| | - type: precision_at_1 |
| | value: 47.068 |
| | - type: precision_at_10 |
| | value: 12.824 |
| | - type: precision_at_100 |
| | value: 1.986 |
| | - type: precision_at_1000 |
| | value: 0.246 |
| | - type: precision_at_3 |
| | value: 28.807 |
| | - type: precision_at_5 |
| | value: 20.772 |
| | - type: recall_at_1 |
| | value: 23.526 |
| | - type: recall_at_10 |
| | value: 53.242999999999995 |
| | - type: recall_at_100 |
| | value: 78.309 |
| | - type: recall_at_1000 |
| | value: 93.92099999999999 |
| | - type: recall_at_3 |
| | value: 38.716 |
| | - type: recall_at_5 |
| | value: 44.921 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: hotpotqa |
| | name: MTEB HotpotQA |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 41.641 |
| | - type: map_at_10 |
| | value: 67.24 |
| | - type: map_at_100 |
| | value: 68.108 |
| | - type: map_at_1000 |
| | value: 68.157 |
| | - type: map_at_3 |
| | value: 63.834999999999994 |
| | - type: map_at_5 |
| | value: 65.995 |
| | - type: mrr_at_1 |
| | value: 83.282 |
| | - type: mrr_at_10 |
| | value: 88.22 |
| | - type: mrr_at_100 |
| | value: 88.35499999999999 |
| | - type: mrr_at_1000 |
| | value: 88.358 |
| | - type: mrr_at_3 |
| | value: 87.571 |
| | - type: mrr_at_5 |
| | value: 88.01299999999999 |
| | - type: ndcg_at_1 |
| | value: 83.282 |
| | - type: ndcg_at_10 |
| | value: 75.066 |
| | - type: ndcg_at_100 |
| | value: 77.952 |
| | - type: ndcg_at_1000 |
| | value: 78.878 |
| | - type: ndcg_at_3 |
| | value: 70.482 |
| | - type: ndcg_at_5 |
| | value: 73.098 |
| | - type: precision_at_1 |
| | value: 83.282 |
| | - type: precision_at_10 |
| | value: 15.608 |
| | - type: precision_at_100 |
| | value: 1.7840000000000003 |
| | - type: precision_at_1000 |
| | value: 0.191 |
| | - type: precision_at_3 |
| | value: 45.324999999999996 |
| | - type: precision_at_5 |
| | value: 29.256 |
| | - type: recall_at_1 |
| | value: 41.641 |
| | - type: recall_at_10 |
| | value: 78.042 |
| | - type: recall_at_100 |
| | value: 89.223 |
| | - type: recall_at_1000 |
| | value: 95.341 |
| | - type: recall_at_3 |
| | value: 67.988 |
| | - type: recall_at_5 |
| | value: 73.14 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/imdb |
| | name: MTEB ImdbClassification |
| | config: default |
| | split: test |
| | revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
| | metrics: |
| | - type: accuracy |
| | value: 93.50520000000002 |
| | - type: ap |
| | value: 90.36560251927821 |
| | - type: f1 |
| | value: 93.50064413170799 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: msmarco |
| | name: MTEB MSMARCO |
| | config: default |
| | split: dev |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 23.355999999999998 |
| | - type: map_at_10 |
| | value: 36.082 |
| | - type: map_at_100 |
| | value: 37.239 |
| | - type: map_at_1000 |
| | value: 37.285000000000004 |
| | - type: map_at_3 |
| | value: 32.16 |
| | - type: map_at_5 |
| | value: 34.469 |
| | - type: mrr_at_1 |
| | value: 23.968 |
| | - type: mrr_at_10 |
| | value: 36.708 |
| | - type: mrr_at_100 |
| | value: 37.795 |
| | - type: mrr_at_1000 |
| | value: 37.836 |
| | - type: mrr_at_3 |
| | value: 32.865 |
| | - type: mrr_at_5 |
| | value: 35.154 |
| | - type: ndcg_at_1 |
| | value: 23.968 |
| | - type: ndcg_at_10 |
| | value: 43.152 |
| | - type: ndcg_at_100 |
| | value: 48.615 |
| | - type: ndcg_at_1000 |
| | value: 49.714000000000006 |
| | - type: ndcg_at_3 |
| | value: 35.208 |
| | - type: ndcg_at_5 |
| | value: 39.342 |
| | - type: precision_at_1 |
| | value: 23.968 |
| | - type: precision_at_10 |
| | value: 6.784 |
| | - type: precision_at_100 |
| | value: 0.951 |
| | - type: precision_at_1000 |
| | value: 0.104 |
| | - type: precision_at_3 |
| | value: 14.995 |
| | - type: precision_at_5 |
| | value: 11.092 |
| | - type: recall_at_1 |
| | value: 23.355999999999998 |
| | - type: recall_at_10 |
| | value: 64.828 |
| | - type: recall_at_100 |
| | value: 89.888 |
| | - type: recall_at_1000 |
| | value: 98.181 |
| | - type: recall_at_3 |
| | value: 43.336000000000006 |
| | - type: recall_at_5 |
| | value: 53.274 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_domain |
| | name: MTEB MTOPDomainClassification (en) |
| | config: en |
| | split: test |
| | revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
| | metrics: |
| | - type: accuracy |
| | value: 94.97948016415869 |
| | - type: f1 |
| | value: 94.77285510790911 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/mtop_intent |
| | name: MTEB MTOPIntentClassification (en) |
| | config: en |
| | split: test |
| | revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
| | metrics: |
| | - type: accuracy |
| | value: 78.49749202006383 |
| | - type: f1 |
| | value: 59.36772995632707 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_intent |
| | name: MTEB MassiveIntentClassification (en) |
| | config: en |
| | split: test |
| | revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
| | metrics: |
| | - type: accuracy |
| | value: 77.64290517821117 |
| | - type: f1 |
| | value: 75.33296771580456 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/amazon_massive_scenario |
| | name: MTEB MassiveScenarioClassification (en) |
| | config: en |
| | split: test |
| | revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
| | metrics: |
| | - type: accuracy |
| | value: 80.76664425016811 |
| | - type: f1 |
| | value: 80.79147962348141 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/medrxiv-clustering-p2p |
| | name: MTEB MedrxivClusteringP2P |
| | config: default |
| | split: test |
| | revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
| | metrics: |
| | - type: v_measure |
| | value: 35.158637354708034 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/medrxiv-clustering-s2s |
| | name: MTEB MedrxivClusteringS2S |
| | config: default |
| | split: test |
| | revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
| | metrics: |
| | - type: v_measure |
| | value: 32.39319499403552 |
| | - task: |
| | type: Reranking |
| | dataset: |
| | type: mteb/mind_small |
| | name: MTEB MindSmallReranking |
| | config: default |
| | split: test |
| | revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
| | metrics: |
| | - type: map |
| | value: 32.19460802526735 |
| | - type: mrr |
| | value: 33.39458959690712 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: nfcorpus |
| | name: MTEB NFCorpus |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 7.225 |
| | - type: map_at_10 |
| | value: 15.609 |
| | - type: map_at_100 |
| | value: 20.067 |
| | - type: map_at_1000 |
| | value: 21.709999999999997 |
| | - type: map_at_3 |
| | value: 11.518 |
| | - type: map_at_5 |
| | value: 13.469999999999999 |
| | - type: mrr_at_1 |
| | value: 50.15500000000001 |
| | - type: mrr_at_10 |
| | value: 58.711 |
| | - type: mrr_at_100 |
| | value: 59.333000000000006 |
| | - type: mrr_at_1000 |
| | value: 59.362 |
| | - type: mrr_at_3 |
| | value: 56.65599999999999 |
| | - type: mrr_at_5 |
| | value: 57.972 |
| | - type: ndcg_at_1 |
| | value: 48.452 |
| | - type: ndcg_at_10 |
| | value: 38.845 |
| | - type: ndcg_at_100 |
| | value: 36.597 |
| | - type: ndcg_at_1000 |
| | value: 45.472 |
| | - type: ndcg_at_3 |
| | value: 43.947 |
| | - type: ndcg_at_5 |
| | value: 42.097 |
| | - type: precision_at_1 |
| | value: 49.845 |
| | - type: precision_at_10 |
| | value: 28.638 |
| | - type: precision_at_100 |
| | value: 9.229 |
| | - type: precision_at_1000 |
| | value: 2.234 |
| | - type: precision_at_3 |
| | value: 40.867 |
| | - type: precision_at_5 |
| | value: 36.285000000000004 |
| | - type: recall_at_1 |
| | value: 7.225 |
| | - type: recall_at_10 |
| | value: 19.272 |
| | - type: recall_at_100 |
| | value: 37.299 |
| | - type: recall_at_1000 |
| | value: 68.757 |
| | - type: recall_at_3 |
| | value: 12.350999999999999 |
| | - type: recall_at_5 |
| | value: 15.369 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: nq |
| | name: MTEB NQ |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 34.453 |
| | - type: map_at_10 |
| | value: 50.748000000000005 |
| | - type: map_at_100 |
| | value: 51.666000000000004 |
| | - type: map_at_1000 |
| | value: 51.687000000000005 |
| | - type: map_at_3 |
| | value: 46.300000000000004 |
| | - type: map_at_5 |
| | value: 49.032 |
| | - type: mrr_at_1 |
| | value: 38.673 |
| | - type: mrr_at_10 |
| | value: 53.11 |
| | - type: mrr_at_100 |
| | value: 53.772 |
| | - type: mrr_at_1000 |
| | value: 53.784 |
| | - type: mrr_at_3 |
| | value: 49.483 |
| | - type: mrr_at_5 |
| | value: 51.751999999999995 |
| | - type: ndcg_at_1 |
| | value: 38.673 |
| | - type: ndcg_at_10 |
| | value: 58.60300000000001 |
| | - type: ndcg_at_100 |
| | value: 62.302 |
| | - type: ndcg_at_1000 |
| | value: 62.763999999999996 |
| | - type: ndcg_at_3 |
| | value: 50.366 |
| | - type: ndcg_at_5 |
| | value: 54.888999999999996 |
| | - type: precision_at_1 |
| | value: 38.673 |
| | - type: precision_at_10 |
| | value: 9.522 |
| | - type: precision_at_100 |
| | value: 1.162 |
| | - type: precision_at_1000 |
| | value: 0.121 |
| | - type: precision_at_3 |
| | value: 22.779 |
| | - type: precision_at_5 |
| | value: 16.256999999999998 |
| | - type: recall_at_1 |
| | value: 34.453 |
| | - type: recall_at_10 |
| | value: 80.074 |
| | - type: recall_at_100 |
| | value: 95.749 |
| | - type: recall_at_1000 |
| | value: 99.165 |
| | - type: recall_at_3 |
| | value: 58.897999999999996 |
| | - type: recall_at_5 |
| | value: 69.349 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: quora |
| | name: MTEB QuoraRetrieval |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 71.80499999999999 |
| | - type: map_at_10 |
| | value: 85.773 |
| | - type: map_at_100 |
| | value: 86.4 |
| | - type: map_at_1000 |
| | value: 86.414 |
| | - type: map_at_3 |
| | value: 82.919 |
| | - type: map_at_5 |
| | value: 84.70299999999999 |
| | - type: mrr_at_1 |
| | value: 82.69999999999999 |
| | - type: mrr_at_10 |
| | value: 88.592 |
| | - type: mrr_at_100 |
| | value: 88.682 |
| | - type: mrr_at_1000 |
| | value: 88.683 |
| | - type: mrr_at_3 |
| | value: 87.705 |
| | - type: mrr_at_5 |
| | value: 88.30799999999999 |
| | - type: ndcg_at_1 |
| | value: 82.69 |
| | - type: ndcg_at_10 |
| | value: 89.316 |
| | - type: ndcg_at_100 |
| | value: 90.45100000000001 |
| | - type: ndcg_at_1000 |
| | value: 90.525 |
| | - type: ndcg_at_3 |
| | value: 86.68 |
| | - type: ndcg_at_5 |
| | value: 88.113 |
| | - type: precision_at_1 |
| | value: 82.69 |
| | - type: precision_at_10 |
| | value: 13.507 |
| | - type: precision_at_100 |
| | value: 1.5350000000000001 |
| | - type: precision_at_1000 |
| | value: 0.157 |
| | - type: precision_at_3 |
| | value: 37.927 |
| | - type: precision_at_5 |
| | value: 24.823999999999998 |
| | - type: recall_at_1 |
| | value: 71.80499999999999 |
| | - type: recall_at_10 |
| | value: 95.965 |
| | - type: recall_at_100 |
| | value: 99.70400000000001 |
| | - type: recall_at_1000 |
| | value: 99.992 |
| | - type: recall_at_3 |
| | value: 88.268 |
| | - type: recall_at_5 |
| | value: 92.45 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/reddit-clustering |
| | name: MTEB RedditClustering |
| | config: default |
| | split: test |
| | revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
| | metrics: |
| | - type: v_measure |
| | value: 60.24178219867024 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/reddit-clustering-p2p |
| | name: MTEB RedditClusteringP2P |
| | config: default |
| | split: test |
| | revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
| | metrics: |
| | - type: v_measure |
| | value: 64.99552099515469 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: scidocs |
| | name: MTEB SCIDOCS |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 5.4879999999999995 |
| | - type: map_at_10 |
| | value: 14.774999999999999 |
| | - type: map_at_100 |
| | value: 17.285 |
| | - type: map_at_1000 |
| | value: 17.648 |
| | - type: map_at_3 |
| | value: 10.4 |
| | - type: map_at_5 |
| | value: 12.552 |
| | - type: mrr_at_1 |
| | value: 27.1 |
| | - type: mrr_at_10 |
| | value: 39.251000000000005 |
| | - type: mrr_at_100 |
| | value: 40.335 |
| | - type: mrr_at_1000 |
| | value: 40.367 |
| | - type: mrr_at_3 |
| | value: 35.683 |
| | - type: mrr_at_5 |
| | value: 37.733 |
| | - type: ndcg_at_1 |
| | value: 27.1 |
| | - type: ndcg_at_10 |
| | value: 23.974 |
| | - type: ndcg_at_100 |
| | value: 33.161 |
| | - type: ndcg_at_1000 |
| | value: 38.853 |
| | - type: ndcg_at_3 |
| | value: 22.695999999999998 |
| | - type: ndcg_at_5 |
| | value: 19.881 |
| | - type: precision_at_1 |
| | value: 27.1 |
| | - type: precision_at_10 |
| | value: 12.479999999999999 |
| | - type: precision_at_100 |
| | value: 2.571 |
| | - type: precision_at_1000 |
| | value: 0.393 |
| | - type: precision_at_3 |
| | value: 21.367 |
| | - type: precision_at_5 |
| | value: 17.560000000000002 |
| | - type: recall_at_1 |
| | value: 5.4879999999999995 |
| | - type: recall_at_10 |
| | value: 25.290000000000003 |
| | - type: recall_at_100 |
| | value: 52.222 |
| | - type: recall_at_1000 |
| | value: 79.77300000000001 |
| | - type: recall_at_3 |
| | value: 13.001999999999999 |
| | - type: recall_at_5 |
| | value: 17.812 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sickr-sts |
| | name: MTEB SICK-R |
| | config: default |
| | split: test |
| | revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 85.3407705934785 |
| | - type: cos_sim_spearman |
| | value: 81.28145766913589 |
| | - type: euclidean_pearson |
| | value: 82.69277819943873 |
| | - type: euclidean_spearman |
| | value: 81.26097565088551 |
| | - type: manhattan_pearson |
| | value: 82.73440374725746 |
| | - type: manhattan_spearman |
| | value: 81.25376873901254 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts12-sts |
| | name: MTEB STS12 |
| | config: default |
| | split: test |
| | revision: a0d554a64d88156834ff5ae9920b964011b16384 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 87.23415639286914 |
| | - type: cos_sim_spearman |
| | value: 79.80147936079915 |
| | - type: euclidean_pearson |
| | value: 84.324220218071 |
| | - type: euclidean_spearman |
| | value: 79.71794784987208 |
| | - type: manhattan_pearson |
| | value: 84.27523842345964 |
| | - type: manhattan_spearman |
| | value: 79.58070329781553 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts13-sts |
| | name: MTEB STS13 |
| | config: default |
| | split: test |
| | revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 85.90966234413125 |
| | - type: cos_sim_spearman |
| | value: 87.10742652814713 |
| | - type: euclidean_pearson |
| | value: 86.28297063322286 |
| | - type: euclidean_spearman |
| | value: 87.09425001932226 |
| | - type: manhattan_pearson |
| | value: 86.19204338411774 |
| | - type: manhattan_spearman |
| | value: 87.02046826723424 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts14-sts |
| | name: MTEB STS14 |
| | config: default |
| | split: test |
| | revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 84.12351399124411 |
| | - type: cos_sim_spearman |
| | value: 83.32955357808568 |
| | - type: euclidean_pearson |
| | value: 83.81222384305896 |
| | - type: euclidean_spearman |
| | value: 83.1836394454507 |
| | - type: manhattan_pearson |
| | value: 83.79162945392092 |
| | - type: manhattan_spearman |
| | value: 83.14306058903364 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts15-sts |
| | name: MTEB STS15 |
| | config: default |
| | split: test |
| | revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 86.9345194840047 |
| | - type: cos_sim_spearman |
| | value: 88.47286320653176 |
| | - type: euclidean_pearson |
| | value: 87.72825182191445 |
| | - type: euclidean_spearman |
| | value: 88.33484195475864 |
| | - type: manhattan_pearson |
| | value: 87.75121043906692 |
| | - type: manhattan_spearman |
| | value: 88.36695329548576 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts16-sts |
| | name: MTEB STS16 |
| | config: default |
| | split: test |
| | revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 84.80215370816441 |
| | - type: cos_sim_spearman |
| | value: 86.44917331470305 |
| | - type: euclidean_pearson |
| | value: 85.3458573021962 |
| | - type: euclidean_spearman |
| | value: 86.24853627058414 |
| | - type: manhattan_pearson |
| | value: 85.38477148579328 |
| | - type: manhattan_spearman |
| | value: 86.28201585857053 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts17-crosslingual-sts |
| | name: MTEB STS17 (en-en) |
| | config: en-en |
| | split: test |
| | revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 87.20498617189688 |
| | - type: cos_sim_spearman |
| | value: 87.61389142076317 |
| | - type: euclidean_pearson |
| | value: 88.15430699740293 |
| | - type: euclidean_spearman |
| | value: 87.35065666258774 |
| | - type: manhattan_pearson |
| | value: 88.2994571119992 |
| | - type: manhattan_spearman |
| | value: 87.60920178284005 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/sts22-crosslingual-sts |
| | name: MTEB STS22 (en) |
| | config: en |
| | split: test |
| | revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 68.27672577392406 |
| | - type: cos_sim_spearman |
| | value: 68.31250175566586 |
| | - type: euclidean_pearson |
| | value: 69.45016222616813 |
| | - type: euclidean_spearman |
| | value: 67.93461301528046 |
| | - type: manhattan_pearson |
| | value: 69.39774219739259 |
| | - type: manhattan_spearman |
| | value: 67.78124856615536 |
| | - task: |
| | type: STS |
| | dataset: |
| | type: mteb/stsbenchmark-sts |
| | name: MTEB STSBenchmark |
| | config: default |
| | split: test |
| | revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 85.31916148698113 |
| | - type: cos_sim_spearman |
| | value: 87.45541524487057 |
| | - type: euclidean_pearson |
| | value: 86.5845909408775 |
| | - type: euclidean_spearman |
| | value: 87.2373331768082 |
| | - type: manhattan_pearson |
| | value: 86.64467698948668 |
| | - type: manhattan_spearman |
| | value: 87.26707857525533 |
| | - task: |
| | type: Reranking |
| | dataset: |
| | type: mteb/scidocs-reranking |
| | name: MTEB SciDocsRR |
| | config: default |
| | split: test |
| | revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
| | metrics: |
| | - type: map |
| | value: 88.0007930269447 |
| | - type: mrr |
| | value: 96.52852594029063 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: scifact |
| | name: MTEB SciFact |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 60.99400000000001 |
| | - type: map_at_10 |
| | value: 70.923 |
| | - type: map_at_100 |
| | value: 71.299 |
| | - type: map_at_1000 |
| | value: 71.318 |
| | - type: map_at_3 |
| | value: 67.991 |
| | - type: map_at_5 |
| | value: 69.292 |
| | - type: mrr_at_1 |
| | value: 64.333 |
| | - type: mrr_at_10 |
| | value: 71.98400000000001 |
| | - type: mrr_at_100 |
| | value: 72.306 |
| | - type: mrr_at_1000 |
| | value: 72.32499999999999 |
| | - type: mrr_at_3 |
| | value: 69.833 |
| | - type: mrr_at_5 |
| | value: 70.783 |
| | - type: ndcg_at_1 |
| | value: 64.333 |
| | - type: ndcg_at_10 |
| | value: 75.729 |
| | - type: ndcg_at_100 |
| | value: 77.38199999999999 |
| | - type: ndcg_at_1000 |
| | value: 77.788 |
| | - type: ndcg_at_3 |
| | value: 70.774 |
| | - type: ndcg_at_5 |
| | value: 72.478 |
| | - type: precision_at_1 |
| | value: 64.333 |
| | - type: precision_at_10 |
| | value: 10.167 |
| | - type: precision_at_100 |
| | value: 1.0999999999999999 |
| | - type: precision_at_1000 |
| | value: 0.11299999999999999 |
| | - type: precision_at_3 |
| | value: 27.778000000000002 |
| | - type: precision_at_5 |
| | value: 17.867 |
| | - type: recall_at_1 |
| | value: 60.99400000000001 |
| | - type: recall_at_10 |
| | value: 89.48899999999999 |
| | - type: recall_at_100 |
| | value: 97.0 |
| | - type: recall_at_1000 |
| | value: 100.0 |
| | - type: recall_at_3 |
| | value: 75.85 |
| | - type: recall_at_5 |
| | value: 80.328 |
| | - task: |
| | type: PairClassification |
| | dataset: |
| | type: mteb/sprintduplicatequestions-pairclassification |
| | name: MTEB SprintDuplicateQuestions |
| | config: default |
| | split: test |
| | revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
| | metrics: |
| | - type: cos_sim_accuracy |
| | value: 99.86435643564356 |
| | - type: cos_sim_ap |
| | value: 96.78001342960285 |
| | - type: cos_sim_f1 |
| | value: 93.07030854830552 |
| | - type: cos_sim_precision |
| | value: 94.16581371545547 |
| | - type: cos_sim_recall |
| | value: 92.0 |
| | - type: dot_accuracy |
| | value: 99.74653465346535 |
| | - type: dot_ap |
| | value: 92.80391251199522 |
| | - type: dot_f1 |
| | value: 87.36426456071075 |
| | - type: dot_precision |
| | value: 86.25730994152046 |
| | - type: dot_recall |
| | value: 88.5 |
| | - type: euclidean_accuracy |
| | value: 99.86138613861387 |
| | - type: euclidean_ap |
| | value: 96.77007810699926 |
| | - type: euclidean_f1 |
| | value: 92.95065458207452 |
| | - type: euclidean_precision |
| | value: 93.6105476673428 |
| | - type: euclidean_recall |
| | value: 92.30000000000001 |
| | - type: manhattan_accuracy |
| | value: 99.86336633663366 |
| | - type: manhattan_ap |
| | value: 96.78913160708261 |
| | - type: manhattan_f1 |
| | value: 93.03030303030305 |
| | - type: manhattan_precision |
| | value: 93.9795918367347 |
| | - type: manhattan_recall |
| | value: 92.10000000000001 |
| | - type: max_accuracy |
| | value: 99.86435643564356 |
| | - type: max_ap |
| | value: 96.78913160708261 |
| | - type: max_f1 |
| | value: 93.07030854830552 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/stackexchange-clustering |
| | name: MTEB StackExchangeClustering |
| | config: default |
| | split: test |
| | revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
| | metrics: |
| | - type: v_measure |
| | value: 67.80798406371026 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/stackexchange-clustering-p2p |
| | name: MTEB StackExchangeClusteringP2P |
| | config: default |
| | split: test |
| | revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
| | metrics: |
| | - type: v_measure |
| | value: 35.69251193913337 |
| | - task: |
| | type: Reranking |
| | dataset: |
| | type: mteb/stackoverflowdupquestions-reranking |
| | name: MTEB StackOverflowDupQuestions |
| | config: default |
| | split: test |
| | revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
| | metrics: |
| | - type: map |
| | value: 55.04250964616215 |
| | - type: mrr |
| | value: 55.92283125371361 |
| | - task: |
| | type: Summarization |
| | dataset: |
| | type: mteb/summeval |
| | name: MTEB SummEval |
| | config: default |
| | split: test |
| | revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
| | metrics: |
| | - type: cos_sim_pearson |
| | value: 31.05492162235311 |
| | - type: cos_sim_spearman |
| | value: 30.90473006515039 |
| | - type: dot_pearson |
| | value: 26.85480454105073 |
| | - type: dot_spearman |
| | value: 27.02880537417923 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: trec-covid |
| | name: MTEB TRECCOVID |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 0.246 |
| | - type: map_at_10 |
| | value: 2.125 |
| | - type: map_at_100 |
| | value: 12.892999999999999 |
| | - type: map_at_1000 |
| | value: 31.513999999999996 |
| | - type: map_at_3 |
| | value: 0.695 |
| | - type: map_at_5 |
| | value: 1.133 |
| | - type: mrr_at_1 |
| | value: 92.0 |
| | - type: mrr_at_10 |
| | value: 95.667 |
| | - type: mrr_at_100 |
| | value: 95.667 |
| | - type: mrr_at_1000 |
| | value: 95.667 |
| | - type: mrr_at_3 |
| | value: 95.667 |
| | - type: mrr_at_5 |
| | value: 95.667 |
| | - type: ndcg_at_1 |
| | value: 88.0 |
| | - type: ndcg_at_10 |
| | value: 82.464 |
| | - type: ndcg_at_100 |
| | value: 63.351 |
| | - type: ndcg_at_1000 |
| | value: 57.129 |
| | - type: ndcg_at_3 |
| | value: 85.87700000000001 |
| | - type: ndcg_at_5 |
| | value: 86.042 |
| | - type: precision_at_1 |
| | value: 92.0 |
| | - type: precision_at_10 |
| | value: 86.2 |
| | - type: precision_at_100 |
| | value: 65.10000000000001 |
| | - type: precision_at_1000 |
| | value: 25.044 |
| | - type: precision_at_3 |
| | value: 89.333 |
| | - type: precision_at_5 |
| | value: 89.60000000000001 |
| | - type: recall_at_1 |
| | value: 0.246 |
| | - type: recall_at_10 |
| | value: 2.2880000000000003 |
| | - type: recall_at_100 |
| | value: 15.853 |
| | - type: recall_at_1000 |
| | value: 54.05 |
| | - type: recall_at_3 |
| | value: 0.72 |
| | - type: recall_at_5 |
| | value: 1.196 |
| | - task: |
| | type: Retrieval |
| | dataset: |
| | type: webis-touche2020 |
| | name: MTEB Touche2020 |
| | config: default |
| | split: test |
| | revision: None |
| | metrics: |
| | - type: map_at_1 |
| | value: 3.322 |
| | - type: map_at_10 |
| | value: 11.673 |
| | - type: map_at_100 |
| | value: 18.655 |
| | - type: map_at_1000 |
| | value: 20.058999999999997 |
| | - type: map_at_3 |
| | value: 6.265 |
| | - type: map_at_5 |
| | value: 8.549 |
| | - type: mrr_at_1 |
| | value: 42.857 |
| | - type: mrr_at_10 |
| | value: 55.352999999999994 |
| | - type: mrr_at_100 |
| | value: 55.928999999999995 |
| | - type: mrr_at_1000 |
| | value: 55.928999999999995 |
| | - type: mrr_at_3 |
| | value: 50.0 |
| | - type: mrr_at_5 |
| | value: 53.571000000000005 |
| | - type: ndcg_at_1 |
| | value: 39.796 |
| | - type: ndcg_at_10 |
| | value: 28.225 |
| | - type: ndcg_at_100 |
| | value: 40.452 |
| | - type: ndcg_at_1000 |
| | value: 51.332 |
| | - type: ndcg_at_3 |
| | value: 32.308 |
| | - type: ndcg_at_5 |
| | value: 30.942999999999998 |
| | - type: precision_at_1 |
| | value: 42.857 |
| | - type: precision_at_10 |
| | value: 24.490000000000002 |
| | - type: precision_at_100 |
| | value: 8.366999999999999 |
| | - type: precision_at_1000 |
| | value: 1.5709999999999997 |
| | - type: precision_at_3 |
| | value: 32.653 |
| | - type: precision_at_5 |
| | value: 30.203999999999997 |
| | - type: recall_at_1 |
| | value: 3.322 |
| | - type: recall_at_10 |
| | value: 17.857 |
| | - type: recall_at_100 |
| | value: 51.169 |
| | - type: recall_at_1000 |
| | value: 85.382 |
| | - type: recall_at_3 |
| | value: 7.126 |
| | - type: recall_at_5 |
| | value: 11.186 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/toxic_conversations_50k |
| | name: MTEB ToxicConversationsClassification |
| | config: default |
| | split: test |
| | revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
| | metrics: |
| | - type: accuracy |
| | value: 72.1046 |
| | - type: ap |
| | value: 14.84774372187047 |
| | - type: f1 |
| | value: 55.52709376912111 |
| | - task: |
| | type: Classification |
| | dataset: |
| | type: mteb/tweet_sentiment_extraction |
| | name: MTEB TweetSentimentExtractionClassification |
| | config: default |
| | split: test |
| | revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
| | metrics: |
| | - type: accuracy |
| | value: 60.18958687040181 |
| | - type: f1 |
| | value: 60.53154943862625 |
| | - task: |
| | type: Clustering |
| | dataset: |
| | type: mteb/twentynewsgroups-clustering |
| | name: MTEB TwentyNewsgroupsClustering |
| | config: default |
| | split: test |
| | revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
| | metrics: |
| | - type: v_measure |
| | value: 54.61440440799667 |
| | - task: |
| | type: PairClassification |
| | dataset: |
| | type: mteb/twittersemeval2015-pairclassification |
| | name: MTEB TwitterSemEval2015 |
| | config: default |
| | split: test |
| | revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
| | metrics: |
| | - type: cos_sim_accuracy |
| | value: 87.34577099600644 |
| | - type: cos_sim_ap |
| | value: 78.19613471607386 |
| | - type: cos_sim_f1 |
| | value: 71.30501144746884 |
| | - type: cos_sim_precision |
| | value: 68.83595284872298 |
| | - type: cos_sim_recall |
| | value: 73.95778364116094 |
| | - type: dot_accuracy |
| | value: 82.89324670680098 |
| | - type: dot_ap |
| | value: 63.02362697550343 |
| | - type: dot_f1 |
| | value: 59.69837587006961 |
| | - type: dot_precision |
| | value: 53.2712215320911 |
| | - type: dot_recall |
| | value: 67.8891820580475 |
| | - type: euclidean_accuracy |
| | value: 87.24444179531503 |
| | - type: euclidean_ap |
| | value: 78.38356749852895 |
| | - type: euclidean_f1 |
| | value: 71.42133265771471 |
| | - type: euclidean_precision |
| | value: 68.68908382066277 |
| | - type: euclidean_recall |
| | value: 74.37994722955145 |
| | - type: manhattan_accuracy |
| | value: 87.24444179531503 |
| | - type: manhattan_ap |
| | value: 78.27660966609476 |
| | - type: manhattan_f1 |
| | value: 71.42165173165415 |
| | - type: manhattan_precision |
| | value: 66.00268576544315 |
| | - type: manhattan_recall |
| | value: 77.81002638522428 |
| | - type: max_accuracy |
| | value: 87.34577099600644 |
| | - type: max_ap |
| | value: 78.38356749852895 |
| | - type: max_f1 |
| | value: 71.42165173165415 |
| | - task: |
| | type: PairClassification |
| | dataset: |
| | type: mteb/twitterurlcorpus-pairclassification |
| | name: MTEB TwitterURLCorpus |
| | config: default |
| | split: test |
| | revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
| | metrics: |
| | - type: cos_sim_accuracy |
| | value: 88.90829355377032 |
| | - type: cos_sim_ap |
| | value: 85.79696678631824 |
| | - type: cos_sim_f1 |
| | value: 77.8494623655914 |
| | - type: cos_sim_precision |
| | value: 76.32610786417105 |
| | - type: cos_sim_recall |
| | value: 79.43486295041576 |
| | - type: dot_accuracy |
| | value: 86.17223580548765 |
| | - type: dot_ap |
| | value: 79.05804163697516 |
| | - type: dot_f1 |
| | value: 72.38855622089154 |
| | - type: dot_precision |
| | value: 69.61467368121713 |
| | - type: dot_recall |
| | value: 75.39267015706807 |
| | - type: euclidean_accuracy |
| | value: 88.94128148406877 |
| | - type: euclidean_ap |
| | value: 85.86615739743813 |
| | - type: euclidean_f1 |
| | value: 77.97001153402537 |
| | - type: euclidean_precision |
| | value: 75.44099647202822 |
| | - type: euclidean_recall |
| | value: 80.67446874037573 |
| | - type: manhattan_accuracy |
| | value: 88.9781503473435 |
| | - type: manhattan_ap |
| | value: 85.91093266751166 |
| | - type: manhattan_f1 |
| | value: 77.96835723791216 |
| | - type: manhattan_precision |
| | value: 74.98577929465301 |
| | - type: manhattan_recall |
| | value: 81.19802894979982 |
| | - type: max_accuracy |
| | value: 88.9781503473435 |
| | - type: max_ap |
| | value: 85.91093266751166 |
| | - type: max_f1 |
| | value: 77.97001153402537 |
| | --- |
| | |
| | # Sionic AI Embedding API v2 |
| |
|
| | ## About Sionic AI |
| |
|
| | Sionic AI delivers more accessible and cost-effective AI technology addressing the various needs to boost productivity and drive innovation. |
| |
|
| | The Large Language Model (LLM) is not for research and experimentation. |
| | We offer solutions that leverage LLM to add value to your business. |
| | Anyone can easily train and control AI. |
| |
|
| | ## How to get embeddings |
| |
|
| | Currently, we open the beta version of embedding APIs. |
| | To get embeddings, you should call API endpoint to send your text. |
| | You can send either a single sentence or multiple sentences. |
| | The embeddings that correspond to the inputs will be returned. |
| |
|
| | API Endpoint : https://api.sionic.ai/v2/embedding |
| |
|
| | ### Command line Example |
| | Request: |
| | ```shell |
| | curl https://api.sionic.ai/v2/embedding \ |
| | -H "Content-Type: application/json" \ |
| | -d '{ |
| | "inputs": ["first query", "second query", "third query"] |
| | }' |
| | ``` |
| |
|
| | Response: |
| | ```shell |
| | { |
| | "embedding": [ |
| | [ |
| | 0.5567971, |
| | -1.1578958, |
| | -0.7148851, |
| | -0.2326297, |
| | 0.4394867, |
| | ... |
| | ], |
| | [ |
| | 0.5049863, |
| | -0.8253384, |
| | -1.0041373, |
| | -0.6503708, |
| | 0.5007141, |
| | ... |
| | ], |
| | [ |
| | 0.6059823, |
| | -1.0369557, |
| | -0.6705063, |
| | -0.4467056, |
| | 0.8618057, |
| | ... |
| | ] |
| | ] |
| | } |
| | ``` |
| |
|
| | ### Python code Example |
| | Get embeddings by directly calling embedding API. |
| |
|
| | ```python |
| | from typing import List |
| | import numpy as np |
| | import requests |
| | |
| | def get_embedding(queries: List[str], url): |
| | response = requests.post(url=url, json={'inputs': queries}) |
| | return np.asarray(response.json()['embedding'], dtype=np.float32) |
| | |
| | url = "https://api.sionic.ai/v2/embedding" |
| | inputs1 = ["first query", "second query"] |
| | inputs2 = ["third query", "fourth query"] |
| | embedding1 = get_embedding(inputs1, url=url) |
| | embedding2 = get_embedding(inputs2, url=url) |
| | cos_similarity = (embedding1 / np.linalg.norm(embedding1)) @ (embedding2 / np.linalg.norm(embedding1)).T |
| | print(cos_similarity) |
| | ``` |
| |
|
| | Using pre-defined [SionicEmbeddingModel](https://huggingface.co/sionic-ai/sionic-ai-v2/blob/main/model_api.py) to obtain embeddings. |
| |
|
| | ```python |
| | from model_api import SionicEmbeddingModel |
| | import numpy as np |
| | |
| | inputs1 = ["first query", "second query"] |
| | inputs2 = ["third query", "fourth query"] |
| | model = SionicEmbeddingModel(url="https://api.sionic.ai/v2/embedding", |
| | dimension=3072) |
| | embedding1 = model.encode(inputs1) |
| | embedding2 = model.encode(inputs2) |
| | cos_similarity = (embedding1 / np.linalg.norm(embedding1)) @ (embedding2 / np.linalg.norm(embedding1)).T |
| | print(cos_similarity) |
| | ``` |
| | We apply the instruction to encode short queries for retrieval tasks. |
| | By using `encode_queries()`, you can use the instruction to encode queries which is prefixed to each query as the following example. |
| | The recommended instruction for both v1 and v2 models is `"query: "`. |
| |
|
| | ```python |
| | from model_api import SionicEmbeddingModel |
| | import numpy as np |
| | |
| | query = ["first query", "second query"] |
| | passage = ["This is a passage related to the first query", "This is a passage related to the second query"] |
| | model = SionicEmbeddingModel(url="https://api.sionic.ai/v2/embedding", |
| | instruction="query: ", |
| | dimension=3072) |
| | query_embedding = model.encode_queries(query) |
| | passage_embedding = model.encode_corpus(passage) |
| | cos_similarity = (query_embedding / np.linalg.norm(query_embedding)) @ (passage_embedding / np.linalg.norm(passage_embedding)).T |
| | print(cos_similarity) |
| | ``` |
| |
|
| | ## Massive Text Embedding Benchmark (MTEB) Evaluation |
| |
|
| | Both versions of Sionic AI's embedding show the state-of-the-art performances on the MTEB! |
| | You can find a code to evaluate MTEB datasets using Sionic embedding APIs [here](https://huggingface.co/sionic-ai/sionic-ai-v2/blob/main/mteb_evaluate.py). |
| |
|
| | | Model Name | Dimension | Sequence Length | Average (56) | |
| | |:-----------------------------------------------------------------------:|:---------:|:---------------:|:------------:| |
| | | [sionic-ai/sionic-ai-v2](https://huggingface.co/sionic-ai/sionic-ai-v2) | 3072 | 512 | **65.23** | |
| | | [sionic-ai/sionic-ai-v1](https://huggingface.co/sionic-ai/sionic-ai-v1) | 2048 | 512 | 64.92 | |
| | | [bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) | 1024 | 512 | 64.23 | |
| | | [gte-large-en](https://huggingface.co/barisaydin/gte-large) | 1024 | 512 | 63.13 | |
| | | [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings/types-of-embedding-models) | 1536 | 8191 | 60.99 | |
| |
|
| |
|
