LES: Choosing the Right Slovene Embedding Model

The leaderboard contains results for 25 models across 26 tasks. Rather than walking through every number, we focus on four questions that come up repeatedly when choosing a model for a real application.

The WikipediaQA retrieval dataset in LES is evaluated at increasing corpus sizes: approximately 10k, 50k, 100k, 200k, and 330k documents. The queries remain the same, only the amount of noise (irrelevant documents that the model must filter through) increases.

As shown in Figure 1, all models degrade as the corpus grows, but the rate of degradation varies significantly. Among the top 10 models, google/gemini-embedding-001 experiences only a 6% drop in nDCG@10 from the smallest to the largest corpus, while openai/text-embedding-3-large drops by 14%.

More importantly, rankings shift. Qwen/Qwen3-Embedding-8B and BAAI/bge-m3 start at similar scores on the 10k corpus, but by 200k documents their gap widens to nearly 2%. intfloat/multilingual-e5-small begins as the weakest among the top 10 but overtakes 2 models by the time the full corpus is used.

At the extreme end, cjvt/GaMS-1B loses 27% of its performance, the largest drop among all evaluated models.

Figure 2 shows model size against overall LES score which is simply the mean performance over classification, clustering and retrieval measures.

The starkest example: Snowflake/snowflake-arctic-embed-l-v2.0 (0.6B parameters) scores 55.63 overall, while Qwen/Qwen3-Embedding-8B (8B parameters, 13× larger) scores 55.72. That is a 0.09 point improvement for 13× the compute.

At the smaller end, Alibaba-NLP/gte-multilingual-base (0.3B parameters) scores 53.52, roughly 9% behind the top model but at a fraction of the size. For many production scenarios, that tradeoff is worth making.

Furthermore, higher-dimensional embeddings increase storage costs and slow down similarity search. If a 768-dim model scores close to a 1024-dim model, the smaller embedding may be preferable in a vector database at scale. Resource differences extend beyond embedding size: Qwen/Qwen3-Embedding-8B has a context window of 32,768 tokens and 8B parameters, while intfloat/multilingual-e5-large-instruct uses a 512-token window at 0.6B parameters, yet the smaller model performs only 3% worse overall. Depending on your latency and infrastructure constraints, that tradeoff may be easy to accept.

To isolate the effect of class granularity, we compare model performance across datasets with different numbers of target classes: FrenkBinary and FrenkMulticlass (hate speech detection, using the same texts but with 2 and 6 labels respectively), X-GENRE (text genre classification, 9 classes), and KASTitle (university thesis classification by faculty, 9 classes). The table below displays classification performance on these four datasets with the number of classes shown in parentheses. The measure displayed is accuracy (proportion of correctly classified texts, higher is better).

Binary classification is easier for every model. But the size of the drop when moving to more classes varies.

intfloat/multilingual-e5-large-instruct ranks 6th overall on the classification tasks but places 2nd on FrenkBinary. However, its performance drops more steeply than most on the multiclass version, suggesting it captures broad polarity well but struggles with fine-grained distinctions.

Interestingly, the 9-class tasks (X-GENRE and KAS) do not always show a larger drop than the 6-class FrenkMulticlass. This suggests that difficulty depends not just on the number of classes but on how semantically distinct those classes are. Distinguishing between faculties or text genres involves clearer topical boundaries than distinguishing between types of offensive speech.

Here a striking inversion appears: all models score near zero on binary clustering yet achieve V-measure scores above 0.35 on the 9-class tasks. As discussed in the previous blog post, V-measure captures both cluster purity and completeness, so values close to zero suggest near-random grouping. When there are only two clusters, even small impurities destroy the score, while tasks with more classes offer richer topical structure that embeddings can separate more naturally.

Retrieval tasks in LES vary in the relationship between query and document. Some tasks match short texts to longer texts (sentence-to-paragraph, or s2p), while others match long texts to long texts (paragraph-to-paragraph, or p2p).

In the RTVArticles dataset, we can compare both:

• • s2p: title → abstract
• • p2p: abstract → full article body

The results show the nDCG@10 score, which measures how well the model ranks the relevant document in the top 10 results (higher is better, 1.0 is perfect). The s2p setting is easier for all models, with scores around 0.93–0.96, while p2p scores are more spread out between 0.91 and 0.97.

The overall retrieval winner, google/gemini-embedding-001, leads in both settings. But among the remaining models, rankings shift noticeably. BAAI/bge-m3 ranks second highest in the s2p column (behind gemini) but drops to 8th in p2p, while openai/text-embedding-3-large moves in the opposite direction, jumping from 5th in s2p to 2nd in p2p.

Based on the full LES results, here are concrete starting points for common use cases. The table below maps common use cases to the most relevant LES datasets and recommends models in two tiers: best raw performance and best value at a smaller size. Models marked with API are API-only and require sending data to an external service.

Over three posts we introduced LES, explained its evaluation metrics, and analysed what the results reveal. The central finding is simple but important: there is no single best Slovene embedding model. The right choice depends on your task type, your corpus size, the granularity of your categories, and the tradeoffs you are willing to accept between performance and cost.

LES does not yet cover every embedding use case. Tasks such as semantic textual similarity, bitext mining, reranking, and pair classification are not part of the current benchmark. Some domains remain underrepresented and certain datasets are small enough that results should be interpreted with caution.

The right model is the one that fits your task, your corpus, and your constraints. LES gives you the evidence to find it.