Model Evaluation Summary ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓ ┃ Model ┃ Avg ┃ MIRAC ┃ MLQAR ┃ Massi ┃ Multi ┃ STS17 ┃ STS22 ┃ XNLI_ ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩ │ arabic_triplet_matryoshka_v2 │ 0.6610 │ 0.6262 │ 0.5093 │ 0.5577 │ 0.5868 │ 0.8531 │ 0.6396 │ 0.8542 │ │ muffakir_embedding │ 0.6494 │ 0.6424 │ 0.5267 │ 0.5462 │ 0.5943 │ 0.8485 │ 0.6291 │ 0.7583 │ │ arabic_retrieval_v1.0 │ 0.6473 │ 0.6159 │ 0.5674 │ 0.5832 │ 0.5993 │ 0.8002 │ 0.6254 │ 0.7393 │ │ gate_arabert-v1 │ 0.6444 │ 0.5774 │ 0.4808 │ 0.5345 │ 0.5847 │ 0.8278 │ 0.6310 │ 0.8746 │ │ get_multilingual_base │ 0.6440 │ 0.7177 │ 0.5698 │ 0.5071 │ 0.5521 │ 0.7881 │ 0.6145 │ 0.7584 │ │ arabic_sts_matryoshka │ 0.6413 │ 0.5828 │ 0.4840 │ 0.5457 │ 0.5494 │ 0.8290 │ 0.6242 │ 0.8740 │ │ silma_ai_embedding_sts_v0.1 │ 0.6138 │ 0.3799 │ 0.5011 │ 0.5600 │ 0.5749 │ 0.8559 │ 0.6122 │ 0.8125 │ │ Arabic-MiniLM-L12-v2-all-nli-triplet │ 0.5431 │ 0.2240 │ 0.3612 │ 0.4775 │ 0.5698 │ 0.8111 │ 0.5540 │ 0.8043 │ │ paraphrase-multilingual-MiniLM-L12-v2 │ 0.5208 │ 0.2191 │ 0.3496 │ 0.4515 │ 0.5573 │ 0.7916 │ 0.4908 │ 0.7859 │ │ jina_zaraah_256_arabic │ 0.4822 │ 0.2295 │ 0.3473 │ 0.4119 │ 0.5237 │ 0.6469 │ 0.6218 │ 0.5942 │ │ jina_zaraah_256_arabic_int8 │ 0.4809 │ 0.2313 │ 0.3464 │ 0.4121 │ 0.5256 │ 0.6460 │ 0.6113 │ 0.5936 │ │ potion-multilingual-128M │ 0.4699 │ 0.1658 │ 0.3150 │ 0.4285 │ 0.5338 │ 0.6511 │ 0.5951 │ 0.5999 │ │ jina_zaraah_64_int8_arabic │ 0.4276 │ 0.1612 │ 0.1998 │ 0.3084 │ 0.5113 │ 0.5999 │ 0.6290 │ 0.5833 │ │ jina_zaraah_32_int8_arabic │ 0.3892 │ 0.0924 │ 0.0986 │ 0.2186 │ 0.5144 │ 0.5787 │ 0.6323 │ 0.5894 │ │ jina_zaraah_32 │ 0.3889 │ 0.0934 │ 0.0987 │ 0.2192 │ 0.5112 │ 0.5801 │ 0.6294 │ 0.5902 │ │ all_minilm_l6_v2 │ 0.2843 │ 0.0005 │ 0.0064 │ 0.1905 │ 0.4934 │ 0.5089 │ 0.2518 │ 0.5384 │ └───────────────────────────────────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘
Zaraah Embedding models
blogging
embedding
minishlab
model2vec
arabic
Zaraah model2vec family model analysis and testing on Arabic Embeddings tasks.
Arabic Embedding Models
This blog post introduces the Zaraah family of static embedding models, designed for Arabic language tasks and built using the model2vec distillation technique from MinishLab.
These models distill knowledge from larger transformer models, such as SBERT, into compact, efficient embeddings.
This approach balances performance with speed and resource efficiency.
Below, I explore the Zaraah models, their relationship to Potion models, their strengths and limitations, and their applications in Arabic embedding tasks.
What are Potion Models?
Potion models combine innovative techniques to create high-performing, compact static embeddings.
I liken them to Bojji from Ousama Ranking small in size but capable of competing with giants like Jina AI and BGE models.
Key features of Potion models include:
Superior Performance: They outperform traditional static embeddings like GloVe and FastText across various tasks, matching the performance of models like all-MiniLM-L6-v2 in English.
Compact Size: With approximately 2–4 million parameters, they are ~55 times smaller than GloVe, with model sizes ranging from 8 MB to 30 MB.
Efficiency: Designed for CPU execution and browser-based applications, they are ideal for edge devices and low-resource environments.
MTEB Performance: They achieve an average MTEB score above 50%, making them highly competitive for their size.
What is the model2vec Distillation Method?
The model2vec distillation method addresses the challenge of creating fast, compact sentence transformers.
It transforms large sentence transformer models into static embeddings that are up to 500x faster and 15x smaller, with only a minor performance trade-off.
Unlike traditional methods like GloVe, model2vec captures knowledge from large sentence transformers, producing uncontextualized word vectors.
While this sacrifices some contextual nuance, it offers significant advantages in:
Speed: Up to 500x faster inference.
Size: Models reduced by up to 50x, ranging from 8 MB to 30 MB.
Versatility: Sufficient word representations for most NLP applications.
For more details, refer to the MinishLab blog and GitHub repository.
Jina Embeddings v3 for Arabic
The jina-embeddings-v3 model is currently the top-performing open-source, zero-shot embedding model for Arabic on the MTEB leaderboard. It excels across various tasks and has been validated in production for Arabic applications.
However, its large size and high memory requirements make it computationally expensive and slow compared to other embedding models. To address this, I used model2vec to create a compact Arabic version, the Zaraah family, which retains strong performance while being significantly smaller and faster.
Zaraah Family
The Zaraah models are the first static embedding models for Arabic trained with tokenlearn on the Arabic subset of the C4 dataset.
They are optimized for Arabic-specific tasks and come in multiple sizes:
All variants support float32 and int8 quantization without performance loss, making them highly efficient for resource-constrained environments.
Zaraah Model vs. Competitors
To evaluate Zaraah’s performance, I compared it against several multilingual and Arabic-specific sentence transformer models using MTEB tasks tailored for Arabic.
I filtered the most related MTEB tasks that supports Arabic-script only the evalution script is in the references blow. We can say that the average score for the jina_zaraah are very low compared to the other models, but didn’t let the Average score fool you! Average is affected with the outliers so, if one task is the low the final answer with be low also.
But from the first look, we can see the peformance is similar to the Arabic versions of MiniLM-L12 in Average and if you looked at the Sentence similarity for STS22 it’s score are very good compared to static-embedding model.
Understanding MTEB Tasks for Arabic
The Massive Text Embedding Benchmark (MTEB) evaluates embedding models across various tasks. Here’s a breakdown of the tasks used to assess Zaraah:
MIRACLRetrievalHardNegatives: Measures retrieval accuracy for hard negative examples, critical for search and question-answering systems. Zaraah’s lower score here reflects its static embedding nature, which sacrifices some contextual nuance.
MLQARetrieval: Tests retrieval performance on multilingual question-answering datasets, where Zaraah performs comparably to MiniLM models.
STS17 & STS22: Evaluates semantic textual similarity, where Zaraah excels, particularly in STS22, with scores rivaling larger models.
XNLI: Assesses natural language inference, where Zaraah’s performance is competitive despite its compact size.
These tasks highlight Zaraah’s strengths in semantic similarity and efficiency, making it ideal for applications like chatbots and lightweight search systems
You can see the peformance for every task in MTEB here
Sorted by MIRACLRetrievalHardNegatives_main (Score) ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Model Name ┃ MIRACLRetrievalHardNegatives_main ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ get_multilingual_base │ 0.718 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ muffakir_embedding │ 0.642 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ arabic_triplet_matryoshka_v2 │ 0.626 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ arabic_retrieval_v1.0 │ 0.616 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ arabic_sts_matryoshka │ 0.583 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ gate_arabert-v1 │ 0.577 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ silma_ai_embedding_sts_v0.1 │ 0.380 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ jina_zaraah_256_arabic_int8 │ 0.231 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ jina_zaraah_256_arabic │ 0.230 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ Arabic-MiniLM-L12-v2-all-nli-triplet │ 0.224 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ paraphrase-multilingual-MiniLM-L12-v2 │ 0.219 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ potion-multilingual-128M │ 0.166 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ jina_zaraah_64_int8_arabic │ 0.161 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ jina_zaraah_32 │ 0.093 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ jina_zaraah_32_int8_arabic │ 0.092 │ ├───────────────────────────────────────┼───────────────────────────────────┤ │ all_minilm_l6_v2 │ 0.001 │ └───────────────────────────────────────┴───────────────────────────────────┘
Sorted by MLQARetrieval_main (Score) ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓ ┃ Model Name ┃ MLQARetrieval_main ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩ │ get_multilingual_base │ 0.570 │ ├───────────────────────────────────────┼────────────────────┤ │ arabic_retrieval_v1.0 │ 0.567 │ ├───────────────────────────────────────┼────────────────────┤ │ muffakir_embedding │ 0.527 │ ├───────────────────────────────────────┼────────────────────┤ │ arabic_triplet_matryoshka_v2 │ 0.509 │ ├───────────────────────────────────────┼────────────────────┤ │ silma_ai_embedding_sts_v0.1 │ 0.501 │ ├───────────────────────────────────────┼────────────────────┤ │ arabic_sts_matryoshka │ 0.484 │ ├───────────────────────────────────────┼────────────────────┤ │ gate_arabert-v1 │ 0.481 │ ├───────────────────────────────────────┼────────────────────┤ │ Arabic-MiniLM-L12-v2-all-nli-triplet │ 0.361 │ ├───────────────────────────────────────┼────────────────────┤ │ paraphrase-multilingual-MiniLM-L12-v2 │ 0.350 │ ├───────────────────────────────────────┼────────────────────┤ │ jina_zaraah_256_arabic │ 0.347 │ ├───────────────────────────────────────┼────────────────────┤ │ jina_zaraah_256_arabic_int8 │ 0.346 │ ├───────────────────────────────────────┼────────────────────┤ │ potion-multilingual-128M │ 0.315 │ ├───────────────────────────────────────┼────────────────────┤ │ jina_zaraah_64_int8_arabic │ 0.200 │ ├───────────────────────────────────────┼────────────────────┤ │ jina_zaraah_32 │ 0.099 │ ├───────────────────────────────────────┼────────────────────┤ │ jina_zaraah_32_int8_arabic │ 0.099 │ ├───────────────────────────────────────┼────────────────────┤ │ all_minilm_l6_v2 │ 0.006 │ └───────────────────────────────────────┴────────────────────┘
Sorted by MassiveIntentClassification_main (Score) ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Model Name ┃ MassiveIntentClassification_main ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ arabic_retrieval_v1.0 │ 0.583 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ silma_ai_embedding_sts_v0.1 │ 0.560 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ arabic_triplet_matryoshka_v2 │ 0.558 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ muffakir_embedding │ 0.546 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ arabic_sts_matryoshka │ 0.546 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ gate_arabert-v1 │ 0.534 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ get_multilingual_base │ 0.507 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ Arabic-MiniLM-L12-v2-all-nli-triplet │ 0.478 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ paraphrase-multilingual-MiniLM-L12-v2 │ 0.451 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ potion-multilingual-128M │ 0.428 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ jina_zaraah_256_arabic_int8 │ 0.412 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ jina_zaraah_256_arabic │ 0.412 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ jina_zaraah_64_int8_arabic │ 0.308 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ jina_zaraah_32 │ 0.219 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ jina_zaraah_32_int8_arabic │ 0.219 │ ├───────────────────────────────────────┼──────────────────────────────────┤ │ all_minilm_l6_v2 │ 0.190 │ └───────────────────────────────────────┴──────────────────────────────────┘
Sorted by MultiHateClassification_main (Score) ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Model Name ┃ MultiHateClassification_main ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ arabic_retrieval_v1.0 │ 0.599 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ muffakir_embedding │ 0.594 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ arabic_triplet_matryoshka_v2 │ 0.587 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ gate_arabert-v1 │ 0.585 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ silma_ai_embedding_sts_v0.1 │ 0.575 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ Arabic-MiniLM-L12-v2-all-nli-triplet │ 0.570 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ paraphrase-multilingual-MiniLM-L12-v2 │ 0.557 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ get_multilingual_base │ 0.552 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ arabic_sts_matryoshka │ 0.549 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ potion-multilingual-128M │ 0.534 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ jina_zaraah_256_arabic_int8 │ 0.526 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ jina_zaraah_256_arabic │ 0.524 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ jina_zaraah_32_int8_arabic │ 0.514 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ jina_zaraah_64_int8_arabic │ 0.511 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ jina_zaraah_32 │ 0.511 │ ├───────────────────────────────────────┼──────────────────────────────┤ │ all_minilm_l6_v2 │ 0.493 │ └───────────────────────────────────────┴──────────────────────────────┘
Sorted by STS17_main (Score) ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ ┃ Model Name ┃ STS17_main ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ │ silma_ai_embedding_sts_v0.1 │ 0.856 │ ├───────────────────────────────────────┼────────────┤ │ arabic_triplet_matryoshka_v2 │ 0.853 │ ├───────────────────────────────────────┼────────────┤ │ muffakir_embedding │ 0.849 │ ├───────────────────────────────────────┼────────────┤ │ arabic_sts_matryoshka │ 0.829 │ ├───────────────────────────────────────┼────────────┤ │ gate_arabert-v1 │ 0.828 │ ├───────────────────────────────────────┼────────────┤ │ Arabic-MiniLM-L12-v2-all-nli-triplet │ 0.811 │ ├───────────────────────────────────────┼────────────┤ │ arabic_retrieval_v1.0 │ 0.800 │ ├───────────────────────────────────────┼────────────┤ │ paraphrase-multilingual-MiniLM-L12-v2 │ 0.792 │ ├───────────────────────────────────────┼────────────┤ │ get_multilingual_base │ 0.788 │ ├───────────────────────────────────────┼────────────┤ │ potion-multilingual-128M │ 0.651 │ ├───────────────────────────────────────┼────────────┤ │ jina_zaraah_256_arabic │ 0.647 │ ├───────────────────────────────────────┼────────────┤ │ jina_zaraah_256_arabic_int8 │ 0.646 │ ├───────────────────────────────────────┼────────────┤ │ jina_zaraah_64_int8_arabic │ 0.600 │ ├───────────────────────────────────────┼────────────┤ │ jina_zaraah_32 │ 0.580 │ ├───────────────────────────────────────┼────────────┤ │ jina_zaraah_32_int8_arabic │ 0.579 │ ├───────────────────────────────────────┼────────────┤ │ all_minilm_l6_v2 │ 0.509 │ └───────────────────────────────────────┴────────────┘
Sorted by STS22.v2_main (Score) ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Model Name ┃ STS22.v2_main ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ arabic_triplet_matryoshka_v2 │ 0.640 │ ├───────────────────────────────────────┼───────────────┤ │ jina_zaraah_32_int8_arabic │ 0.632 │ ├───────────────────────────────────────┼───────────────┤ │ gate_arabert-v1 │ 0.631 │ ├───────────────────────────────────────┼───────────────┤ │ jina_zaraah_32 │ 0.629 │ ├───────────────────────────────────────┼───────────────┤ │ muffakir_embedding │ 0.629 │ ├───────────────────────────────────────┼───────────────┤ │ jina_zaraah_64_int8_arabic │ 0.629 │ ├───────────────────────────────────────┼───────────────┤ │ arabic_retrieval_v1.0 │ 0.625 │ ├───────────────────────────────────────┼───────────────┤ │ arabic_sts_matryoshka │ 0.624 │ ├───────────────────────────────────────┼───────────────┤ │ jina_zaraah_256_arabic │ 0.622 │ ├───────────────────────────────────────┼───────────────┤ │ get_multilingual_base │ 0.615 │ ├───────────────────────────────────────┼───────────────┤ │ silma_ai_embedding_sts_v0.1 │ 0.612 │ ├───────────────────────────────────────┼───────────────┤ │ jina_zaraah_256_arabic_int8 │ 0.611 │ ├───────────────────────────────────────┼───────────────┤ │ potion-multilingual-128M │ 0.595 │ ├───────────────────────────────────────┼───────────────┤ │ Arabic-MiniLM-L12-v2-all-nli-triplet │ 0.554 │ ├───────────────────────────────────────┼───────────────┤ │ paraphrase-multilingual-MiniLM-L12-v2 │ 0.491 │ ├───────────────────────────────────────┼───────────────┤ │ all_minilm_l6_v2 │ 0.252 │ └───────────────────────────────────────┴───────────────┘
Sorted by XNLI_main (Score) ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Model Name ┃ XNLI_main ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ gate_arabert-v1 │ 0.875 │ ├───────────────────────────────────────┼───────────┤ │ arabic_sts_matryoshka │ 0.874 │ ├───────────────────────────────────────┼───────────┤ │ arabic_triplet_matryoshka_v2 │ 0.854 │ ├───────────────────────────────────────┼───────────┤ │ silma_ai_embedding_sts_v0.1 │ 0.813 │ ├───────────────────────────────────────┼───────────┤ │ Arabic-MiniLM-L12-v2-all-nli-triplet │ 0.804 │ ├───────────────────────────────────────┼───────────┤ │ paraphrase-multilingual-MiniLM-L12-v2 │ 0.786 │ ├───────────────────────────────────────┼───────────┤ │ get_multilingual_base │ 0.758 │ ├───────────────────────────────────────┼───────────┤ │ muffakir_embedding │ 0.758 │ ├───────────────────────────────────────┼───────────┤ │ arabic_retrieval_v1.0 │ 0.739 │ ├───────────────────────────────────────┼───────────┤ │ potion-multilingual-128M │ 0.600 │ ├───────────────────────────────────────┼───────────┤ │ jina_zaraah_256_arabic │ 0.594 │ ├───────────────────────────────────────┼───────────┤ │ jina_zaraah_256_arabic_int8 │ 0.594 │ ├───────────────────────────────────────┼───────────┤ │ jina_zaraah_32 │ 0.590 │ ├───────────────────────────────────────┼───────────┤ │ jina_zaraah_32_int8_arabic │ 0.589 │ ├───────────────────────────────────────┼───────────┤ │ jina_zaraah_64_int8_arabic │ 0.583 │ ├───────────────────────────────────────┼───────────┤ │ all_minilm_l6_v2 │ 0.538 │ └───────────────────────────────────────┴───────────┘
Zaraah vs. all-MiniLM
The Zaraah models outperform the all-MiniLM family from SBERT in Arabic tasks while being significantly faster and capable of running on CPU. This makes Zaraah an excellent lightweight alternative for applications requiring efficient Arabic embeddings.
Also the models performance is most of the time better than the potion-multilingual-128M which indicates that techniques are working with any language not just English.
Arabic RAG Leaderboard
To complement MTEB evaluations, I tested Zaraah on the Arabic-RAG Leaderboard, which provides a robust benchmark for Arabic-specific tasks. Zaraah ranks 37 out of 45 models with an average score of 36.84. This is impressive, as Zaraah is the smallest model in the leaderboard, highlighting its efficiency and competitive performance in resource-constrained settings.
Speed comprsion
Some weights of the model checkpoint at Alibaba-NLP/gte-multilingual-base were not used when initializing NewModel: {'classifier.bias', 'classifier.weight'}
- This IS expected if you are initializing NewModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing NewModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).Completed jina_zaraah_256 on cpu: 11342.11 sentences/second
Completed jina_zaraah_32 on cpu: 10947.39 sentences/second
Completed potion-multilingual-128M on cpu: 9973.48 sentences/second
Completed paraphrase-multilingual-MiniLM-L12-v2 on cuda: 2120.08 sentences/second
Completed silma_ai_embedding_sts_v0.1 on cuda: 611.62 sentences/second
Completed muffakir_embedding on cuda: 606.21 sentences/second
Completed get_multilingual_base on cuda: 876.47 sentences/second
Completed arabic_retrieval_v1.0 on cuda: 600.66 sentences/second
Completed arabic_triplet_matryoshka_v2 on cuda: 590.78 sentences/second
Model Benchmark Results ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ ┃ Model ┃ Speed (sentences/second) ┃ Device ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ │ jina_zaraah_256 │ 11342.11 │ cpu │ │ jina_zaraah_32 │ 10947.39 │ cpu │ │ potion-multilingual-128M │ 9973.48 │ cpu │ │ paraphrase-multilingual-MiniLM-L12-v2 │ 2120.08 │ cuda │ │ silma_ai_embedding_sts_v0.1 │ 611.62 │ cuda │ │ muffakir_embedding │ 606.21 │ cuda │ │ get_multilingual_base │ 876.47 │ cuda │ │ arabic_retrieval_v1.0 │ 600.66 │ cuda │ │ arabic_triplet_matryoshka_v2 │ 590.78 │ cuda │ └───────────────────────────────────────┴──────────────────────────┴────────┘
Results saved to model_info_results.csv
Model Information Results ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Model ┃ Parameters (M) ┃ Size (MB) ┃ Relative to Largest (%) ┃ Less than Largest (x) ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ │ jina_zaraah_32 │ 8.00 │ 7.63 │ 2.62 │ 38.17 │ │ jina_zaraah_256 │ 64.00 │ 61.03 │ 20.96 │ 4.77 │ │ potion-multilingual-128M │ 128.09 │ 122.16 │ 41.95 │ 2.38 │ │ paraphrase-multilingual-MiniLM-… │ 117.65 │ 112.20 │ 38.53 │ 2.60 │ │ silma_ai_embedding_sts_v0.1 │ 135.19 │ 128.93 │ 44.27 │ 2.26 │ │ muffakir_embedding │ 135.19 │ 128.93 │ 44.27 │ 2.26 │ │ arabic_retrieval_v1.0 │ 135.19 │ 128.93 │ 44.27 │ 2.26 │ │ arabic_triplet_matryoshka_v2 │ 135.19 │ 128.93 │ 44.27 │ 2.26 │ │ get_multilingual_base │ 305.37 │ 291.22 │ 100.00 │ 1.00 │ └──────────────────────────────────┴────────────────┴───────────┴─────────────────────────┴───────────────────────┘
What’s Next for Zaraah?
It’s just the start with initial tests, there is more to explore from the base models, datasets and the new features from minishlab which will try to narrow the gab between model2vec and sentence-transformers.