VMLU

Leaderboard

The following table presents benchmarking results for popular foundation models using zero-shot prompting. These models were evaluated either by calling APIs or running models directly. For more information about each specific model, please click on the model's name. Individuals and organizations are welcome to submit their model's predictions to VMLU at any time, with either zero-shot or few-shot evaluation. Click here to navigate to the submission page. Please note that your results will not be published on the Leaderboard unless you request it.

Leaderboard of from-scratch models

#	Model	Creator	Access	Evaluation date	Stem	Social Science	Humanities	Others	Avg
1	Llama-3-70B	Meta	Weight	23/04/2024	61.7	74.91	68.74	63.53	66.44
2	KiLM-13b-v24.7.1	Zalo AI	Private	01/08/2024	60.29	73.15	71.85	60.13	66.07
3	GPT-4	OpenAI	API	08/01/2024	63.84	71.78	66.14	60.37	65.53
4	Gpt-4o-mini	OpenAI	API	01/08/2024	58	70.95	65.03	60.91	62.87
5	gemma-2-9b-it	Google	Weight	01/08/2024	56.31	65.79	60.8	54.44	59.04
6	gemini	Google	API	30/01/2024	42.8	60.31	55.35	51.30	51.03
7	ChatGPT	OpenAI	API	08/01/2024	43.24	51.67	46.96	46.32	46.33
8	ViGPT-1.6B-v1	Vin BigData	Private	08/01/2024	35.06	48.72	47.20	42.54	42.34
9	gemma-7b-it	Google	Weight	22/02/2024	39.95	44.93	43.39	40.11	41.9
10	microsoft/Phi-3-small-128k-instruct	Microsoft	Weight	01/08/2024	39.31	44.82	41.78	40.65	41.24
11	microsoft/Phi-3-small-8k-instruct	Microsoft	Weight	01/08/2024	38.72	43.60	42.32	39.99	40.88
12	Qwen-7B	Alibaba Cloud	Weight	08/01/2024	30.64	35.07	34.15	32.68	32.81
13	Qwen2-7B-Instruct	Alibaba Cloud	Weight	01/08/2024	21.96	35.24	33.13	29.29	28.85
14	gemma-2b-it	Google	Weight	22/02/2024	24.39	29.59	31.01	26.81	27.72
15	sealion7b	AI Singapore	Weight	08/01/2024	26.28	28.57	27.66	27.34	26.73
16	bloom-1b7	BigScience	Weight	08/01/2024	25.13	25.09	26.34	25.19	25.51
17	bloom-7b1	BigScience	Weight	08/01/2024	25.08	26.26	25.74	24.59	25.41
18	falcon-7b	Technology Innovation Institute	Weight	08/01/2024	24.19	23.59	26.72	24.73	24.96
19	PhoGPT-7B5-Instruct	Vin AI	Weight	08/01/2024	21.97	25.93	24.32	26.00	24.01
20	Llama-2-7b-hf	Facebook Research - Meta	Weight	08/01/2024	21.48	23.41	24.10	23.59	22.95
21	falcon-7b-instruct	Technology Innovation Institute	Weight	08/01/2024	9.50	13.63	14.98	6.13	11.39

Leaderboard of fine-tuned models

#	Model	Creator	Access	Base Model	Evaluation date	Stem	Social Science	Humanities	Others	Avg
1	CakebyVPBank-Large	BeFinancial	Private	Unknown	22/10/2024	77.75	78.11	70.38	67.82	73.99
2	VNPTAI.IO-Large-v2	VNPT AI	Private	UNKNOWN	28/09/2024	70.07	79.5	73.77	68.79	72.65
3	CakebyVPBank-Small	BeFinancial	Private	Unknown	22/10/2024	63.95	70.68	67.63	61.17	65.82
4	Llama3-ZAI	Zalo AI	Private	Llama3-8b	01/08/2024	59.17	71.73	70.98	61.37	65.34
5	Llama3-ViettelSolutions-8B	VTS DASC	Private	Llama3-8b	01/08/2024	51.52	62.42	60.12	52.37	56.20
6	VNPTAI.IO-14B	VNPT AI	Private	Qwen1.5-14B-Chat	11/03/2024	51.64	61.75	58.09	54.51	55.83
7	Vintern-3B-beta	5CD-AI	Private	Qwen2.5-3B-Instruct	22/10/2024	51.7	61.01	58.41	51.98	54.81
8	SeaLLM-7B-v2.5	DAMO Academy	Private	llama-2-7b	09/04/2024	49.35	60.66	55.95	49.05	53.30
9	Ml4uLLM-7B-Chat	ML4U	Weight	Mistral-7B-v0.1	27/05/2024	44.72	58.69	56.86	52.36	52.08
10	Vistral-7B-Chat	UONLP x Ontocord	Weight	Mistral-7B-v0.1	16/01/2024	43.32	57.02	55.12	48.01	50.07
11	SDSRV-7B-chat	SDSRV teams	Private	Mistral-7B-v0.1	26/04/2024	36.29	60.55	55.95	49.05	48.55
12	Arcanic Cono 1.5	Arcanic AI	Private	Mistral-7B-v0.1	04/05/2024	45.11	52.44	51.97	45.36	47.45
13	SeaLLM-7b-v2	DAMO Academy	Weight	llama-2-7b	15/02/2024	39.95	52.02	49.38	45.27	45.79
14	bloomz-7b1	BigScience	Weight	Bloom-7b1	08/01/2024	32.63	45.73	41.85	39.89	38.87
15	T-Llama-7b	FPTU HCM	Weight	llama-2-7b	18/03/2024	32.2	43.15	40.31	36.57	37.28
16	vbd-llama2-7b-50b-chat	Vin BigData	Weight	llama-2-7b	08/01/2024	31.45	40.34	40.24	39.62	36.98
17	vietcuna-3b	Virtual Interactive	Weight	bloomz-3b	08/01/2024	30.12	39.92	37.86	33.83	34.79
18	bloomz-1b7	BigScience	Weight	Bloom-1b7	08/01/2024	29.72	40.17	34.73	33.41	33.65
19	SeaLLM-7B-Hybrid	DAMO Academy	Weight	llama-2-7b	08/01/2024	29.49	34.61	36.68	34.52	33.39
20	ura-llama-7b	Ho Chi Minh City University of Technology	Weight	llama-2-7b	08/01/2024	29.19	33.31	34.64	32.97	32.18
21	vinallama-7b-chat	Virtual Interactive	Weight	llama-2-7b	08/01/2024	25.70	34.50	33.87	31.41	30.64
22	vietcuna-7b-v3	Virtual Interactive	Weight	bloomz-7b	08/01/2024	28.70	33.94	31.32	28.24	30.34
23	vietnamese-llama2-7b-40GB	BKAI - HUST	Weight	llama-2-7b	08/01/2024	23.22	25.61	26.71	26.30	25.19

DISCLAIMER: Please note that evaluating models like LLMs can be challenging, as leaderboards might be susceptible to manipulation, and a small tweak in prompting can lead to totally different results. It's especially concerning because some models are not publicly accessible. For instance, good results can be achieved through distilling answers from stronger models like GPT-4 or even from humans. Therefore, it's important to approach leaderboard scores with caution. Most of the models assessed here are public With Open Access, which have public weights or APIs for verification.