VMLU

Leaderboard

The following table presents benchmarking results for popular foundation models using zero-shot prompting. These models were evaluated either by calling APIs or running models directly. For more information about each specific model, please click on the model's name. Individuals and organizations are welcome to submit their model's predictions to VMLU at any time, with either zero-shot or few-shot evaluation. Click here to navigate to the submission page. Please note that your results will not be published on the Leaderboard unless you request it.

Leaderboard of from-scratch models

#	Model	Creator	Access	Evaluation date	Stem	Social Science	Humanities	Others	Avg
1	GPT-4	OpenAI	API	08/01/2024	63.84	71.78	66.14	60.37	65.53
2	gemini	Google	API	30/01/2024	42.8	60.31	55.35	51.30	51.03
3	ChatGPT	OpenAI	API	08/01/2024	43.24	51.67	46.96	46.32	46.33
4	ViGPT-1.6B-v1	Vin BigData	Private	08/01/2024	35.06	48.72	47.20	42.54	42.34
5	gemma-7b-it	Google	Weight	22/02/2024	39.95	44.93	43.39	40.11	41.9
6	Qwen-7B	Alibaba Cloud	Weight	08/01/2024	30.64	35.07	34.15	32.68	32.81
7	gemma-2b-it	Google	Weight	22/02/2024	24.39	29.59	31.01	26.81	27.72
8	sealion7b	AI Singapore	Weight	08/01/2024	26.28	28.57	27.66	27.34	26.73
9	bloom-1b7	BigScience	Weight	08/01/2024	25.13	25.09	26.34	25.19	25.51
10	bloom-7b1	BigScience	Weight	08/01/2024	25.08	26.26	25.74	24.59	25.41
11	falcon-7b	Technology Innovation Institute	Weight	08/01/2024	24.19	23.59	26.72	24.73	24.96
12	PhoGPT-7B5-Instruct	Vin AI	Weight	08/01/2024	21.97	25.93	24.32	26.00	24.01
13	Llama-2-7b-hf	Facebook Research - Meta	Weight	08/01/2024	21.48	23.41	24.10	23.59	22.95
14	falcon-7b-instruct	Technology Innovation Institute	Weight	08/01/2024	9.50	13.63	14.98	6.13	11.39

Leaderboard of fine-tuned models

#	Model	Creator	Access	Base Model	Evaluation date	Stem	Social Science	Humanities	Others	Avg
1	VNPTAI.IO-14B	VNPT AI	Private	Qwen1.5-14B-Chat	11/03/2024	51.64	61.75	58.09	54.51	55.83
2	SeaLLM-7B-v2.5	DAMO Academy	Private	llama-2-7b	09/04/2024	49.35	60.66	55.95	49.05	53.30
3	Vistral-7B-Chat	UONLP x Ontocord	Weight	Mistral-7B-v0.1	16/01/2024	43.32	57.02	55.12	48.01	50.07
4	SeaLLM-7b-v2	DAMO Academy	Weight	llama-2-7b	15/02/2024	39.95	52.02	49.38	45.27	45.79
5	bloomz-7b1	BigScience	Weight	Bloom-7b1	08/01/2024	32.63	45.73	41.85	39.89	38.87
6	T-Llama	FPTU HCM	Weight	llama-2-7b	18/03/2024	32.2	43.15	40.31	36.57	37.28
7	vbd-llama2-7b-50b-chat	Vin BigData	Weight	llama-2-7b	08/01/2024	31.45	40.34	40.24	39.62	36.98
8	vietcuna-3b	Virtual Interactive	Weight	bloomz-3b	08/01/2024	30.12	39.92	37.86	33.83	34.79
9	bloomz-1b7	BigScience	Weight	Bloom-1b7	08/01/2024	29.72	40.17	34.73	33.41	33.65
10	SeaLLM-7B-Hybrid	DAMO Academy	Weight	llama-2-7b	08/01/2024	29.49	34.61	36.68	34.52	33.39
11	ura-llama-7b	Ho Chi Minh City University of Technology	Weight	llama-2-7b	08/01/2024	29.19	33.31	34.64	32.97	32.18
12	vinallama-7b-chat	Virtual Interactive	Weight	llama-2-7b	08/01/2024	25.70	34.50	33.87	31.41	30.64
13	vietcuna-7b-v3	Virtual Interactive	Weight	bloomz-7b	08/01/2024	28.70	33.94	31.32	28.24	30.34
14	vietnamese-llama2-7b-40GB	BKAI - HUST	Weight	llama-2-7b	08/01/2024	23.22	25.61	26.71	26.30	25.19

DISCLAIMER: Please note that evaluating models like LLMs can be challenging, as leaderboards might be susceptible to manipulation, and a small tweak in prompting can lead to totally different results. It's especially concerning because some models are not publicly accessible. For instance, good results can be achieved through distilling answers from stronger models like GPT-4 or even from humans. Therefore, it's important to approach leaderboard scores with caution. Most of the models assessed here are public With Open Access, which have public weights or APIs for verification.