Building Large Language Models in Kazakh Language: Kazakhstan’s Experience and Global Trends

The article analyzes the scientific and practical aspects of developing an artificial intelligence-based Large Language Model (LLM) oriented toward the Kazakh language as a basic platform for 2024-2026. The relevance of the study is determined by the need to create a full-fledged digital representation of Kazakh, preserve national identity, and ensure linguistic sovereignty in the context of digitalization. The aim of the work is to describe the theoretical principles of the LLM, the structure of a transformer-type neural network, and the composition and annotation of the dataset, as well as to demonstrate their compliance with national and international ethical and legal norms. The data corpus covers national mentality, historical and cultural traditions, and current legislation. The materials, grouped in several key categories, include question–answer pairs and expert commentary and are formulated with ethical requirements in mind. The paper discusses modern methods of text processing, approaches to reducing erroneous and fabricated model outputs, and a qualitative assessment of the results obtained. The scientific novelty is reflected in the systematic description of the first comprehensive large language model (LLM) for the Kazakh language; its practical significance is demonstrated by its potential applications in education, scientific analytics, digital services, and the implementation of state language policy. 

1. Bektaev, Q.B., Zhubanov, A.Q., Myrzabekov, S., Belbotaev, A.B. (1995) M.O. Auezovting 20 tomdyq shygarmalar teksterining zhiilik sozdikteri. Almaty-Turkistan. 346 b. [Bektayev, K.B., Zhubanov, A.K., Myrzabekov, S., Belbotayev, A.B. (1995) Frequency dictionaries of texts of 20-volume works of M.O. Auezov. Almaty-Turkestan. 346 p.] (in Kazakh)

2. Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S. (2021) On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? FAccT. https://dl.acm.org/doi/10.1145/3442188.3445922 (in English)

3. BigScience Workshop et al. (2022) Bloom: A 176B-Parameter Open-Access Multilingual Language Model. arXiv. https://arxiv.org/abs/2211.05100 (in English)

4. Bommasani, R., et al. (2021) On the Opportunities and Risks of Foundation Models. arXiv preprint. https://arxiv.org/abs/2108.07258 (in English)

5. Brown, T.B., Mann, B., Ryder, N., et al. (2020) Language Models are Few-Shot Learners. NeurIPS. https://arxiv.org/abs/2005.14165 (in English)

6. Bubeck, S., et al. (2023) Sparks of Artificial General Intelligence: Early experiments with GPT-4. https://arxiv.org/abs/2303.12712 (in English)

7. Chung, H.W., Hou, L., et al. (2022) Scaling Instruction-Finetuned Language Models. arXiv. Conneau, A., Khandelwal, K. et al. (2020) Unsupervised cross-lingual representation learning at scale. NeurIPS. (in English)

8. Goodfellow, I., Bengio, Y., Courville, A. (2016) Deep Learning. MIT Press. P. 801. (in English)

9. Google Kazakh NLP team (2021) Kazakh Language and Speech Resources for the Next Generation. https://sites.google.com/view/kaznlp (in English)

10. Hoffmann, J. et al. (2022) Training Compute-Optimal Large Language Models. https://arxiv.org/abs/2203.15556 (in English)

11. Khan, A., Ismail, S.A. (2022) Challenges for Building Natural Language Generation Models in Turkic Languages. IEEE Access. (in English)

12. Levine, Y., et al. (2021) The Depth-to-Width Ratio for ReLU Networks: Past and Present. NeurIPS. (in English)

13. Serikbayeva, A., Olach, Z. (2025) Kazakh language in the context of globalization and digital transformation: language changes. Tiltanym. No.1 (97). Pp. 151-160. (in English)

14. Tokaev, K.-Zh. (2025) Poslanie glavy gosudarstva narodu Kazahstana «Kazahstan v jepohu iskusstvennogo intellekta: aktual'nye zadachi i ih reshenija cherez cifrovuju transformaciju». Astana, 8 sentjabrja 2025 goda. [Tokayev, K.-J. (2025) Address of the Head of State to the People of Kazakhstan “Kazakhstan in the Era of Artificial Intelligence: Current Tasks and Their Solutions through Digital Transformation.” Astana, September 8, 2025.] (in Russian)

15. Tulegenov, E. Myrzabekov, S. Abduvaliyev, A. (2023) Kazakh LLM: Technological Foundation and Challenges in Local Adaptation. IMBISS Project Report, Almaty, Kazakhstan. (in English)

16. Vaswani, A., et al. (2017) Attention is All You Need. NeurIPS. https://arxiv.org/abs/1706.03762 (in English)

17. Weidinger, L., et al. (2022) Ethical and social risks of harm from language models. Science, 376(6595). Pp. 939-944. (in English)

18. Wilkinson, S., Cunningham, S.J. (2022) Low-Resource Language Modelling and Data Augmentation». Machine Learning for NLP. https://arxiv.org/abs/2310.15799 (in English)

19. Zhang, S., Sun, A. et al. (2021) Ethics and Bias in Large Language Models: A Survey. ACM Computing Surveys. (in English)

20. Zhubanov, A., Zhanabekova, A., Karbozova, B., Qozhahmetova, A. (2016) Qazaq tilining zhiilik sozdigi. Almaty: Qazaq tili. 792 b. [Zhubanov, A., Zhanabekova, A., Karbozova, B., Kozhakhmetova, A. (2016) Frequency dictionary of the Kazakh language. Almaty: Qazaq tili. 792 p.] (in Kazakh)