Международный научный вестник2782-384910.24412/2782-3849-2026-6-602-612OtherМетоды представления текстовой информации и архитектура трансформеров в задачах обработки языкаMethods of representing text information and transformer architecture in language processing problemsСеменовАлександр АлександровичSemenovAleksandr Aleksandrovichmail@law-books.ruАветисянКарэн РафаеловичAvetisyanKaren RafaelovichKaren-Avetisyan-1989@bk.ruChief Researcher, Federal State Budgetary Institution “STIS” of the Ministry of Internal Affairs of the Russian Federationнаучный сотрудник ФКУ НПО «СТиС» МВД РоссииSenior Lecturer, Department of Information Technology and Cybercrime Investigation Organization Moscow Academy of the Investigative Committee of the Russian Federation named after A. Ya. SukharevСтарший преподаватель кафедры информационных технологий и организации расследования киберпреступлений Московская академия Следственного комитета Российской Федерации имени А.Я. Сухарева3006202666026121806202625062026

Статья посвящена методам преобразования текстовой информации в числовую форму, включая токенизацию и эмбэддинги. Рассматриваются алгоритмы Word2Vec и BPE, а также подробно описывается механизм внимания и архитектура трансформеров. Отдельное внимание уделено большим языковым моделям, их обучению и ограничениям, включая проблему галлюцинаций.

This article discusses methods for converting textual information into numerical form, including tokenization and embedding. The Word2Vec and BPE algorithms are discussed, along with a detailed description of the attention mechanism and Transformer architecture. Special attention is given to large language models, their training, and limitations, including the problem of hallucinations.

обработка текстатокенизацияэмбэддингиWord2VecBPEмеханизм вниманиятрансформерattentionбольшие языковые моделигенерация текстанейросетиtext processingtokenizationembeddingWord2VecBPEattention mechanismTransformerattentionlarge language modelstext generationneural networksChen Zhang, Joohee Kim Video object detection with two-path convolutional LSTM pyramid. 2016. – URL: https://www.researchgate.net/ publication /343702579_Video_Object_Detection_ With_ Two-Path_ Convolutional_LSTM_ Pyramid/ fulltext/5f3b2658928 51cd302013482/Video-Object- Detection-With-Two-Path-Convolutional-LSTM- Pyramid.pdf/ (дата обращения: 16.01.2026).Hanson A., Koutilya PNVR, Sanjukta Krishnagopal, Larry Davis Bidirectional Convolutional LSTM for the Detection of Violence in Videos. 2018. – URL: https://openaccess.thecvf. com/content_eccv_2018_workshops/w10 /html/ Hanson_Bidirectional_Convolutional_LSTM_or_the_ Detection_of_ Violence_in_ Videos_ ECCVW_2018_ paper.html (дата обращения: 15.01.2026).Howard A.G., Menglong Zhu, Bo Chen, Kalenichenko D. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. 2017. – URL: https://arxiv.org/pdf/1704.04861/ (дата обращения: 11.01.2026).Iandola F.N., Han S., Moskewicz M.W., Ashraf K., Dally W.J., Keutzer K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size Recognition. 2016. Published as a conference paper at ICLR. 2015. – URL: https:// www. researchgate.net/publication/ 301878495_ SqueezeNet_AlexNet-level_accuracy_ with_50x_ fewer_ parameters_ and_05MB_model_size (дата обращения: 10.01.2026).Ioffe S., Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. 2015. – URL: https://arxiv.org/pdf/1502.03167/ (дата обращения: 13.01.2026).Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun Deep Residual Learning for Image Recognition. 2015. – URL: https://arxiv.org/pdf/1512.03385 (дата обращения: 11.01.2026).Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun Spatial pyramid pooling in deep convolutional networks for visual recognition. 2015. – URL: https://arxiv.org/pdf/1406.4729/ (дата обращения: 11.01.2026).Krizhevsky I., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural networks. 2012. – URL: https://proceedings. neurips.cc/paper_files/paper/2012/file/ c399862d3b9d6b76c8436e 924a68c45b-Paper.pdf (дата обращения: 12.01.2026).Lavin A., Scott Gray Fast Algorithms for Convolutional Neural Networks. 2015. – URL: https://arxiv.org/abs/1509.09308 (дата обращения: 11.01.2026).Mikolov T., Kai Chen, Corrado G., Jeffrey Dean Efficient Estimation of Word Representations in Vector Space. 2013. – URL: https://arxiv.org/abs/1301.3781 (дата обращения: 15.01.2026).Norman P. Jouppi, Cliff Young, Nishant Patil and others In-Datacenter Performance Analysis of a Tensor Processing Unit. 2017. – URL: https://arxiv.org/pdf/1704.04760 (дата обращения: 11.01.2026).Ronneberger O., Fischer Ph., Brox Th. U-Net: Convolutional Networks for Biomedical Image Segmentation. 2015. – URL: https://arxiv.org/pdf/1505.04597/ (дата обращения: 13.01.2026).Szegedy C., Wei Liu, Chapel Hill, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich Going Deeper with Convolutions. 2014. – URL: https://arxiv.org/pdf/1409.4842 (дата обращения: 12.01.2026)Simonyan K., Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition. 2015. Published as a conference paper at ICLR 2015. – URL: https://arxiv.org/pdf/1409.1556 (дата обращения: 10.01.2026).Wei Liu, Anguelov D., Erhan D., Szegedy C., Reed S., Cheng-Yang Fu, Berg A.C. SSD: Single Shot MultiBox Detector. 2016. – URL: https://arxiv.org/pdf/1512.02325 (дата обращения: 11.01.2026).Официальный сайт ONNX. – URL: https://onnx.ai/.Основы технологий искусственного интеллекта: учебное пособие/ А.А. Семенов, В.В. Гончар, А.С. Эрдниев. – Московский университет МВД России имени В.Я, Кикотя, 2025. – 190с.Финансовый мониторинг: учебное пособие / В.И. Глотов, А.У. Альбеков, О.Н. Тисен и др. – КноРус, 2022. – 198 с.Тисен О.Н. Противодействие современным приемам и способам вербовки в международные террористические организации // Российская юстиция. 2018, №9. С. 51-54.Гончар В.В. О важности формирования единообразного понятийного аппарата необходимого для расследования преступлений в сфере компьютерной информации// Вестник Экономической безопасности. 2018. №1. С. 225- 230.Орлова А.А., Гончар В.В. Особенности расследования преступлений, совершаемых в сфере информационных технологий // Безопасность бизнеса. 2021. № 4. С. 25-29.Chen Zhang, Joohee Kim Video object detection with two-path convolutional LSTM pyramid. 2016. – URL: https://www.researchgate.net/ publication /343702579_Video_Object_Detection_ With_ Two-Path_ Convolutional_LSTM_ Pyramid/ fulltext/5f3b2658928 51cd302013482/Video-Object- Detection-With-Two-Path-Convolutional-LSTM- Pyramid.pdf/ (дата обращения: 16.01.2026).Hanson A., Koutilya PNVR, Sanjukta Krishnagopal, Larry Davis Bidirectional Convolutional LSTM for the Detection of Violence in Videos. 2018. – URL: https://openaccess.thecvf. com/content_eccv_2018_workshops/w10 /html/ Hanson_Bidirectional_Convolutional_LSTM_or_the_ Detection_of_ Violence_in_ Videos_ ECCVW_2018_ paper.html (дата обращения: 15.01.2026).Howard A.G., Menglong Zhu, Bo Chen, Kalenichenko D. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. 2017. - URL: https://arxiv.org/pdf/1704.04861/ (access date: 11.01.2026).Iandola F.N., Han S., Moskewicz M.W., Ashraf K., Dally W.J., Keutzer K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size Recognition. 2016. Published as a conference paper at ICLR. 2015. – URL: https:// www. researchgate.net/publication/ 301878495_ SqueezeNet_AlexNet-level_accuracy_ with_50x_ fewer_ parameters_ and_05MB_model_size (дата обращения: 10.01.2026).Ioffe S., Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. 2015. – URL: https://arxiv.org/pdf/1502.03167/ (access date: 13.01.2026).Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun Deep Residual Learning for Image Recognition. 2015. – URL: https://arxiv.org/pdf/1512.03385 (access date: 11.01.2026).Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun Spatial pyramid pooling in deep convolutional networks for visual recognition. 2015. - URL: https://arxiv.org/pdf/1406.4729/ (access date: 11.01.2026).Krizhevsky I., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural networks. 2012. - URL: https://proceedings. neurips.cc/paper_files/paper/2012/file/ c399862d3b9d6b76c8436e 924a68c45b-Paper.pdf (accessed on: 12.01.2026).Lavin A., Scott Gray Fast Algorithms for Convolutional Neural Networks. 2015. - URL: https:// arxiv.org/abs/1509.09308 (access date: 11.01.2026).Mikolov T., Kai Chen, Corrado G., Jeffrey Dean Efficient Estimation of Word Representations in Vector Space. 2013. – URL: https://arxiv.org/abs/1301.3781 (access date: 15.01.2026).Norman P. Jouppi, Cliff Young, Nishant Patil and others In-Datacenter Performance Analysis of a Tensor Processing Unit. 2017. – URL: https://arxiv.org/pdf/1704.04760 (access date: 11.01.2026).Ronneberger O., Fischer Ph., Brox Th. U-Net: Convolutional Networks for Biomedical Image Segmentation. 2015. – URL: https://arxiv.org/pdf/1505.04597/ (access date: 13.01.2026).Szegedy C., Wei Liu, Chapel Hill, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich Going Deeper with Convolutions. 2014. – URL: https://arxiv.org/pdf/1409.4842 (accessed on: 12.01.2026)Simonyan K., Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition. 2015. Published as a conference paper at ICLR 2015. - URL: https://arxiv.org/pdf/1409.1556 (access date: 10.01.2026).Wei Liu, Anguelov D., Erhan D., Szegedy C., Reed S., Cheng-Yang Fu, Berg A.C. SSD: Single Shot MultiBox Detector. 2016. – URL: https://arxiv.org/pdf/1512.02325 (access date: 11.01.2026).ONNX official website. – URL: https:// onnx.ai/.Fundamentals of artificial intelligence technologies: a textbook/A.A. Semenov, V.V. Gonchar, A.S. Erdniev. - Moscow University of the Ministry of Internal Affairs of Russia named after V.Ya, Kikotya, 2025. - 190s.Financial monitoring: training manual/V.I. Glotov, A.U. Albekov, O.N. Tisen et al. - KnoRus, 2022. - 198 s.Tisen O.N. Countering modern methods and methods of recruitment to international terrorist organizations//Russian Justice. 2018, №9. S. 51-54.Gonchar V.V. On the importance of forming a uniform conceptual apparatus necessary for investigating crimes in the field of computer information//Bulletin of Economic Security. 2018. №1. S. 225-230.Orlova A.A., Gonchar V.V. Features of the investigation of crimes committed in the field of information technology//Business security. 2021. № 4. S. 25-29.