Research on the quality of automated program code generation using neural networks

Abstract

Relevance. Rapid advancements in large language models have significantly impacted software engineering, necessitating a rigorous evaluation of their capabilities in automated code generation. The aim of the study is to develop and validate a methodology for comprehensive, multi-dimensional quality assessment of program code generated by modern large language models in production-grade environments. Objectives. The study conducts a comparative analysis of the code generation quality produced by five leading neural networks – GPT-5, Claude Opus 4.1, Gemini 3.1 Pro, Grok 4, and DeepSeek-V3.1 – within the context of modern web development, evaluating their capacity to generate a production-ready, standalone Angular 19 component featuring complex drag-and-drop functionality, smooth animations, and mock Hypertext Transfer Protocol service integration. Methods. A two-level evaluation methodology was employed, combining automatic quantitative metrics – such as build correctness, TypeScript and ESLint error rates, cyclomatic complexity, bundle size, and security auditing – with a qualitative expert assessment of architectural integrity, maintainability, and documentation completeness. An Integrated Model Quality Assessment Metric was derived to rank the models based on weighted factors, prioritizing correctness (35%) and maintainability (30%) over performance (20%), documentation (10%), and security (5%). Scientific novelty. The proposed methodology integrates automated static analysis with structured expert evaluation into a single comparable metric grounded in the ISO/IEC 25010 quality framework, addressing the gap left by existing benchmarks that evaluate only functional correctness on isolated tasks. Practical significance. The findings provide crucial empirical data for selecting artificial intelligence tools in development workflows and demonstrate that production-oriented quality assessment requires multi-dimensional evaluation beyond syntactic correctness. Results. The empirical analysis revealed significant disparities across the tested architectures. Claude Opus 4.1 achieved the highest Integrated Quality Score (0.636), demonstrating superior code structure and documentation standards. Gemini 3.1 Pro followed closely (0.620), excelling in performance optimization and build stability. GPT-5 (0.447), while syntactically accurate, suffered from performance optimization issues, while DeepSeek-V3.1 (0.530) required substantial manual debugging. Grok 4 scored the lowest (0.240), with its reliance on outdated module-based architectures resulting in systemic deficiencies. Conclusions. While modern large language models are capable of generating valid code, substantial human oversight remains essential to ensure production readiness. The Integrated Quality Score proved effective in differentiating models whose surface-level syntactic performance appears similar.