# Reliability of Large Language Models for Design Synthesis: An Empirical Study of Variance, Prompt Sensitivity, and Method Scaffolding

**Authors:** Rabia Iftikhar, Andreas Rausch  
**arXiv:** [2604.00851](https://arxiv.org/abs/2604.00851)  
**Date:** 2026-04-01  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2604.00851v1/

## Key points

- Large Language Models (LLMs) are used in software engineering for generating UML class diagrams from natural language descriptions
- LLMs can produce syntactically correct diagrams but may lack meaningful design synthesis
- A preference-based few-shot prompting approach is introduced to bias LLM outputs towards object-oriented principles and pattern-consistent structures
- Evaluation involves three LLMs and three modeling strategies, showing that preference-based alignment improves adherence to design intent but does not eliminate non-determinism
- Model behavior significantly influences the reliability of generated designs, highlighting the importance of effective prompting techniques and understanding model behavior for dependable LLM-assisted software design
- LLMs may not capture essential design principles like abstraction and encapsulation, leading to inconsistency and unreliability in downstream implementation processes

## Summary

, , , , 


Large Language Models (LLMs) are increasingly being utilized in software engineering tasks, particularly in the automation of generating UML class diagrams from natural language descriptions. Previous research has shown that LLMs can produce syntactically correct diagrams, but simply adhering to syntax does not ensure meaningful design. This study delves into whether LLMs can go beyond mere diagram translation and engage in design synthesis, while also exploring how consistently they maintain design-oriented reasoning amidst variations. To enhance the quality of generated designs, a preference-based few-shot prompting approach is introduced. This method biases LLM outputs towards designs that align with object-oriented principles and exhibit pattern-consistent structures. The evaluation involves three LLMs (ChatGPT 4o-mini, Claude 3.5 Sonnet, Gemini 2.5 Flash) across three modeling strategies: standard prompting, rule-injection prompting, and preference-based prompting. A total of 540 experiments are conducted using two design-intent benchmarks with domain-only prompts and repeated runs. Results indicate that the preference-based alignment improves adherence to design intent but does not completely eliminate non-determinism. Moreover, the behavior of the models significantly influences the reliability of the generated designs. These findings underscore the importance of not only effective prompting techniques but also careful consideration of model behavior and robustness when aiming for dependable LLM-assisted software design. Furthermore, it is highlighted that while LLMs can produce syntactically valid diagrams from natural language descriptions, there is a gap in capturing essential design principles such as abstraction and encapsulation that experienced designers apply for extensibility and reusability. Without this crucial knowledge embedded in the generated diagrams, there is a risk of inconsistency and unreliability in downstream implementation processes. In conclusion, achieving reliable LLM-assisted software design requires a holistic approach that encompasses effective prompting methods alongside an understanding of model behavior and robustness to ensure consistent adherence to design intent and principles throughout the generation process.