# Structured Prompt Tuning

**Authors:** Chi-Liang Liu, Hung-yi Lee, Wen-tau Yih  
**arXiv:** [2205.12309](https://arxiv.org/abs/2205.12309)  
**Date:** 2022-05-24  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2205.12309v1/

## Key points

- Structured prompt tuning is a simple and effective method in NLP to enhance model performance
- Soft prompt embeddings are generated through a hypernetwork, providing flexibility in model design for single-task and multi-task scenarios
- Empirical studies show structured prompt tuning leads to a significant improvement on the GLUE benchmark and reduced sensitivity to learning rate changes
- Various strategies exist for designing effective prompts in NLP research, such as AutoPrompt and soft-prompt methods with continuous embeddings
- Structured prompt tuning streamlines prompt generation and integration into pre-trained language models, enhancing model performance across tasks

## Summary

, , , , 

Structured prompt tuning has emerged as a simple yet effective method in the field of Natural Language Processing (NLP) to enhance model performance. Unlike traditional approaches that involve adding tunable embeddings to the input, this innovative technique generates soft prompt embeddings through a hypernetwork. This not only encompasses standard prompt tuning but also offers greater flexibility in model design, making it suitable for both single-task and multi-task training scenarios. Empirical studies have shown that structured prompt tuning leads to a significant improvement of +1.2 to 1.5 points on the General Language Understanding Evaluation (GLUE) benchmark and exhibits reduced sensitivity to changes in learning rate compared to conventional methods. The concept of prompting or in-context learning has gained traction in NLP research, with various studies exploring different strategies for designing effective prompts. For instance, AutoPrompt leverages gradient-guided search while other approaches focus on generating high-quality prompts using template candidates or soft-prompt methods with continuous embeddings. Structured prompt tuning builds upon these existing techniques by introducing a novel approach that streamlines the process of prompt generation and integration into pre-trained language models. By utilizing a hypernetwork to generate soft prompt embeddings, this method enhances model performance and adaptability across diverse tasks. In summary, structured prompt tuning represents a promising advancement in NLP research, offering a practical solution for improving model efficiency and performance on challenging tasks like those included in the GLUE benchmark. The integration of soft prompts through a hypernetwork demonstrates the potential for further advancements in this area of study.