The ability to answer complex questions that require making latent decisions is a challenging task, particularly when limited supervision is available. Recent works have utilized large language models (LMs) to perform complex question answering in a few-shot setting by outputting intermediate rationalizations while solving the complex question in a single pass. However, this approach requires manually written intermediate supervision, which can be expensive to collect. To address this issue, researchers have introduced "Successive Prompting," where they iteratively break down a complex task into simple tasks and solve them until they arrive at the final solution. This approach decouples the supervision for decomposing complex questions from the supervision for answering simple questions, allowing for multiple opportunities to query in-context examples at each reasoning step and learn question decomposition separately from question answering using synthetic data. There are three types of model outputs: intermediate questions qk, intermediate answers ak, and the final answer y. The first type of output is referred to as question decomposition (QD), while the second type is known as question answering (QA). Final answer prediction is treated as a special case of QD where the model decides that no more decomposition is necessary and outputs a final answer. The model iteratively alternates between QD and QA until it terminates. Successive prompting can also be used with model fine-tuning, where each intermediate output serves as training examples for LMs. The researchers generated a synthetic dataset that can bootstrap a model's ability to decompose and answer intermediate questions. Their best model achieved an improvement of approximately 5% absolute F1 on a few-shot version of the DROP dataset compared with state-of-the-art models with similar supervision. Overall, Successive Prompting provides an effective way to decompose complex questions into simpler ones while learning how to answer them separately using large language models or bespoke components when necessary.
- Complex question answering with limited supervision is challenging
- Large language models (LMs) can perform complex question answering in a few-shot setting
- Previous approach requires manually written intermediate supervision, which is expensive to collect
- Successive Prompting breaks down complex tasks into simple tasks and solves them iteratively
- Decouples supervision for decomposing complex questions from the supervision for answering simple questions
- Three types of model outputs: intermediate questions qk, intermediate answers ak, and the final answer y
- Model alternates between QD and QA until it terminates
- Successive prompting can be used with model fine-tuning using synthetic data
- Best model achieved an improvement of approximately 5% absolute F1 on a few-shot version of the DROP dataset compared with state-of-the-art models with similar supervision.
"Scientists are trying to teach computers how to answer complex questions with limited help. They use big computer programs called Large Language Models (LMs) to do this. Before, people had to give the computer lots of help, which was hard and expensive. Now, they use a method called Successive Prompting that breaks down hard questions into easier ones for the computer to solve step by step. The computer gives three types of answers: intermediate questions, intermediate answers, and the final answer. This method helps the computer learn better and faster."
Definitions- Complex question answering: when a machine tries to understand and answer difficult questions
- Limited supervision: when a machine is given only a little bit of help or guidance
- Large language models (LMs): big computer programs that can understand and interpret human language
- Intermediate supervision: giving the machine guidance or feedback during its learning process
- Successive Prompting: breaking down complex tasks into simpler ones for easier understanding
Breaking Down Complex Questions with Successive Prompting
Solving complex questions that require making decisions is a difficult task, especially when limited supervision is available. Recent works have used large language models (LMs) to answer complex questions in a few-shot setting by outputting intermediate rationalizations while solving the question in one pass. However, this approach requires manually written intermediate supervision which can be costly to obtain. To address this issue, researchers have introduced "Successive Prompting," where they break down a complex task into simpler tasks and solve them until they reach the final solution.
What is Successive Prompting?
Successive prompting decouples the supervision for decomposing complex questions from the supervision for answering simple questions, allowing multiple opportunities to query in-context examples at each reasoning step and learn question decomposition separately from question answering using synthetic data. The model outputs three types of information: intermediate questions qk, intermediate answers ak, and the final answer y. Question decomposition (QD) refers to the first type of output while question answering (QA) refers to the second type of output. Final answer prediction is treated as a special case of QD where no more decomposition is necessary and an answer is provided instead. The model iterates between QD and QA until it terminates.
Model Fine-Tuning with Synthetic Data
Successive prompting can also be used with model fine-tuning where each intermediate output serves as training examples for LMs. Researchers generated a synthetic dataset that can bootstrap a model's ability to decompose and answer intermediate questions accurately without manual intervention or expensive annotation costs associated with supervised learning methods like human labeling or crowdsourcing services like Amazon Mechanical Turk (AMT). Their best model achieved an improvement of approximately 5% absolute F1 on a few-shot version of the DROP dataset compared with state-of-the-art models using similar supervision techniques such as AMT or human labeling approaches which are both expensive methods for obtaining data annotations.
Conclusion
In conclusion, successive prompting provides an effective way to break down complex questions into simpler ones while learning how to answer them separately using large language models or bespoke components when necessary without having to rely on expensive annotation costs associated with supervised learning methods like human labeling or crowdsourcing services like Amazon Mechanical Turk (AMT). This technique has been shown to improve accuracy on datasets such as DROP by up to 5% absolute F1 score compared with existing state-of-the art models utilizing similar techniques such as AMT or human labeling approaches
