# Let's Trace It: Fine-Grained Serverless Benchmarking using Synchronous and Asynchronous Orchestrated Applications

**Authors:** Joel Scheuner, Simon Eismann, Sacheendra Talluri, Erwin van Eyk, Cristina Abad, Philipp Leitner, Alexandru Iosup  
**arXiv:** [2205.07696](https://arxiv.org/abs/2205.07696)  
**Date:** 2022-05-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.07696v1/

## Key points

- A serverless benchmark that provides performance information at the application level is necessary
- Existing approaches do not sufficiently cover the diversity of production-ready serverless applications, particularly in terms of external services
- Application-level benchmarks have proven useful in related fields like container-based microservices and scale-out workloads
- ServiBench is proposed as a comprehensive benchmark suite for serverless applications
- ServiBench includes 10 realistic open-source applications that cover different forms of orchestration, triggers, programming models, and external service dependencies
- The suite leverages synchronous and asynchronous serverless applications representative of production usage and extrapolates cloud-provider data to generate realistic workloads
- Comprehensive end-to-end experiments are conducted to capture application-level performance using distributed serverless tracing
- Sharing tools and data for serverless benchmarking in a FAIR manner (findable, accessible, interoperable, reusable) is emphasized
- Releasing software, data, and results adhering to FAIR principles is essential in science and engineering
- ServiBench is presented as a solution to the challenges of application-level serverless benchmarking

## Summary

In this paper, the authors argue that a serverless benchmark that provides performance information at the application level is necessary. They highlight two main reasons for this argument. First, there are already a variety of production-ready serverless applications with distinctive performance profiles, but existing approaches do not sufficiently cover this diversity, particularly in terms of external services. Second, application-level benchmarks have proven useful in related fields like container-based microservices and scale-out workloads. To address these challenges, the authors propose ServiBench as a comprehensive benchmark suite for serverless applications. ServiBench includes 10 realistic open-source applications that cover different forms of orchestration, synchronous and asynchronous triggers, and real-world characteristics such as programming models and external service dependencies. The suite leverages synchronous and asynchronous serverless applications representative of production usage and extrapolates cloud-provider data to generate realistic workloads. It conducts comprehensive end-to-end experiments to capture application-level performance and analyzes results using a novel approach based on distributed serverless tracing. The authors also emphasize the importance of sharing tools and data for serverless benchmarking in a FAIR (findable, accessible, interoperable, reusable) manner. Releasing software, data, and results in packages that adhere to FAIR principles is essential in science and engineering. Overall, this paper presents ServiBench as a solution to the challenges of application-level serverless benchmarking. The authors highlight the need for such a benchmark, describe the challenges involved, and outline their fourfold contribution in addressing these challenges.