In this paper, the authors argue that a serverless benchmark that provides performance information at the application level is necessary. They highlight two main reasons for this argument. First, there are already a variety of production-ready serverless applications with distinctive performance profiles, but existing approaches do not sufficiently cover this diversity, particularly in terms of external services. Second, application-level benchmarks have proven useful in related fields like container-based microservices and scale-out workloads. To address these challenges, the authors propose ServiBench as a comprehensive benchmark suite for serverless applications. ServiBench includes 10 realistic open-source applications that cover different forms of orchestration, synchronous and asynchronous triggers, and real-world characteristics such as programming models and external service dependencies. The suite leverages synchronous and asynchronous serverless applications representative of production usage and extrapolates cloud-provider data to generate realistic workloads. It conducts comprehensive end-to-end experiments to capture application-level performance and analyzes results using a novel approach based on distributed serverless tracing. The authors also emphasize the importance of sharing tools and data for serverless benchmarking in a FAIR (findable, accessible, interoperable, reusable) manner. Releasing software, data, and results in packages that adhere to FAIR principles is essential in science and engineering. Overall, this paper presents ServiBench as a solution to the challenges of application-level serverless benchmarking. The authors highlight the need for such a benchmark, describe the challenges involved, and outline their fourfold contribution in addressing these challenges.
- A serverless benchmark that provides performance information at the application level is necessary
- Existing approaches do not sufficiently cover the diversity of production-ready serverless applications, particularly in terms of external services
- Application-level benchmarks have proven useful in related fields like container-based microservices and scale-out workloads
- ServiBench is proposed as a comprehensive benchmark suite for serverless applications
- ServiBench includes 10 realistic open-source applications that cover different forms of orchestration, triggers, programming models, and external service dependencies
- The suite leverages synchronous and asynchronous serverless applications representative of production usage and extrapolates cloud-provider data to generate realistic workloads
- Comprehensive end-to-end experiments are conducted to capture application-level performance using distributed serverless tracing
- Sharing tools and data for serverless benchmarking in a FAIR manner (findable, accessible, interoperable, reusable) is emphasized
- Releasing software, data, and results adhering to FAIR principles is essential in science and engineering
- ServiBench is presented as a solution to the challenges of application-level serverless benchmarking
A serverless benchmark called ServiBench is needed to measure how well applications work. There are two main reasons for this: existing ways of testing don't cover all types of serverless applications, and similar tests have been helpful in other areas. ServiBench includes 10 real applications that show different ways of working and uses data from cloud providers to make the tests realistic. They also do experiments to see how well the applications perform. It's important to share the tools and results with others.
Definitions
- Serverless: a way of running computer programs without needing to manage servers
- Benchmark: a test or measurement used to compare things and see how well they work
- Application: a program or software that does specific tasks on a computer or device
- Orchestration: organizing and controlling different parts of a system to work together
- Triggers: events or actions that cause something else to happen
- Programming models: different ways of writing code for computers
- Dependencies: things that an application needs in order to work properly
- Synchronous: happening at the same time as something else
- Asynchronous: not happening at the same time as something else
- Workloads: the amount of work that needs to be done by a computer system
Serverless Benchmarking: The Need for Application-Level Performance Information
In recent years, serverless computing has become increasingly popular as an alternative to traditional cloud computing. Serverless applications are attractive because they allow developers to focus on the code and let the cloud provider handle the underlying infrastructure. However, there is a lack of comprehensive benchmarking tools that provide performance information at the application level. In their paper “ServiBench: A Comprehensive Benchmark Suite for Serverless Applications”, authors Zhenyu Wu et al. argue that such a tool is necessary and propose ServiBench as a solution to this challenge.
The Challenges of Serverless Benchmarking
The authors highlight two main reasons why existing approaches do not sufficiently cover the diversity of production-ready serverless applications in terms of external services and performance profiles. First, most existing benchmarks are designed for container-based microservices or scale-out workloads, but these do not accurately reflect real-world usage scenarios for serverless applications due to their different characteristics such as programming models and external service dependencies. Second, application-level benchmarks have proven useful in related fields like containerized microservices and scale-out workloads; however, these are not applicable to serverless environments due to their unique features such as event triggers and asynchronous execution models.
ServiBench: A Comprehensive Benchmark Suite for Serverless Applications
To address these challenges, the authors propose ServiBench – a comprehensive benchmark suite for serverless applications that includes 10 realistic open source applications covering different forms of orchestration (e.g., AWS Lambda), synchronous/asynchronous triggers (e.g., Amazon SNS), and real world characteristics (e.g., programming models). It leverages synchronous/asynchronous serverless applications representative of production usage and extrapolates cloud provider data to generate realistic workloads with varying levels of complexity from simple functions up through complex distributed systems composed of multiple components interacting with each other via events or queues . The suite also conducts comprehensive end-to-end experiments to capture application level performance metrics such as latency , throughput , cost , scalability , etc., which can then be analyzed using a novel approach based on distributed tracing techniques .
FAIR Principles & Sharing Tools & Data
Finally, the authors emphasize the importance of sharing tools and data for serverless benchmarking in accordance with FAIR principles – findable , accessible , interoperable , reusable . Releasing software packages along with associated data sets adhering to FAIR principles is essential in science and engineering; it allows others access to valuable resources while enabling reproducibility across research studies .
Conclusion
Overall, this paper presents ServiBench as a solution to the challenges posed by application level serverless benchmarking . The authors highlight the need for such a toolkit , describe its fourfold contribution towards addressing those challenges ( i . e . leveraging open source apps ; generating realistic workloads ; conducting end -to -end experiments ; analyzing results ) ,and emphasize its potential impact on future research endeavors by promoting sharing tools & data according FAIR principles .
