The Importance of Batch Size and Batch Time

Overview

Besides --total-time, --batch-size and --batch-time are the two most important arguments for configuring and optimizing test execution. Batch size directly impacts the number of concurrent requests executed, and batch time determines the amount of time all requests in a given batch have to either successfully complete or fail. Hedra does not return or calculate results for actions that fail to return once a batch’s time has been reached - such actions are simply discarded in order to preserve execution efficiency and provide clear insight into the performance of the framework itself.

The challenge comes in determining the balance of batch size/time that will generate the best test results. Too small a batch size will limit the load the framework places on the target, while too large may result in inaccurate response timings or rate-limiting by the target. To small of batch time will result in batches of actions not being terminated prematurely (thus unnecessarily limiting concurrency) while too large a batch time can result in long-running requests trying up the event loop (preventing future batches from executing).

There is no exact science to determining these values - however Hedra offers the Optimization stage as a means of both automating this discovery and providing suggestions to users and to good potential candidates. In general, for HTTP/HTTPS/Websocket/GraphQL/GRPC engines we recommend starting with a batch size of a few thousand and a batch time of a few seconds. For Playwright, we recommend a batch size of a few dozen and a batch time of five to ten seconds (to account for animations, page loading, etc.).

---

Under-Utilization vs. Over-Utilization

Hedra places no limits on the batch sizes you can set, though batch time has a maximum of 60 seconds. However, setting the batch size too low or too high can drastically reduce Hedra’s performance. This section aims to help you identify ideal batch size/ batch time settings.

In general, ideal batch size for single-threaded tests on modern computers ranges between 4000-8000 with a batch time of 3-5 seconds. This allows batches to quickly complete while maximizing concurrency and minimizing measurement error. For multi-threaded tests (using the parallel runner), we generally recommend reducing batch size by 20-30% per two threads added. So (for example) if we wanted to parallelize a test whose batch size was initially specified as 8000, we would reduce the batch size to 5600-6400 for two threads and 4000