When an output worker is ready to send data, it requests a batch of events from the internal queue. The size of this request is controlled by bulk_max_size, an important output tuning parameter. If bulk_max_size is 100, then the queue will try to provide 100 events for the output worker to send.
The queue also has a flush.timeout parameter. When this is zero, the queue will return the events immediately, even if it doesn’t have enough. In our example, if 100 events are requested but there are only 50 in the queue, then the output worker will get 50 events. But when the flush timeout is positive, the queue will instead wait for up to the specified timeout to collect more events.
But hold on: let’s say we set a flush timeout of 5 seconds and request 100 events. You might expect that the queue will return 100 events immediately if it has them or else it will delay for up to 5 seconds trying to get to 100. And starting in version 8.13, you’d be right. But that isn’t how the old queue behaved! The queue didn’t wait to fill an output request — it waited to fill an internal queue buffer, which could be a completely different size.
The size of the internal queue buffers was controlled by flush.min_events, a parameter that looks very similar to bulk_max_size and is often confused for it but can have very different effects.
Leave a Reply