What’s New with Fluentd & Fluent Bit

At the recent KubeCon EU, we learned the significant news of the FluentBit v2.0 major release with numerous new features. What’s new and what’s to come for this key log aggregation tool?

On the latest OpenObservability Talks, I hosted Eduardo Silva, one of the maintainers of Fluentd, a creator of Fluent Bit and co-founder of Calyptia. We delved into the significant updates and features of Fluent Bit v2.0, the current state of the Fluentd project, and the thriving open-source community behind these powerful observability data collection and processing tools.

Fluent Bit and Fluentd have been essential tools for log aggregation and processing, and are expanding to additional observability data types. They aid users in managing and analyzing the vast amounts of data generated by modern applications. With his expertise and insights as a maintainer, Eduardo shed light on the evolution and value of these projects.

Introduction and Background of Fluentd and Fluent Bit

Eduardo started the conversation by introducing Fluentd and Fluent Bit, highlighting their popularity and widespread adoption in the industry. Fluentd is a veteran project, having graduated from the Cloud Native Computing Foundation (CNCF), with over 12,000 stars on GitHub and more than 1,300 forks.

Fluentd was initially created to solve the challenge of centralizing and processing log data generated by multiple diverse sources in a distributed system landscape. This became crucial as technologies like Kubernetes and Docker gained popularity.

Fluentd was born in 2011 at Treasure Data with the aim of ingesting data into the platform and enabling analysis. Fluentd was written in Ruby, and offered a pluggable architecture, which allowed for easy integration of various data sources and destinations.

Eduardo shared an interesting anecdote from his days at Treasure Data, that the Ruby version was supposed to only be the proof of concept, and the eventual tool should have been rewritten in the C language. But the Ruby version persisted, and turned out to be a major advantage for community adoption.

The open source community around Fluentd has created and maintained a rich and active ecosystem of plugins, which are available on Rubygems, to deal with data from multiple sources, which is one of Fluentd’s strong points.

Fluent Bit’s Role in IoT and Beyond

Fluent Bit was originally created to cater to the demands of IoT devices and Embedded Linux environments. As IoT gained traction, the need to analyze log data generated by connected devices became evident.

However, Fluentd wasn’t suitable for constrained IoT environments, leading to the birth of Fluent Bit. Fluent Bit was written in C (unlike Fluentd which was written in Ruby), and offered a more lightweight and performant solution. It paved the way for efficient data collection, processing, and transmission in resource-constrained environments, which aligned with the explosion of cloud-native computing.

Fluentd and Fluent Bit coexist in the same family of projects, providing solutions for different use cases. While Fluentd is the parent project, Fluent Bit addresses cloud-native needs and IoT scenarios.

Over time, Fluent Bit’s performance improvements and versatility led to major cloud providers like Google, Microsoft, and AWS adopting it, making it the preferred choice for many users.

Expanding Capabilities with Fluent Bit v2.0

Fluent Bit v2.0 marks a major release with significant enhancements and new features. The conversation shifted to the innovations introduced in this release, with a focus on expanding support for various telemetry data types.

Going beyond log data, Fluent Bit now supports native metrics payloads, enabling integration with Prometheus and OpenTelemetry. This aligns with the growing demands of cloud-native and distributed systems, where collecting, processing, and analyzing various types of telemetry data is essential.

Eduardo highlighted the new Kubernetes Events plugin in Fluent Bit, which enables users to collect Kubernetes events for analysis and monitoring. This plugin extends Fluent Bit’s capabilities within Kubernetes environments, complementing its existing features like collecting logs and metrics with labels and annotations.

The conversation also touched on the vibrant community behind Fluent Bit and Fluentd. Eduardo mentioned that the community around Fluentd is vibrant and keeps building and maintaining the rich Ruby plugins. He also said that admittedly it’s not as easy to contribute code to Fluent Bit’s C codebase, but while not everyone is directly contributing code, the community’s feedback, bug reports, and user experiences play a crucial role in shaping the projects.

Looking Ahead for Fluent Bit

We concluded by discussing the future of Fluentd and Fluent Bit. The Calyptia team is focused on Fluent Bit, pushing continuous innovation, scalability, and performance improvements together with the community. They plan to invest more in extending connectors, addressing Windows environments and enhancing its new distributed tracing data processing.

Want to learn more? Check out the OpenObservability Talks latest episode: What’s New with Fluentd & Fluent Bit.