Logstash is a powerful tool for ingesting, transforming, and shipping data from various sources. Visibility into Logstash is critical for optimizing performance and troubleshooting issues related to data ingestion. We’ve greatly improved the Logstash integration to display the status of your Logstash nodes and pipelines at a glance. The integration is now powered by Elastic Agent, which queries Logstash monitoring APIs for data that populates managed dashboards.
This blog will guide you through the visualizations available from the integration, how to configure and install them, and the underlying APIs that provide the data. See the health of your Logstash nodes at a glance with the updated integration.
Table of Contents
Monitoring with the Logstash integration
The dashboards available through the integration completely break open the black box. Insights that would take extensive queries are available at your fingertips in the Overview dashboard. Here, you can see what matters at a high level, the number of events Logstash has received, how many it has processed and sent, and how long it took.
For the performance of a single node, the Node Overview dashboard provides details on system and process health. This helps to determine if an issue is localized to an individual host or if it is pervasive throughout a deployment.
Pipelines are where the work in Logstash really occurs, and we have no shortage of insights related to pipeline performance. The Pipeline Overview dashboard provides high-level details on pipeline activity across your entire deployment to quickly surface problems and assess performance. Broken down by pipeline, you can see the average time an event takes to process, which is how long it takes pushing to queue. Items that are outside the allotted tolerance are bolded in red to quickly attract attention and direct users to take action.
As you drill down into individual pipelines, you’ll see details on worker utilization and processing over time. This is invaluable in determining the cause of problems. Isolating spikes in time makes it easier to correlate with configuration changes or external events.
But you shouldn’t stop at pipelines. A huge benefit of using Logstash is the vast ecosystem of plugins. That strength does introduce variability, which can make troubleshooting more difficult. Thankfully, the integration addresses that problem by providing plugin details by type, enabling users to drill down further.
See what’s new
The health report dashboards are the newest addition, and the amount of information they can quickly convey will make every Logstash operator’s life easier. One quick look is all you need to see if there are problems with your Logstash deployment.
Pipeline health provides more detailed data as well as actionable insights on troubleshooting steps.
Logstash integration with Elastic Agent
Installing the integration can be done in minutes, and the data you get will save you hours of troubleshooting. Here are the steps to install the integration and deploy agents to your Logstash instances that will collect and transform the data from Logstash monitoring APIs.
1. Add the Logstash integration to monitor your deployment
From the integrations page, search for Logstash. Click Add Logstash, and you’ll be guided through the configuration options on how an Elastic Agent can collect monitoring data from your instance.
Be sure to select Metrics (Elastic Agent) to get the most data and our newest dashboards. You can specify what data to collect and the API polling intervals or stick with our defaults. Health Reporting is disabled by default, but you’ll want to enable it to get the most out of the integration. If you have modified your Logstash configuration to listen on a different port, specify it here.
2. Configure and install Elastic Agents on Logstash nodes
You will likely be deploying this to new agents. Create a name that is meaningful, and then save and continue. If you already have agents monitoring your Logstash deployment, you can add this policy to existing hosts.
You will be prompted to add Elastic Agent to your Logstash nodes.
Choose Add Elastic Agent to your hosts, and you’ll be guided through Fleet enrollment, agent installation, and verification.
Once you’ve verified that agents are communicating with Fleet, the data will start flowing in and populate the dashboards.
Data available through Logstash monitoring APIs
Logstash exposes several monitoring APIs that give extensive visibility into single instances. The Logstash integration uses these APIs with the Elastic Agent to parse and deliver these data to your monitoring cluster. Customers that wish to create a custom integration for Logstash monitoring can query these APIs according to their requirements.
Node Stats
Provides detailed information at a node and pipeline level for a wide range of resources, events, and utilization. In version 8.5, we added an additional data type — Flow Stats — which includes derivative measures of performance, including throughput, backpressure, and worker concurrency.
Health Report
New in 8.16, the Logstash health report returns a color coded health status for your Logstash instance based on various indicators, such as pipeline status and worker utilization. The indicators include symptom details. You can also view unhealthy results, which include potential impacts, diagnoses, and suggested actions to address the problem.
Get started today
Ready to get started? Explore now in a free 14-day trial on Elastic Cloud — the hosted Elasticsearch service that includes all of the latest features.
The release and timing of any features or functionality described in this post remain at Elastic’s sole discretion. Any features or functionality not currently available may not be delivered on time or at all.
Leave a Reply