Comparing OpenSearch Managed Services | Logz.io

In March of 2022, Elastic decided to close source the most popular log management and analytics solution in the world: the ELK Stack. Millions chose ELK as their logging platform and made it the heart of their troubleshooting operations because it was open source. And suddenly, it wasn’t – leaving many looking for other options.

Shortly after, AWS launched OpenSearch and OpenSearch Dashboards as open source alternatives to Elasticsearch and Kibana, respectively.

Since then, the OpenSearch community has taken off. Licensed under Apache 2.0, contributions are welcome from anyone. There are already additional OpenSearch features not offered in the free versions of Elasticsearch and Kibana, including RBAC, ML-powered anomaly detection, and other security features.

There is no OpenSearch equivalent of Logstash because it’s heavy and burdensome. Logstash can be replaced by technologies like Fluentd, which can handle log processing capabilities with a much lighter computing footprint.

Like Elasticsearch and Kibana, OpenSearch and OpenSearch Dashboards are used together to store and analyze log data. Since its inception, companies like AWS, Aiven, and Logz.io have built managed services around OpenSearch to make it easier to use at scale.

Before we start comparing these managed services, we should start with the question…why bother?

Why bother with a Managed OpenSearch Service?

OpenSearch is a flexible and powerful open source logging tool. It seamlessly balances the load of log data across nodes as data volumes fluctuate, it can easily integrate with modern cloud environments, and it’s highly familiar among ELK users (in its infancy, the OpenSearch and OpenSearch Dashboards user’s experience is similar to ELK).

So why pay for a great open source tool?

In small deployments (meaning maybe a few nodes in your OpenSearch cluster) and low stakes projects (maybe a logging solution for your staging environment), OpenSearch is relatively easy to install and maintain. It won’t require much maintenance to keep the OpenSearch data pipeline up and running.

However, as your log data volumes grow, you’ll need to manage larger clusters, implement a queuing system like Kafka, upgrade more components, shard your indices more often, monitor your cluster performance, implement more log parsing…the list goes on. 

Failing to properly execute these tasks could result in slow queries, dropped data, or a complete crash.

Obviously, in high stakes situations (where you may depend on your log data to troubleshoot customer-facing services), those outcomes are unacceptable. And the time needed to prevent them can distract engineers from their core business.

That’s why managed OpenSearch services exist. A managed service can reduce the amount of time and expertise needed to maintain a large OpenSearch cluster and data pipeline, while ensuring higher availability and better performance.

Let’s compare some of the more popular managed OpenSearch services.

AWS Managed OpenSearch Service

AWS OpenSearch Service is one of many AWS services that simplify adoption, deployment, and management of popular open source technologies. Others include AWS Prometheus Service, AWS Grafana Service, AWS Kafka Service, and more.

AWS states that by spinning up an AWS OpenSearch Service cluster, the users avoid many of the manual configurations needed to set up your own OpenSearch cluster. According to AWS, they manage “software installation, upgrades, patching, scaling (up to 3 PB), and cross-region replication with no downtime.”

In addition to OpenSearch, AWS also hosts OpenSearch Dashboards instances for the customer to analyze the data stored in OpenSearch.

To manage costs, customers can choose from hot, UltraWarm, and cold tiers – like other similar solutions, the tradeoff of lower costs is poor query performance.

While AWS OpenSearch Service can simplify deployment and management for OpenSearch, AWS doesn’t manage the integrity of the entire data pipeline, including ingestion, queuing, or processing components.

This means AWS OpenSearch users are still on the hook to maintain the pipeline’s performance and reliability. If something breaks and the log data stops reaching OpenSearch Dashboards, it’s up to the customer to find the problem and fix it. This can be challenging when there is no log data to troubleshoot the problem!

Since high log volumes can create log pipeline/cluster issues (like overwhelmed data ingestion) and require manual maintenance tasks (like index sharding), AWS OpenSearch may be the best solution for logging projects that don’t require huge data volumes and clusters.

Aiven for OpenSearch

Aiven for OpenSearch is one of Aiven’s many managed data infrastructure offerings. They host the infrastructure for popular open source data technologies like Kafka, Flink, M3DB, Redis, PostgreSQL, and others. From a single developer portal, Aiven users can manage their data infrastructure deployments in one place.

As the OpenSearch community continues to add additional features, Aiven states that they ensure quick turnaround time so their customers can upgrade their clusters and access the most recent components.

In addition to hosting these technologies, they also participate and contribute to these open source communities via their Open Source Program Office.

Aiven for OpenSearch includes OpenSearch to store the data, OpenSearch Dashboards to analyze and monitor the data, and a wide variety of OpenSearch plugins that extend OpenSearch capabilities – including alerting, trace analytics, ML-powered anomaly detection, and more. 

Like AWS Managed OpenSearch Service, customers are responsible for parsing the data so it can be easily searched and visualized by the end user. This can be done with data collection and processing tools like Fluentd.

While Aiven simplifies deployment and management for these common data infrastructure technologies, customers remain responsible for monitoring the health and performance of their deployments on Aiven, including their managed OpenSearch service. 

According to Aiven documentation, for example, customers may need to access their logs to debug query performance or to inspect errors caused by a specific workload. This data is available through the Aiven Console.

Aiven for OpenSearch may be a great solution for those who want a simplified and reliable way to begin logging with OpenSearch and don’t mind monitoring their log data pipelines or parsing their data. Plus, Aiven offers a single place to manage not only OpenSearch instances, but other data infrastructure components like Kafka, Redis, and others.

Opster

Opster offers a solution that is a bit different from the ones above, however,it helps accomplish a similar goal: to simplify OpenSearch deployment and management, while improving OpenSearch reliability. It does the same for Elasticsearch.

Rather than fully managing and running the OpenSearch deployment for the user, according to Opster, their  ‘AutoOps’ product and support team can help you run a better performing, more reliable, and more cost efficient OpenSearch, while using less engineering resources.

AutoOps can automate tasks like: incident prevention, incident resolution, performance optimization, and cost monitoring/optimization. 

Opster states that their support team continuously monitors your OpenSearch clusters to highlight issues and suggestion remediation strategies. Opster support handles tasks like: cluster design and capacity planning, optimizing OpenSearch mapping, and support for issues related to infrastructure.

Opster can also be used to provision and manage OpenSearch clusters in your infrastructure.

Opster may be a great solution for those who still want to run their own OpenSearch clusters, but need additional assistance to boost performance, reduce hardware costs, and improve OpenSearch reliability.

Logz.io

A key difference between Logz.io and the rest of the services above is that OpenSearch is delivered via SaaS.

This means that Logz.io assumes the entire responsibility of the data pipeline – from ingestion, to scaling, to the OpenSearch cluster, to the analysis interface. If the log volumes spike, it’s up to Logz.io to adjust the infrastructure according to the load. It’s also on Logz.io to ensure the health and performance of the entire data pipeline. 

To get logs parsed, users can leverage Logz.io’s parsing-as-a-service to get it done in minutes.

Another key difference is how Logz.io enhances OpenSearch and OpenSearch Dashboards. Rather than leveraging OpenSearch plugins, Logz.io builds its own capabilities within the interface that are delivered immediately to customers. Examples include RBAC, data filtering to reduce costs, alerting, and ML-powered log analysis to surface critical errors and exceptions.

Finally, Logz.io is the only platform to unify log data with OpenSearch, metric data with Prometheus (stored in M3DB), and traces with Jaeger (a popular open source tool for tracing) – and by doing so, it provides full observability in one place.

Logz.io is a great option for those who want a reliable, out-of-the-box, and zero-maintenance OpenSearch experience that unifies telemetry data in a single interface.

Is an OpenSearch Managed Service right for you?

As OpenSearch popularity grows, more engineers and IT Ops professionals will realize the challenges of scaling it themselves, and many will alleviate those challenges with managed OpenSearch services. Others will have no trouble at all scaling their own OpenSearch clusters.

OpenSearch Managed Services depend on your resources at hand. If you have the time and resources to manage your own OpenSearch pipeline, why pay for it? If you think your resources would be better spent on other initiatives, an OpenSearch Managed Service could be a great option.

The best OpenSearch service for you depends on your priorities. Your decision will probably be impacted by factors like cost, available engineering resources, troubleshooting requirements, and other influencers.