The popularity of Prometheus speaks for itself. The project doesn’t post official numbers, but there are at least 500,000 companies using this project today as one of the most mature CNCF projects – one that has over 40k Github stars as of the writing of this blog. And since Prometheus is highly interoperable, compatibility is key. This comes into play not only with the exporters, but also with long-term storage options and alerting systems.
In late 2020, Prometheus co-creator Julius Volz began running compatibility tests against PromQL querying and compatibility with Prometheus itself. This is important when people add long term storage or use 3rd party services for storing data. He did so to determine which choices would provide the same compatibility that using Prometheus itself would allow, namely for long-term stores for offload analysis.
Specifically, this effort relates to testing the ability to answer queries using PromQL and the supporting constructs of the language, ensuring compatibility with Prometheus which was later expanded to include additional test concepts including OpenMetrics. Julius and team built a test suite which ran against various backends and services to determine the level of Prometheus compatibility. You can read the results yourself, which showed a variable level of conformance.
Since the last blog was published, Julius and Richard Hartmann also began working on a formal project to turn this into a CNCF authorized program called “Prometheus Conformance” which they launched and outlined in October 2021. Today this initiative includes OpenMetrics, PromQL, and Remote Write, with future plans for expansion as the blog explains.
To hear more about the Prometheus Conformance Program from the people who founded it, check out the explanation by CNCF’s Richard Hartmann at OpenObservability Talks:
The code used to conduct the tests can be found on Github. We ran them against the Logz.io infrastructure monitoring software offering which now exposes a Prometheus query API endpoint for customers to use for various purposes.
Under our service, we are running the open source M3DB database, which is 100% compliant. We’ve also built layers of multi-tenancy on top of it, which is why we utilize specific configurations to pass along the tenants.
reference_target_config:
query_url: 'http://localhost:9090'
#--- configure Logz.io as your remote write https://docs.logz.io/shipping/prometheus-sources/prometheus-remote-write-shipping.html
#--- How to look up your account region https://docs.logz.io/user-guide/accounts/account-region.html#how-to-look-up-your-account-region
#--- Contact our support to enable API on your Prometheus account [email protected]
test_target_config:
query_url: https://{{YOUR_API_HOST}}/v1/metrics/prometheus
headers:
X-API-TOKEN: {{YOUR_API_TOKEN}}
query_tweaks:
- note: 'Logz.io adds a "p8s_logzio_name" label to distinguish Prometheus servers, leading to extra labels in query results. These need to be stripped before comparisons.'
no_bug: true
drop_result_labels:
- p8s_logzio_name
With this config we run ./promql-compliance-tester -config-file=promql-test-queries.yml -config-file=test-logzio.yml and get the following output showing 100% PromQL Compatibility.
This effort is great validation that Logz.io is implementing all of the relevant standards to future proof against evolving Prometheus requirements, allowing you to reuse existing dashboards in Grafana along with any scripts or queries you may be running on Prometheus.
As a huge proponent of this work by the CNCF and Prometheus contributors, it’s our pledge to remain fully compliant with all the relevant conformance standards. We look forward to testing against the forthcoming Remote Write tests and other components.
Kudos to the Prometheus community for advancing this critical work to ensure that vendors are discouraged from making empty promises about Prometheus compatibility, protecting all of our shared interests!
Leave a Reply