Trace-based testing with Elastic APM and Tracetest

This post was originally published on the Tracetest blog.

Want to run trace-based tests with Elastic APM? Today is your lucky day. We’re happy to announce that Tracetest now integrates with Elastic Observability APM.

Check out this hands-on example of how Tracetest works with Elastic Observability APM and OpenTelemetry!

Tracetest is a CNCF project aiming to provide a solution for deep integration and system testing by leveraging the rich data in distributed system traces. In this blog, we intend to provide an introduction to Tracetest and its capabilities, including how it can be integrated with Elastic Application Performance Monitoring and OpenTelemetry to enhance the testing process.

Your good friend distributed tracing

Distributed tracing is a way to understand how a distributed system works by tracking the flow of requests through the system. It can be used for a variety of purposes, such as identifying and fixing performance issues, figuring out what went wrong when an error occurs, and making sure that the system is running smoothly. Here are a few examples of how distributed tracing can be used:

• Monitoring performance: Distributed tracing can help you keep an eye on how your distributed system is performing by showing you what’s happening in real time. This can help you spot and fix problems like bottlenecks or slow response times that can make the system less reliable.
• Finding the source of problems: When something goes wrong, distributed tracing can help you figure out what happened by showing you the sequence of events that led up to the problem. This can help you pinpoint the specific service or component that’s causing the issue and fix it.
• Debugging: Distributed tracing can help you find and fix bugs by giving you detailed information about what’s happening in the system. This can help you understand why certain requests are behaving in unexpected ways and how to fix them.
• Security: Distributed tracing can help you keep an eye on security by showing you who is making requests to the system, where they are coming from, and what services are being accessed.
• Optimization: Distributed tracing can help you optimize the performance of the system by providing insight into how requests are flowing through it, which can help you identify areas that can be made more efficient and reduce the number of requests that need to be handled.

Distributed tracing — Now also for testing

Observability, previously only used in operations, is now being applied in other areas of development, such as testing. This shift has led to the emergence of “Observability-driven development” and “trace-based testing” as new methods for using distributed tracing to test distributed applications. 

Instead of just checking that certain parts of the code are working, trace-driven testing follows the path that a request takes as it goes through the system. This way, you can make sure that the entire system is working properly and that the right output is produced for a given input. By using distributed tracing, developers can record what happens during the test and then use that information to check that everything is working as it should. 

This method of testing can help to find problems that may be hard to detect with other types of testing and can better validate that the new code is working as expected. Additionally, distributed tracing provides information about what is happening during the test, such as how long it takes for a request to be processed and which services are being used, which can help developers understand how the code behaves in a real-world scenario.

Enters Tracetest

Tracetest is a CNCF project that can run tests by verifying new traces against previously created assertions against other traces captured from the real systems. Here’s how you can use Tracetest:

• Capture the baseline good known trace. This will be the golden standard that you will use to write your tests and assertions. Trace-driven development is a better way to test how different parts of the system work together because it allows developers to test the entire process from start to finish, making sure that everything is working as it should and giving a more complete view of how the system is functioning instead of trying to create disjointed assertions validating the request execution.
• Now you can start validating your code changes against good known behavior captured previously.
• Tracetest can validate the resulting traces from the test and see if the system is working as it should. This can help you find problems that traditional testing methods might not catch.
• Create reports: Tracetest can also create reports that summarize the results of the test so that you can share the information with your team.
• Help you validate in production that the new requests follow the known path and run the predefined assertions against them.

The APM tool in Kibana, which is a familiar UI for many developers, can provide extra information when used with Tracetest. The APM tool can show you how the system is performing during the test and help you find issues using the familiar user interface. For example, the APM tool can show you how requests are moving through the system, how long requests take to be processed, and which parts of the system are being used. This information can help you identify and fix problems during testing. 

Furthermore, the APM tool can be set to show you all the data in real-time, which allows you to monitor the system’s behavior during the test or even in production and helps you make sense of what Tracetest is showing.

How Tracetest works with Elastic APM to test the application

The components work together to provide a complete solution for testing distributed systems. The telemetry captured by the OpenTelemetry agent is sent to the Elastic APM Server, which processes and formats the data for indexing in Elasticsearch. The data can then be queried and analyzed using Kibana APM UI, and Tracetest can be used to conduct deep integration and system tests by utilizing the rich data contained in the distributed system trace.

For more details on Elastic’s support for OpenTelelemetry, check out Independence with OpenTelemetry on Elastic.