Table of Contents
Troubleshooting
There are some common problems you might run into when creating your plugin:
Still experimental?
The OpenTelemetry bridge was added in Elastic APM Java Agent version 1.30.0 — so that is the earliest version you can use this plugin mechanism with — and it was initially added as experimental technology. Depending on which version you are using, you may need to have enable_experimental_instrumentations set. You can check this is on by looking for the following log output from the Elastic APM Java Agent:
INFO co.elastic.apm.agent.configuration.StartupInfo - enable_experimental_instrumentations: 'true'
Correct plugins dir?
If you are using the plugins_dir option, is it being picked up correctly and do you have the jar you created present in that directory? Look for the following log output from the Elastic APM Java Agent:
INFO co.elastic.apm.agent.configuration.StartupInfo - plugins_dir: '...
Jar being loaded?
Check to ensure the plugin jar is being loaded. Look for the following log output from the Elastic APM Java Agent:
INFO co.elastic.apm.agent.bci.ElasticApmAgent - Loading plugin ...
Classes compiled to a more recent version than the JDK?
If you have compiled the classes to a JDK version that’s more advanced than the JVM you are running, you’ll likely get loading errors. A typical thing to look out for after turning the log level to DEBUG are lines like:
DEBUG co.elastic.apm.agent.util.DependencyInjectingServiceLoader - Skipping … because it only applies to more recent Java versions and the JVM running this app is too old to load it.
Class loading errors or dependencies missing
Remember that you need all the dependencies in the plugin jar — if some are missing, there will be errors. Make sure you have turned the log level to DEBUG, and search through the output for errors. For example, missing the OpenTelemetry dependency produces this output:
java.lang.Error: Unresolved compilation problems:
Tracer cannot be resolved to a type
GlobalOpenTelemetry cannot be resolved
Span cannot be resolved to a type
Other errors vs. what success looks like
Make sure you have turned the log level to DEBUG. The output when the plugin is successfully instrumenting should include lines like:
DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation YOUR_PLUGIN_INSTRUMENTATION_CLASS
DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation YOUR_PLUGIN_INSTRUMENTATION_CLASS: … matches …
DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation YOUR_PLUGIN_INSTRUMENTATION_CLASS: …
And in particular, you should see the instrumentation working, with output for the tracing you’ve added like:
DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to " TRACE_ID (OBJECT_ID) (REFERENCE_COUNT)
DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction " TRACE_ID (OBJECT_ID) (REFERENCE_COUNT)
DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to 'SPAN NAME' TRACE_ID (OBJECT_ID) (REFERENCE_COUNT)
DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - endTransaction 'SPAN NAME' TRACE_ID (OBJECT_ID) (REFERENCE_COUNT)
DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'SPAN NAME' TRACE_ID (OBJECT_ID) (REFERENCE_COUNT)
Leave a Reply