Observability For Java Spring Based Applications Using Opentelemetry

Project demonstrating Complete Observability Stack utilizing Prometheus, Loki (For distributed logging), Tempo (For Distributed tracing, this basically uses Jaeger Internally), Grafana for Java/spring based applications (With OpenTelemetry auto / manual Instrumentation) involving multiple microservices with DB interactions

https://github.com/mnadeem/boot-opentelemetry-tempo

Demo

Clone the project and run the following commands

mvn clean package docker:build

docker-compose up

Access the endpoint

View the log and trace in Grafana

Get the trace information Using Jaeger as well

Basic Trace

Complex Trace

Opentelemetry Components

Library	Purpose
opentelemetry-proto
opentelemetry-api
opentelemetry-context
opentelemetry-exporter-jaeger
opentelemetry-sdk
opentelemetry-sdk-trace

Opentelemetry is still in early stages when it comes to metrics export, hence we would be using promeheus, and spring boot support for prometheus for metrics

There is no parallel to Grafana Loki for distributed logging.

docker-compose.yaml file would take care of the following things

Basic Case

Complex Case

Enabling Auto Instrumentation

pom.xml

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-actuator</artifactId>
		</dependency>
		
		<dependency>
			<groupId>io.micrometer</groupId>
			<artifactId>micrometer-registry-prometheus</artifactId>
			<scope>runtime</scope>
		</dependency>

Dockerfile

Refer the docker project

ENV JAVA_OPTS "${JAVA_OPTS} \
  -Dotel.trace.exporter=jaeger \
  -Dotel.exporter.jaeger.endpoint=tempo:14250 \
  -Dotel.metrics.exporter=none \
  -Dotel.resource.attributes="service.name=${APP_NAME}" \
  -Dotel.javaagent.debug=false \
  -javaagent:${APP_HOME}/${OTEL_AGENT_JAR_FILE}"

application.properties

# JMX
management.endpoints.web.exposure.include=*
management.endpoints.web.exposure.include=prometheus,health,info,metric

management.health.probes.enabled=true
management.endpoint.health.show-details=always

Manual Instrumentation

pom.xml

       <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-api</artifactId>
        </dependency>
         
         <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-extension-annotations</artifactId>
        </dependency>

    public List<Flight> getFlights(String origin) {
		LOGGER.info("Getting flights for {}", origin);
		List<Flight> flights = this.flightClient.getFlights(origin);
		doSomeWorkNewSpan();
		return flights;
    }

    @WithSpan
    private void doSomeWorkNewSpan() {
		LOGGER.info("Doing some work In New span");
        Span span = Span.current();
 
        span.setAttribute("attribute.a2", "some value");
 
        span.addEvent("app.processing2.start", atttributes("321"));
        span.addEvent("app.processing2.end", atttributes("321"));
    }
 
    private Attributes atttributes(String id) {
        return Attributes.of(AttributeKey.stringKey("app.id"), id);
    }

References

• Opentelemetry Specification
• Opentelemetry Java Getting Started
• Opentelemetry Java
• Opentelemetry Java Instrumentation
• Opentelemetry Java Contrib
• Opentelemtery @ lightstep
• Opentelemetry Specification V1