Grafana - All-in-one
In this example, we are going to use Prometheus and Jaeger, both displayed inside Grafana. Just like Jaeger example, we will need to set up an Open Telemetry Collector to gather both metrics and traces.
Project setup
Configure the project using your favorite tool, once again setup is similar to the Jaeger example, only difference being that we export metrics as they will be stored inside Prometheus:
Add settings to the build.sbt:
libraryDependencies ++= Seq(
"org.typelevel" %% "otel4s-oteljava" % "0.11.1", // <1>
"io.opentelemetry" % "opentelemetry-exporter-otlp" % "1.44.1" % Runtime, // <2>
"io.opentelemetry" % "opentelemetry-sdk-extension-autoconfigure" % "1.44.1" % Runtime // <3>
)
run / fork := true
javaOptions += "-Dotel.java.global-autoconfigure.enabled=true" // <4>
javaOptions += "-Dotel.service.name=grafana-example" // <5>
javaOptions += "-Dotel.exporter.otlp.endpoint=http://localhost:4317" // <6>Add directives to the grafana.scala:
//> using scala 3.3.0
//> using dep "org.typelevel::otel4s-oteljava:0.11.1" // <1>
//> using dep "io.opentelemetry:opentelemetry-exporter-otlp:1.44.1" // <2>
//> using dep "io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.44.1" // <3>
//> using javaOpt "-Dotel.java.global-autoconfigure.enabled=true" // <4>
//> using javaOpt "-Dotel.service.name=grafana-example" // <5>
//> using javaOpt "-Dotel.exporter.otlp.endpoint=http://localhost:4317" // <6>1) Add the otel4s library
2) Add an OpenTelemetry exporter. Without the exporter, the application will crash
3) Add an OpenTelemetry autoconfigure extension
4) Enable OpenTelemetry SDK autoconfigure mode
5) Add the name of the application to use in the traces
6) Add the OpenTelemetry Collector endpoint
OpenTelemetry SDK configuration
As mentioned above, we use otel.service.name and otel.metrics.exporter system properties to configure the
OpenTelemetry SDK.
The SDK can be configured via environment variables too. Check the full list
of environment variable configurations
for more options.
Observability stack configuration
Use the following docker-compose.yaml file:
version: '3.7'
services:
otel-collector: # receives application metrics and traces via gRPC or HTTP protocol
image: otel/opentelemetry-collector-contrib
command: [--config=/etc/otel-collector-config.yaml]
volumes:
- "./config/otel-collector-config.yaml:/etc/otel-collector-config.yaml"
ports:
- "8888:8888" # Prometheus metrics exposed by the collector
- "8889:8889" # Prometheus exporter metrics
- "4317:4317" # OTLP gRPC receiver
- "4318:4318" # OTLP http receiver
networks:
- static-network
jaeger: # stores traces received from the OpenTelemetry Collector
image: jaegertracing/all-in-one:latest
volumes:
- "./config/jaeger-ui.json:/etc/jaeger/jaeger-ui.json"
command: --query.ui-config /etc/jaeger/jaeger-ui.json
environment:
- METRICS_STORAGE_TYPE=prometheus
- PROMETHEUS_SERVER_URL=http://prometheus:9090
ports:
- "14250:14250"
- "16685:16685" # GRPC
- "16686:16686" # UI
networks:
- static-network
prometheus: # scrapes metrics from the OpenTelemetry Collector
image: prom/prometheus:latest
volumes:
- "./config/prometheus.yml:/etc/prometheus/prometheus.yml"
ports:
- "9090:9090"
networks:
- static-network
grafana: # queries Jaeger and Prometheus to visualize traces and metrics
image: grafana/grafana-oss
restart: unless-stopped
ports:
- "3000:3000"
networks:
- static-network
networks:
static-network:Configuration files
./config/otel-collector-config.yaml
OpenTelemetry Collector configuration: receivers, exporters, and processing pipelines.
receivers:
otlp:
protocols: # enable OpenTelemetry Protocol receiver, both gRPC and HTTP
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
otlp/jaeger: # export received traces to Jaeger
endpoint: jaeger:4317
tls:
insecure: true
prometheus: # run Prometheus exporter server on port 8889, so Prometheus can scrape the metrics
endpoint: 0.0.0.0:8889
send_timestamps: true
processors:
batch:
timeout: 10s
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/jaeger]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]./config/prometheus.yml
Prometheus server configuration: scrape interval and targets.
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus" # scrape prometheus itself to collect the internal metrics (e.g. scrape stats, etc)
static_configs:
- targets: ["localhost:9090"]
- job_name: "otel-collector" # scrape metrics from the OpenTelemetry collector
static_configs:
- targets: ["otel-collector:8889"]./config/jaeger-ui.json
Jaeger configuration: enable Service Performance Monitor (SPM).
{
"monitor": {
"menuEnabled": true
},
"dependencies": {
"menuEnabled": true
}
}Application example
Example service mocking a call to a remote API: here the remote API returns apples or bananas. We're using the metrics to measure the apple/banana ratio returned by the API, and the traces to measure the latency of this API.
import cats.effect.{Async, IO, IOApp}
import cats.effect.std.Random
import cats.syntax.apply._
import cats.syntax.flatMap._
import cats.syntax.functor._
import org.typelevel.otel4s.Attribute
import org.typelevel.otel4s.oteljava.OtelJava
import org.typelevel.otel4s.metrics.Meter
import org.typelevel.otel4s.trace.Tracer
import java.util.concurrent.TimeUnit
import scala.concurrent.duration.FiniteDuration
case class ApiData(result: String)
trait ApiService[F[_]] {
def getDataFromSomeAPI: F[ApiData]
}
object ApiService {
def apply[F[_]: Async: Tracer: Meter: Random](
minLatency: Int,
maxLatency: Int,
bananaPercentage: Int
): F[ApiService[F]] =
Meter[F]
.counter[Long]("RemoteApi.fruit.count")
.withDescription("Number of fruits returned by the API.")
.create
.map { remoteApiFruitCount =>
new ApiService[F] {
override def getDataFromSomeAPI: F[ApiData] = for {
latency <- Random[F].betweenInt(minLatency, maxLatency)
isBanana <- Random[F].betweenInt(0, 100).map(_ <= bananaPercentage)
duration = FiniteDuration(latency, TimeUnit.MILLISECONDS)
fruit <- Tracer[F].span("remoteAPI.com/fruit").surround(
Async[F].sleep(duration) *>
Async[F].pure(if (isBanana) "banana" else "apple")
)
_ <- remoteApiFruitCount.inc(Attribute("fruit", fruit))
} yield ApiData(s"Api returned a $fruit !")
}
}
}
object ExampleService extends IOApp.Simple {
def run: IO[Unit] =
OtelJava.autoConfigured[IO]()
.evalMap { otel4s =>
(
otel4s.tracerProvider.get("com.service.runtime"),
otel4s.meterProvider.get("com.service.runtime"),
Random.scalaUtilRandom[IO]
).flatMapN { case components =>
implicit val (tracer: Tracer[IO], meter: Meter[IO], random: Random[IO]) =
components
for {
service <- ApiService[IO](
minLatency = 40,
maxLatency = 80,
bananaPercentage = 70
)
data <- service.getDataFromSomeAPI
_ <- IO.println(s"Service data: $data")
} yield ()
}
}
.use_
}Run the application
Setup your metrics dashboard
Connect to grafana and Add your Jaeger and Grafana data sources. Once done, you can create your first dashboard.
Now after making a few calls to ApiService.getDataFromSomeAPI, you should get some data points:
Adding your traces
Just like you just did with Prometheus, you can use Jaeger as data source and display traces using the appropriate data visualization in your new dashboard:
Note: a complete example is available here