Honeycomb - metrics and traces

In this example, we are going to use Honeycomb to collect and visualize metrics and traces produced by an application. We will cover the configuration of OpenTelemetry exporter, as well as the instrumentation of the application using the otel4s library.

Unlike Jaeger example, you do not need to set up a collector service locally. The metrics and traces will be sent to a remote Honeycomb API.

At the time of writing, Honeycomb allows having up to 20 million spans per month for a free account. It offers robust analysis and visualization tools that are handy for exploring the world of telemetry.

Project setup

Configure the project using your favorite tool:

Add settings to the build.sbt:

libraryDependencies ++= Seq(
  "org.typelevel" %% "otel4s-oteljava" % "0.6.0", // <1>
  "io.opentelemetry" % "opentelemetry-exporter-otlp" % "1.37.0" % Runtime, // <2>
  "io.opentelemetry" % "opentelemetry-sdk-extension-autoconfigure" % "1.37.0" % Runtime // <3>
run / fork := true
javaOptions += "-Dotel.java.global-autoconfigure.enabled=true"            // <4>
javaOptions += "-Dotel.service.name=honeycomb-example"                    // <5>
javaOptions += "-Dotel.exporter.otlp.endpoint=https://api.honeycomb.io/"  // <6>

Add directives to the tracing.scala:

//> using lib "org.typelevel::otel4s-oteljava:0.6.0" // <1>
//> using lib "io.opentelemetry:opentelemetry-exporter-otlp:1.37.0" // <2>
//> using lib "io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.37.0" // <3>
//> using `java-opt` "-Dotel.java.global-autoconfigure.enabled=true"            // <4>
//> using `java-opt` "-Dotel.service.name=honeycomb-example"                    // <5>
//> using `java-opt` "-Dotel.exporter.otlp.endpoint=https://api.honeycomb.io/"  // <6>

1) Add the otel4s library
2) Add an OpenTelemetry exporter. Without the exporter, the application will crash
3) Add an OpenTelemetry autoconfigure extension
4) Enable OpenTelemetry SDK autoconfigure mode
5) Add the name of the application to use in the traces
6) Add the Honeycomb API endpoint

OpenTelemetry SDK configuration

As mentioned above, we use otel.java.global-autoconfigure.enabled and otel.service.name system properties to configure the OpenTelemetry SDK. The SDK can be configured via environment variables too. Check the full list of environment variable configurations for more options.

Acquiring a Honeycomb API key

The Honeycomb official guide.

First, you must create an account on the Honeycomb website. Once you have done this, log into your account and navigate to the environment settings page. There you can find a generated API key.

Use a different environment for test, production, and local development. Each will have its own API Key. This organizes your data in Honeycomb.

Honeycomb configuration

The Honeycomb official configuration guide.

In order to send metrics and traces to Honeycomb, the API key and metrics dataset name need to be configured. Since the API key is sensitive data, we advise providing them via environment variables:

$ export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=otel-metrics"

1) x-honeycomb-team - the API key
2) x-honeycomb-dataset - the name of the dataset to send metrics to.

Each service's traces will land in a dataset defined in 'otel.service.name'.

Note: if the x-honeycomb-dataset header is not configured, the metrics will be sent to a dataset called unknown_metrics.

Application example

import java.util.concurrent.TimeUnit

import cats.effect.{Async, IO, IOApp}
import cats.effect.std.Console
import cats.effect.std.Random
import cats.syntax.all._
import org.typelevel.otel4s.{Attribute, AttributeKey}
import org.typelevel.otel4s.oteljava.OtelJava
import org.typelevel.otel4s.metrics.Histogram
import org.typelevel.otel4s.trace.Tracer

import scala.concurrent.duration._

trait Work[F[_]] {
  def doWork: F[Unit]

object Work {
  def apply[F[_]: Async: Tracer: Console](histogram: Histogram[F, Double]): Work[F] =
    new Work[F] {
      def doWork: F[Unit] =
        Tracer[F].span("Work.DoWork").use { span =>
          span.addEvent("Starting the work.") *>
            doWorkInternal(steps = 10) *>
            span.addEvent("Finished working.")

      def doWorkInternal(steps: Int): F[Unit] = {
        val step = Tracer[F]
          .span("internal", Attribute(AttributeKey.long("steps"), steps.toLong))
          .surround {
            for {
              random <- Random.scalaUtilRandom
              delay <- random.nextIntBounded(1000)
              _ <- Async[F].sleep(delay.millis)
              _ <- Console[F].println("Doin' work")
            } yield ()

        val metered = histogram.recordDuration(TimeUnit.MILLISECONDS).surround(step)

        if (steps > 0) metered *> doWorkInternal(steps - 1) else metered

object TracingExample extends IOApp.Simple {
  def run: IO[Unit] = {
      .evalMap { otel4s =>
          .flatMap { implicit tracer: Tracer[IO] =>
            for {
              meter <- otel4s.meterProvider.get("com.service.runtime")
              histogram <- meter.histogram[Double]("work.execution.duration").create
              _ <- Work[IO](histogram).doWork
            } yield ()

Run the application

$ export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=honeycomb-example"
$ sbt run
$ export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=honeycomb-example"
$ scala-cli run tracing.scala

Query collected traces and metrics

You can query collected traces and metrics at https://ui.honeycomb.io/.


Honeycomb Traces Example


Honeycomb Metrics Example