Netty Prometheus Integration

[TannerYoung]

Following the Metrics Reporting Guide the server sets up a metric reporter using GrpcServer.default. When I attempt to use GrpcServer.netty I get the following error when pointing prometheus at the service:

INFO: Transport failed
io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: GET /metrics my_server
at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:314)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:251)
....

Is it possible to use this metric reporter with the netty server version?

Hopefully relevant code bits:

object CombinedServerMain extends IOApp with CommonRuntime {
...
override def run(args: List[String]): IO[ExitCode] = {
    for {
      metricsOps <- PrometheusMetrics.build[IO](CollectorRegistry.defaultRegistry)
      interceptor = MetricsServerInterceptor(metricsOps)
      accountService <- AccountService.bindService[IO].map(_.interceptWith(interceptor)).map(AddService)
      server <- GrpcServer.netty[IO](10123, accountService :: Nil)
      runServer <- GrpcServer.server[IO](server).as(ExitCode.Success)
    } yield runServer
}

[TannerYoung]

When using GrpcServer.default[IO] instead of netty; I get the following exception:

java.lang.NoClassDefFoundError: io/grpc/internal/WithLogId
	at java.lang.ClassLoader.defineClass1(Native Method)
 	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
 	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
 	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 	at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.buildTransportServer(NettyServerBuilder.java:432)
 	at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.buildTransportServer(NettyServerBuilder.java:55)
	at io.grpc.internal.AbstractServerImplBuilder.build(AbstractServerImplBuilder.java:223)
	at higherkindness.mu.rpc.server.GrpcServer$.buildServer(GrpcServer.scala:113)

Is prometheus integration in a known good state and I'm doing something wrong? Is there updated documentation on how to use it? I'm dependening on the following targets:

io.higherkindness:mu-rpc-netty:0.18.4
io.higherkindness:mu-rpc-server:0.18.4
io.higherkindness:mu-rpc-prometheus:0.18.4

[fedefernandez]

It looks a binary incompatibility. Could you please share the output of the sbt evicted command?

I've got this project with a sample server/client using prometheus

https://github.com/fedefernandez/scala-italy-2019

[TannerYoung]

I'm using bazel (not sbt) so that very likely could be the issue..

Thanks for the pointer to the example; I'll take a look tomorrow.

[TannerYoung]

Looks like I was transitively depending on an incompatible version of grpc_netty_shaded via firebase_admin:

com/google/firebase:firebase_admin
com/google/cloud:google_cloud_firestore
io/grpc:grpc_netty_shaded

I've now pinned grpc-netty-shaded to version to 1.18.0 which solves the NoClassDefFound problem.

I'm still experiencing the Unexpected HTTP/1.x request: GET /metrics with both nettty and default with slightly different stack traces:

default

INFO: Transport failed
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: GET /metrics 
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:314)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode
...

netty

INFO: Transport failed
io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: GET /metrics 
at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString
...

Not sure where to go from here, I've made a small IOApp which serves a single service with a single proto endpoint. I've copied the imports from the example app you gave to make sure I wasn't missing anything, and otherwise I have just 3 dependencies beyond my proto definition:

io/higherkindness:mu_rpc_prometheus:0.18.4
io/higherkindness:mu_rpc_server:0.18.4
org/typelevel:cats_effect:2.0.0

Do I need additional dependencies to support the http requests to /metrics? It doesn't look like example project has any additional dependencies I'm missing that the server is using save slf4j and maybe fs2. But I've attempted adding those as well for a sanity check and they have no impact. I've also enabled the paradise macro compiler plugin, but unsurprisingly that didn't fix it either.

Not sure what else to try.

[fedefernandez]

@TannerYoung yeah, you need to provide that HTTP endpoint. We've got an article that describes all the process: https://www.47deg.com/blog/metrics-integration-with-mu/#checking-metrics-through-prometheus-server-11

Basically, you need to start a server that uses the CollectorRegistry for exporting the metrics.

[TannerYoung]

Thanks - got it working. I think this should be an update documentation, because outside of that blog post I haven't seen anything on exporting the metrics. Even something along the lines of "An additional http server integration is required to export the metrics" would have tipped me off. Maybe I just didn't see the right places though. Feel free to close now.

Thanks a ton for all the help.

[fedefernandez]

Glad it worked. I've created an issue #677