langfuse, weave, or other evaluation platform bindings?

[snickell]

Anyone investigated hooking in bindings for langfuse or other LLM metrics calling platforms? Langfuse seems particularly interesting because, while they have a pasted hosting version, they're fully open source.

The langfuse service (hosted by them or self hosted open source) provides an (OpenTelemetry REST API)[https://langfuse.com/integrations/native/opentelemetry] intended for language bindings to hook into.

This could probably (?) be done by monkeypatching RubyLLM, but it feels like it'd be healthy to build in a general metrics-hooks API that could be used to provide bindings to multiple evaluation platforms.

[crmne]

There are a bunch of gems in our ecosystem that can help with that! https://rubyllm.com/ecosystem/