Splunk OpenTelemetry Collector Distribution Roadmap Update: Breaking Changes

Since 2021, we have been making steady releases of our distribution of the OpenTelemetry (OTel) collector following the biweekly schedule of upstream releases.

By following a tight schedule, we keep on top of vulnerability reports and make incremental changes.

We announce upcoming breaking changes via deprecation notices and warnings during collector startup.

We are now releasing Splunk OTel Collector v0.154.0.

This new release is significant, because it brings one of the biggest changes we have made since the end of life of the SignalFx agent: we are removing a set of functionalities from our distribution by removing the Agent Bundle.

Back in 2016, SignalFx started working on Getting Data In (GDI) using a proven open-source project named CollectD, which is composed of Java and Python plugins to monitor a variety of systems.

In time, SignalFx invested in building its own set of plugins and maintained them under its github organization.

In 2019, SignalFx decided to invest in a richer metrics model and moved to build its own open-source data collection tool named the SignalFx Smart Agent. Written in Go, it allowed integration with telegraf libraries. For backwards compatibility purposes, the Smart Agent wrapped its CollectD functionality as a set of plugins. When running, say, the `collectd/marathon` plugin, the Golang-written Smart Agent would instantiate a C++ written CollectD process, which would then configure a Python process. The processes communicate over stdout.

In 2021, we wrapped the Smart Agent as the smartagentreceiver into our distribution of the OpenTelemetry Collector. The receiver would allow to run any Smart Agent plugin. We then started to donate upstream each of the plugins we could port over to OpenTelemetry concepts. This took years as we modernized and collaborated with the community to make this code resilient and modern.

Over time, we were left with a number of CollectD plugins that we had deprecated in favor of OpenTelemetry Collector receivers. For some, we retired them as the technology was no longer of use, such as our Mesos or Marathon integration. For others, we identified native replacements such as the Jenkins OpenTelemetry plugins or moving to support the OpenStack Prometheus endpoints.

Today, we are removing the custom CollectD build, the Python runtime, and all the Python packages we have been shipping with the collector. This reduces the size of our artifacts meaningfully ; which also reduces our vulnerability surface greatly.

Looking Forward

We will continue to replace the Smart Agent receiver plugins with native OpenTelemetry receivers, as we enjoy building upstream, using the OpenTelemetry de facto standard.

We will also continue to implement new functionalities upstream to make the collector a world class experience. We are collaborating with customers to implement better DNS monitoring to close the functionality gap of the DNS plugin.

With those changes, we have freed engineering resources to work closer to OpenTelemetry and push for a modern observability experience. The team views this achievement as a major milestone and is so thankful for your continued support of both the project and Splunk. Thank you!

There are more OpenTelemetry innovations on the horizon for Splunk customers. For the latest on Splunk and OpenTelemetry, check out these blogs.