Author: Ramadan Khalifa

In this article, we’re going to learn about semantic conventions in OpenTelemetry and how they are used to make data processing much easier. We’ll also discuss the different types of semantic conventions. without further ado let’s get started.

What Are Semantic Conventions?

Semantic conventions in general are the agreed-upon meaning of words and phrases within a particular language or culture. They help us communicate with each other by providing a shared understanding of our symbols.

For example, the “thumbs up” gesture is a convention that means “good job” or “I agree” in many cultures.

Without these conventions, communication would be much more difficult; we would constantly have to explain the meaning of every single word we use.

What Do Semantic Conventions Mean In OpenTelemetry?

Semantic conventions are important in OpenTelemetry because they help to define the meaning of resources and metrics. Semantic conventions provide a common language for all users of the system. This allows for a more accurate interpretation of data and helps to ensure that everyone is on the same page when it comes to resource usage and performance.

In this lesson, We will take a quick look at the different kinds of semantic conventions provided by Opentelemetry. Let’s start with metric semantic conventions,

What Are Metric Semantic Conventions?

The OpenTelemetry project has published a set of metric semantic conventions that can be used by any software that collects or displays metrics data.

The metric semantic conventions define a set of core dimensions that should be used when recording metric data. These dimensions include name, description, unit, and type. In addition, the conventions define a set of recommended labels that can be used to further describe the data. By following these standards, it is possible to create easily understood metrics that can be effectively compared.

A quick example for naming conventions is limit which means the known total amount of something. For example, system.memory.limit for the total amount of memory on a system.

utilization – means the fraction of usage out of its limit should be called entity.utilization. For example, system.memory.utilization for the fraction of memory in use. Utilization values are in the range [0, 1].

For more information about metric semantic conventions, please check the official documentation of opentelemetry from here

In addition to the metric semantic conventions, the OpenTelemetry team has also published standards for logging and tracing data. By using these standards, it is possible to create software that can generate consistent results regardless of the underlying implementation.

Let’s take a closer look at semantic conventions for spans and traces.

semantic conventions for spans and traces

It’s recommended to use attributes to describe the dimensions of the telemetry data collected. For example, when dealing with network data, attributes might describe the source and destination IP address, the port numbers, etc. Attributes can also be used to describe metadata about the data itself. For example, when dealing with log data, attributes might describe the timestamp, the logging level, etc.

Attributes are stored in so-called AttributeMaps. An AttributeMap is a map from attribute keys to attribute values. The keys are typically strings, but they can also be other data types. The values can be any data type that can be represented as a JSON value.

One of the benefits of using attributes is that they provide a way to add additional information to the data without changing the data itself. This is especially useful when dealing with legacy systems that cannot be modified.

Another benefit of attributes is that they can be used to filter and group data. For example, if one has a log file that contains messages from multiple sources, they can use attributes to filter out messages from certain sources. Or, if they want to group all messages with the same logging level, they can use attributes.

Here is an example of manually defining attributes for a service

# Resource attributes that describe a service.
namespace = Company.Shop
service.name = shoppingcart

manually defining attributes for a service

Events

Events are one of the core concepts in OpenTelemetry, providing a way to record significant moments or states in the system that can be used for monitoring and analysis. They can be generated manually by operators or automatically by the OpenTelemetry SDK. OpenTelemetry events contain metadata about the event and any relevant data that was collected at the time of the event.

Events can be used to track the progress of a system through its lifecycle or to identify changes in state that may indicate an issue. They can also be used to record performance data, such as response times or resource utilization. By analyzing events, it is possible to understand how a system is functioning and where potential problems may lie.

Here is an example event in a span

{
  "name": "Hello",
  "context": {
    "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df2",
    "span_id": "0x051581bf3cb55c13"
  },
  "parent_id": null,
  "start_time": "2023-01-19T18:52:58.114201Z",
  "end_time": "2023-01-19T18:52:58.114687Z",
  "attributes": {
    "namespace": "Company.Shop",
    "service.name": "shoppingcart"
  },
  "events": [
    {
      "name": "Guten Tag!",
      "timestamp": "2023-01-19T18:52:58.114561Z",
      "attributes": {
        "event_attributes": 12
      }
    }
  ]
}

The event here has a name, timestamp and some attributes.

Conclusion

Semantic conventions are important in OpenTelemetry because they provide a common language for all users of the system. The OpenTelemetry project has published a set of metric semantic conventions that can be used by any software that collects or displays metrics data. In addition, the OpenTelemetry team has also published standards for logging and tracing data. By using these standards, it is possible to create software that can generate consistent results regardless of the underlying implementation.

To learn more Semantic Conventions, Please check official OpenTelemetry documentation:

• Trace Semantic Conventions
• Metric Semantic Conventions
• Resource Semantic Conventions

Distributed tracing is a method of tracking the propagation of a single request as it’s handled by various services that make up an application. Tracing in that sense is “distributed” because in order to fulfill its function, a single request must often traverse process, machine and network boundaries.

Once we instrumented our application and exported our telemetry data to an observability backend (like somulogic or new relic), It’s time to use this data to debug our production system efficiently. In this article, we will explore debugging techniques applied to observability data and what separates them from traditional techniques used to debug production applications.

To learn more about tracing and what it means to instrument an application and export telemetry data, please check this article.

Before we start with how we can use traces and spans to debug our production applications during incidents, it’s important to take a brief look at how we used to do it using logs and metrics, the old way.

Old way of debugging an application using logs and metrics

Prior to distributed tracing, system and application debugging mostly occurred by building upon what you know about a system. This can be observed when looking at the way the most senior members of an engineering team approach troubleshooting. It can seem magical when they know what is the right question to ask and instinctively know the right place to look at. That magic is born from intimate familiarity with the application.

To pass this magic to other team members, managers usually ask senior engineers to write detailed runbooks in an attempt to identify and solve every possible problem (Root Cause) they might encounter. But that time spent creating runbooks and dashboards is largely wasted, because modern systems rarely fail in precisely the same way twice.

Anyone who has ever written or used a runbook can tell you a story about just how woefully inadequate they are. Perhaps they work to temporarily address technical debt: there’s one recurring issue, and the runbook tells other engineers how to mitigate the problem until the upcoming sprint when it can finally be resolved. But more often, especially with distributed systems, a long thin tail of problems that almost never happen are responsible for cascading failures in production. Or, five seemingly impossible conditions will align just right to create a large-scale service failure in ways that might happen only once every few years.

Yet engineers typically embrace that dynamic as just the way that troubleshooting is done—because that is how the act of debugging has worked for decades. First, you must intimately understand all parts of the system—whether through direct exposure and experience, documentation, or a runbook. Then you look at your dashboards and then you…intuit the answer? Or maybe you make a guess at the root cause, and then start looking through your dashboards for evidence to confirm your guess.

Even after instrumenting your applications to emit observability data, you might still be debugging from known conditions. For example, you could take that stream of arbitrarily wide events and pipe it to tail -f and grep it for known strings, just as troubleshooting is done today with unstructured logs. Or you could take query results and stream them to a series of infinite dashboards, as troubleshooting is done today with metrics. You see a spike on one dashboard, and then you start flipping through dozens of other dashboards, visually pattern-matching for other similar shapes.

But what happens when you don’t know what’s wrong or where to start looking? When debugging conditions are completely unknown to you??

The real power of observability is that you don’t have to know so much in advance of debugging an issue. You should be able to systematically and scientifically take one step after another, to methodically follow the clues to find the answer, even when you are unfamiliar (or less familiar) with the system. The magic of instantly jumping to the right conclusion by inferring an unspoken signal, relying on past scar tissue, or making some leap of familiar brilliance is instead replaced by methodical, repeatable, verifiable process.

Debugging a production application using traces

Debugging a production application using traces and spans is different. It doesn’t require much experience with the application itself. you just need to be curious to learn more about what’s actually happening with the application in the production environment. It simply works like this:

1. Start with the overall view of what prompted your investigation: what did the customer or alert tell you?
2. then verify that what you know so far is true: is a notable change in performance happening somewhere in this system? Data visualizations can help you identify changes of behaviour as a change in a curve somewhere in the graph.
3. Search for dimensions that might drive that change in performance. Approaches to accomplish that might include: Examining sample rows from the area that shows the change: are there any outliers in the columns that might give you a clue? Slicing those rows across various dimensions looking for patterns: do any of those views highlight distinct behaviour across one or more dimensions? Try an experimental group by on commonly useful fields, like status_code. Filtering for particular dimensions or values within those rows to better expose potential outliers.
4. Do you now know enough about what might be occurring? If so, you’re done! If not, filter your view to isolate this area of performance as your next starting point. Then return to step 3.

You can use this loop as a brute-force method to cycle through all available dimensions to identify which ones explain or correlate with the outlier graph in question, with no prior knowledge or wisdom about the system required.

Example

For example, Let’s say we have a spike in request latency of some APIs for different users. If we isolated those slow requests, we would easily see that slow-performing events are mostly originating from one particular availability zone (AZ) from our cloud infrastructure provider (assuming we have the AZ information in the spans). After digging deeper, we might notice that one particular virtual machine instance type appears to be more affected than others.

This information has been tremendously helpful: we now know the conditions that appear to be triggering slow performance. A particular type of instance in one particular AZ is much more prone to very slow performance than other infrastructure we care about. In that situation, the glaring difference pointed to what turned out to be an underlying network issue with our cloud provider’s entire AZ.

Another Example

Here’s another example of root cause analysis using spans to make sure it’s clear. Let’s assume that after deploying a new version of our application, we noticed that some APIs are getting slower. To investigate this issue, we will follow our traditional way of debugging using distributed tracing. We started by taking a deeper look at the slow APIs and looking for dimensions that might drive that change in performance. After diving deeper, we found out that all those APIs are calling a payment_service. After diving deeper into spans related to payment_service, we found out that it fetches data from a postgresql db specifically from a db table called user_payments_history. Comparing those spans with similar spans from the same API calls but before that deployment, we found that those queries to user_payments_history table are new and actually they are taking some time to get the required data.

The problem here might be a missing index that causes the query to be slow or that db table user_payments_history might have too many records. There is no way to be sure what is the root cause here but at least we know for sure that there is something wrong with this db table user_payments_history in the payment_service.

Not all issues are as immediately obvious as this underlying infrastructure issue. Often you may need to look at other surfaced clues to triage code-related issues. The process remains the same, and you may need to slice and dice across dimensions until one clear signal emerges, similar to the preceding example.

Conclusion

With complex distributed systems, It became really hard to figure out what is really going on in a production application. That’s why metrics and logs alone are not enough to debug those apps and find what is the root cause of an incident.

Traces and spans can help in that situation. With high cardinality events, We can collect lots of information about our system that will be really handy when dealing with incidents under time pressure. We have a systematic approach to find the root cause of incidents assuming we are collecting enough information (dimensions) in spans.

To learn more about observability, please check:

• Observability and how to instrument applications
• Sampling Traces In OpenTelemetry

References

• Observability Engineering: Achieving Production Excellence

At a scale, the cost to collect, process and save traces can dramatically outweigh the benefits because many of these events are virtually identical and successful. The point of debugging is to search for patterns or examine failed events during an outage. That’s why it’s wasteful to transmit 100% of all events to the observability backend.

To debug effectively, we just need a representative sample of successful events which can be compared to bad events.

We can sample events by using the strategies outlined in this article and still provide granular visibility into system state. Unlike pre-aggregated metrics that collapse all events into one coarse representation of system state over a given period of time, sampling allows us to make informed decisions about which events can help us surface unusual behaviour, while still optimizing for resource constraints. The difference between sampled events and aggregated metrics is that full cardinality is preserved on each dimension included in the representative event.

In OpenTelemetry, there are two approaches to achieve sampling, Head-Based Sampling and Tail Based Sampling. Let’s review both approaches and see when to use them.

Head-based sampling

As the name suggests, head-based sampling means to make the decision to sample or not at the beginning of the trace.

This is the most common way of doing sampling today because of the simplicity, but since we don’t know everything in advance we’re forced to make arbitrary decisions (like a random percentage of all spans to sample) that may limit our ability to understand everything.

A disadvantage of head-based sampling is the fact that you can’t decide that you want to sample only spans with errors since you do not know this in advance because the decision to sample or not happens before the error happens.

Built-in samplers include ( ParentBased,  AlwaysOn,  AlwaysOff and ParentBased Samplers)

“AlwaysOn” (AlwaysSample) sampler

As the name suggests, It essentially means to sample all events – and take 100% of the spans. In a perfect world, we would use this only, without any cost considerations.

“AlwaysOff” (NeverSample) sampler

Also as the name suggests, the AlwaysOff sampler samples 0% of the spans. This means that no data will be collected whatsoever. You probably won’t be using this one much, but it could be useful in certain cases. For example, when you run load tests and don’t want to store the traces created by them.

ParentBased Sampler

This is the most popular sampler and is the one recommended by the official OpenTelemetry documentation. When a trace begins we make a decision whether to sample it or not. Whatever the decision is, The child span will follow it.

The main advantage to ParentBased Sampler is that you always get the complete picture.

How does this work? For the root span, we decide whether it will be sampled  or not. The decision is sent to the rest of the child spans in this trace via context propagation, making each child know if it needs to be sampled or not.

It is important to understand that this is a composite sampler, which means it does not live on its own but it lets us define how to sample for each use case. For example, we can define what to do when we have no parent by using the root sampler.

ParentBased(root=TraceIDRatioBased)

It’s recommended to use the parent-based sampler with TraceIDRatioBased sampler as the root sampler.

The TraceIDRatioBased based sampler uses the trace ID to calculate whether or not the trace should be sampled or not, with respect to the sample rate we choose.

Tail-based sampling

Contrary to head-based sampling, in Tail-based sampling we make the decision to sample or not at the collector level. This can be useful for metrics, for example, when we want to gather the latency, We must know the exact start and end times which cannot be done in advance.

Also, what was a disadvantage of the head-based is an advantage for tail-based sampling which is being able to only sample spans with errors.

So where should sampling be implemented?

Well, that depends on your specific use case so there is no one solution that fits all.

If you choose to do it at the OTEL distro level (Head-based sampling) , you remove redundant data at the source, never needing to worry about it again. You also minimize data transported in the network. However, when you need to update the sample rate you have to redeploy your services each time.

If you implement it in the collector you have a centralized place that controls sampling so you don’t need to redeploy your server when you change your sample rate. However, making the sampling decision requires buffering the data until a decision can be made and thus adds overhead.

Conclusion

Sampling traces and spans is almost always a good idea since it will save lots of money and most likely won’t affect the debugging process using spans and traces in production. There are different approaches to implement it in opentelemetry. Head-based sampling is simpler to implement but it requires redeploying for each change. Tail-based sampling is a little bit harder to implement but it gives us the ability to only sample traces with errors.