The hidden cost of running your own observability stack

At my latest $job, I was tasked of setting up the LGTM stack (Loki, Grafana, Tempo, Mimir) for observability. Fast forward a few months, I noticed there's a hidden aspect to running the stack that I was not expecting before and that is the network cost, specifically the network transfer cost for cross AZ traffic. At one point we were paying more than $100 per day just for the cross AZ network traffic.

Update: since this article was written, I've found out that the official Loki helm chart have a section addressing the cross-az issue in the values file. It does not recommend running Loki across multiple AZs on the cloud.

Note: This can be used to run Loki over multiple cloud provider availability zones however this is not currently recommended as Loki is not optimized for this and cross zone network traffic costs can become extremely high extremely quickly. Even with zone awareness enabled, it is recommended to run Loki in a single availability zone.

Cross AZ Traffic Amplification

While investigating where does the traffic coming from I compared the load balancer "Processed Bytes" metrics with the Cost Explorer usage for cross AZ traffic and noticed that there's a 10x increase in the reported values by the load balancer to the actual charged traffic. It baffled me a bit and made me step back and take a deeper look at the possible points where I'm getting charged.

Collector to load balancer node
load balancer node to ingress controller pod
ingress controller pod to distributor
distributor to ingester

Animation sequence.

One telemetry batch starts at a collector in AZ A. With client affinity off, it reaches a load balancer node in AZ B, recording the first cross-AZ transfer.
The load balancer forwards to ingress in AZ B without crossing zones. Default round-robin service routing then sends the batch to a distributor in AZ C, recording a second cross-AZ transfer.
With a replication factor of three, the distributor sends a copy to one ingester in each AZ. Two copies cross zones, bringing this illustrative route to four cross-AZ transfer events.
The same batch is replayed with AZ affinity and topology-aware routing. The first three hops remain in AZ A, while two replicated writes still cross to ingesters in AZ B and AZ C.

Conclusion: affinity and topology-aware routing remove avoidable transfers, but cross-AZ ring replication remains. The actual amplification depends on placement and replication factor.

Collector to Load Balancer Node: client routing policy

In my setup, the services are exposed through a load balancer and given a DNS name like loki.example.com. The collectors are configured to send the telemetry data to that URL. Here is my fist mistake, I didnt' enable "Availability Zone affinity" for the client routing policy. When enabled, this will route traffic from the collector to the load balancer node in the same AZ avoiding being charged for cross AZ traffic.

The load balancer node will then forward the traffic to the ingress controller pod in the same AZ.

Ingress controller pod to distributor: Kubernetes Topology Aware Routing

From the load balancer node, the traffic will be forwarded to the k8s pod through the k8s service. The default behavior of service in k8s is it will route the traffic using the round-robin algorithm. This means that from the ingress controller pod to the distributor pod the traffic will go cross AZ. If you have 3 distributor pods, this means 2 out of 3 connections will be routed to pods in different AZ.

To avoid the traffic from crossing AZ, we can use kubernetes topology aware routing feature. Downside of using this is that we need to have at least 3 pods in each AZ but compute is cheaper in my use case since I'm using spot instances through Karpenter and getting up to 70% discount on the node price.

Distributor to Ingester: no workaround

This the only part I haven't solved. In the LGTM stack, the distributors uses an internal discovery mechanism to get the IP of the ingesters. This means that we cannot use kubernetes topology aware routing here.

To make things worse, depending on the replication_factor configuration, each distributor might be sending the logs to multiple ingesters, each one multiplying the cross AZ cost that we have to pay.

Special use case: getting logs from external source

Other than the above use case, our company also have other use case where the logs comes from external sources instead of from our internal network. In this case, I actually managed to eliminate the cross AZ cost completely by deploying the LGTM stack in just one AZ. The load balancer is also configured to use only one subnet that is in the same AZ.

Conclusion

All the above factor might explain why the amount I was charged for cross AZ traffic is 10x bigger than the amount that is received at the load balancer. I outlined some the possible points where the cross AZ charges are coming from and how to fix it. Hope it helps!