Aiman Ismail

Scale down your instances, cut down your AWS bills

Tue, 29 Apr 2025 06:14:00 GMT

I joined a webinar from Platformatic on the best practices of running nodejs services in production and one of the suggestion is to give each of the pod 1 full CPU core. All went well, our latency metrics seems to have improved, case closed. Little did I know that at the end of the month I would get hit by 3x increase in our monthly AWS bills. So I started optimizing.

Know your workload patterns

Our company mainly serves restaurants so the traffic to our services has a predictable pattern; it starts ramping up at around breakfast time and peaks at around lunch and dinner. We have a few customers that operates until midnight but they're the minority so during those ours incoming traffic to our services it at the minimum. This pattern is something I noticed early when I first joined as possible optimization area but never got to it until now.

Our workloads run on AWS EKS and mainly using managed node groups provisioned statically with no cluster-autoscaler configured to add or remove nodes. There is also not HorizontalPodAutoscaler (HPA) setup for our Pods. This decision is done mainly to simplify operations. Just add more replicas and more nodes when the pods are getting overwhelmed by requests. It works at a smaller scale since the AWS bills is still manageable but we went through a growth phase recently and our infra costs also jumped. We figured that we have to tackle this now instead of pushing it back for later.

Know your tools

When talking about autoscaling in kubernetes-land, there's the HorizontalPodAutoscaler (HPA). HPA allows you to scale your pods based on cpu, memory, or in the more recent versions, any custom metrics. The custom metrics autoscaling works but the UX leaves much to be desired. That's why I chose to use KEDA instead.

KEDA builts on top of HPA, providing a simplified interface. You also have more options to scale on metrics from almost anywhere using the available Scalers. For my use case, I'll be using the Cron and Prometheus scalers.

Additional note for why to use KEDA is that it supports scaling down pods to 0 and scaling back up. This is called Activation inside KEDA where the pod replicas goes from 0 to 1 and vice versa. This doesn't apply to our use case since we still have to keep some pods running during midnight but if your workload allows it this will definitely gives you more savings.

Setting up the cron scaler

As mentioned above, we have a regular and predictable traffic pattern for our workloads, starts at 7AM and peaks during lunch and dinner time. So, the cron scaler fits perfectly for this use case. We set a schedule to scale up our services during those hours and for the rest of the hours, scale it back down to the minimum possible. KEDA uses a custom resource called ScaledObject to specify the definition for the autoscaling.

In my setup, I'll configure it to scale based on the following rule:

7AM - 11AM: 10 replicas
11AM - 2PM: 20 replicas
2PM - 6PM: 10 replicas
6PM - 10PM: 20 replicas
outside of those hours, scale down to minimum replica count which is 1

This is how the manifest looks like:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: backend-service
spec:
  maxReplicaCount: 30
  minReplicaCount: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backend-service
  triggers:
    - metadata:
        desiredReplicas: '10'
        start: 0 7 * * *
        end: 0 11 * * *
        timezone: Asia/Singapore
      type: cron
    - metadata:
        desiredReplicas: '20'
        start: 0 11 * * *
        end: 0 14 * * *
        timezone: Asia/Singapore
      type: cron
    - metadata:
        desiredReplicas: '10'
        start: 0 14 * * *
        end: 0 18 * * *
        timezone: Asia/Singapore
      type: cron
    - metadata:
        desiredReplicas: '20'
        start: 0 18 * * *
        end: 0 22 * * *
        timezone: Asia/Singapore
      type: cron

Combining cron and prometheus scalers

Cron and prometheus scalers on its own is not enough. If I'm using cron alone, what if suddenly one day the traffic suddenly higher than usual? This where prometheus scalers comes in. It looks at the actual metrics and scale accordingly.

Then you might think, why not just use prometheus scalers on its own then? It depends. If your traffic always grows slowly and gradually then ya it might work. New pods starting up can catchup to the traffic coming in but during rush hours the traffic can increase really fast and to avoid waiting for our pods to scale up which might take some time, we just decided to pre-scale up our pods during the expected rush hours. With this setup, the prometheus scaler can supplements the pods if the configured cron autoscaling is not enough.

Setting up the prometheus scalers

For the prometheus scalers, we decided to scale our pods based on the response latency served from the service. This is based on the RED method. The "D" inside "RED" stands for Duration - or latency. This is a great metric to measure the performance of our service since it directly correlates to the experience faced by the user when using that service. High latency means your customer needs to wait longer, which is bad.

This latency metrics however does not come out of the box. In our setup, we generate this metrics from the traces emitted by our service which is instrumented using OpenTelemetry (otel). All the traces are sent to Tempo, our backend for storing traces, and Tempo will generate the metric traces_spanmetrics_latency from it. Generating the metrics on Tempo is out of the scope for this article but you can refer to the Tempo docs.

Issues arising from autoscaling

If you think once the autoscaling is rolled out then all is good.. then you're dead wrong - so was I. The first day we rolled out the autoscaling, we monitor the service closely for any increase in errors and error it did. First, it was just not enough capacity. The original capacity we put in for autoscaling is not enough. Easy fix just add more capacity. Then, we were scaling up too late and scaling down too early. Also easy fix, just move the scaling up period higher and scaling down period later.

Connections being terminated prematurely

The not so obvious one tho is that, every time the we scales down by half we see a lot of errors from the service coming from the service. Those errors mostly related to the connection being terminated prematurely. There are two ways to reduce this but I'll explain the one way we took for now which is configuring the HPA behavior.

The first one is that HPA by default scales down too fast for us. This cause the service to scale down fast, then notice that the resource is not enough and it scales back up - rinse and repeat. In autoscaling we call this behavior as "flapping" and we don't want that. We want our service to be stable. This is the modification I've done to our above ScaledObject - adding stabilizationWindowSeconds and set it to remove pods one by one.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: backend-service
spec:
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 600 # wait 10 minutes before scaling down
          policies:
            - type: Pods
              value: 2
              periodSeconds: 180  # remove two pods every 3 minutes

HPA config: stabilizationWindowSeconds

KEDA allows you to configure the underlying HPA object directly from the ScaledObject. First, we'll configure the HPA stabilization window so that it will look at the last 10 minutes of recommendations by the HPA and only apply the highest value. This means if within the last 10 minutes your HPA recommended to scale down from 10 to 8, then a few minutes later to 5. Then it will scale down to 8 only and not 5 directly. It'll have to wait until the recommedation to scale down to 8 is outside of the window then only it'll scale down to 5.

HPA config: pod scale down policy

HPA allows configuring the scaling policies separately for scaling up and down. By default, HPA will scale down by up to 100% of the available replicas every 15s. This means, if you have 20 replicas, HPA will immediately terminates half the pods from 20 to 10 when scaling down. If the HPA recommendation says it should go down further to 1, then after 15s it will scale down further to 5.

In this above snippet, we changed it to be less aggressive by allowing only 2 pods to scale down every 3 minutes. We've seen massive reductions in the number of connections being terminated. I started with only 1 pod per minute but find it too fast then increase it to the current amount.

Scaling down your nodes using Karpenter

After the pods has been scaled down, your kubernetes nodes would be running underutilized. You can use any cluster autoscaler of your choice for scaling down underutilized nodes but in my case I used Karpenter since I'm already running on EKS and Karpenter was built for it originally. For this part there is less suprise tho I do plan to write more on running Karpenter in production, hopefully it will come out soon. Ping me on my socials if it is not out yet after 3 months you're reading this (random deadline for myself lol).

Summary

After all this autoscaling exercise we actually reduced our AWS spending for EC2 instances used by EKS clusters by approximately 50%. This is a huge amount for us and my boss was defnitely happy (promotion soon?). Hope this helps anyone going on this journey :)

On how to tackle the disconnect issues, you can also configure graceful shutdown for your pods. I'll link to this detailed article from learnk8s on how to do graceful shutdown in kubernetes.

Eliminating cross-AZ traffic cost on AWS

Sat, 14 Dec 2024 09:23:06 GMT

Imagine going through your AWS bills and noticing that APS1-DataTransfer-Regional-Bytes is 1/3 of your monthly AWS cost. After reading a bit more on how AWS charges for network traffic you know that this is referring to the cost incurred when your traffic crosses an availability zone (AZ). This article will go through what you can do to eliminate this cross-AZ traffic cost.

Disclaimer: this is not necessarily the best practice. I try my best to highlight the caveat and tradeoff you're making in this article but please make your own judgement before making the changes.

The Traffic Flow

In this article I will use the above diagram as our example network flow. Our example scenario will use the AWS Network Load Balancer (NLB) as the load balancer (LB), then routes the traffic to the ingress-nginx pods running inside our cluster. Finally, ingress-controller pods will route the traffic to the backend services serving the API.

The top and bottom part of the diagram will show how the traffic flow will be between AZs. Once we implemented the steps I will be describing below, you should be able to eliminate the cross-AZ traffic in our network.

Incoming traffic to Load Balancer Nodes

When you provision a load balancer on AWS, you will get a domain name that can be used to resolve to the IP addresses of your LB nodes. Depending on where the DNS resolving happens, this might be the first contributor to your cross-AZ traffic cost.

Resolving LB Nodes IP from the Internet

There is nothing we can do if the source of the traffic is from the internet. The DNS will resolve to one of the IP addresses of the LB and we wouldn't be charged for it as there is no AZ yet here.

Resolving LB Nodes IP from within your AWS Network

If the source traffic originates from within your AWS network, there is the possibility that the LB nodes IP resolved is not within the same AZ as your source traffic. To avoid this, on the LB there is an option to set the client routing policy to resolve to LB nodes IP that is within the same AZ.

There is 3 options to choose from:

AZ affinity: queries may resolve to other zones if there are no healthy load balancer IP addresses in their own zone
Partial AZ affinity: 85% of client DNS queries will favor load balancer IP addresses in their own Availability Zone, remaining resolves to any zone
Any Availability Zone (default)

In my opinion, it is safe to always set the LB to use the AZ affinity policy because the DNS queries will automatically resolve to other healthy LB IP in other zones when the one in the same zone is down.

Load Balancer Nodes to ingress-nginx pods

Once the traffic reaches the LB node, the LB now have to decide to which target to send the traffic to. To improve our reliability, we have the option to enable cross-zone load balancing. With this enabled, the LB node can also send the traffic to targets in other zones. It is disabled by default.

There is a tradeoff between reliability and cost here. If cross-zone load balancing is disabled, you have to make sure that the target(s) in each zone are healthy. If there are no healthy target in the zone, the request might fail. This means your service can be less reliable but if you enabled cross-zone load balancing you might incur cross-AZ traffic cost. So do your research and decide what's best for you.

ingress-nginx pods to backend services pods

Kubernetes 1.31 Service traffic distribution

After the ingress-nginx pods, the traffic will be routed to the backend services pods. Kubernetes version 1.31 introduced a new traffic distribution mechanism for Service resources which will influence how traffic is routed to your pods. You can now set the Service .spec.trafficDistribution to PreferClose to route the traffic to endpoints that are "topologically proximate". The details of what that means depends on the implementation but for kube-proxy this means sending the traffic to endpoints that are within the same zone when available. This means we can achieve our goal here to avoid the cross-AZ traffic cost. To understand more please refer to the Kubernetes documentation and KEP-4444.

CAVEAT: as mentioned in the Risks and Mitigation section of KEP-4444, enabling this feature might cause the pods in one AZ getting overloaded with traffic if the originating traffic is skewed towards one AZ.

ingress-nginx default routing behaviour

NOTE: please be extra careful with this section as this influences the routing of traffic to pods within your cluster. Do test this out on a non-production environment and make sure your workloads are still running fine before introducing it in production.

That alone is not enough though due to how ingress-nginx works. In the Ingress configuration, you specified the Service that ingress-nginx should route the traffic to but I was surprised to know that, by default, ingress-nginx does not actually uses the Service ClusterIP. It will actually search for the Endpoint of the services and get the IP of the pods behind the service. Then, it will distribute the traffic to the pod IPs using the round robin algorithm.

To leverage the newly introuduced traffic distribution feature I mentioned above, we need to make ingress-nginx routes the traffic using the ClusterIP of the Service. To do this globally for the ingress controller we can set service-upstream: true in the ingress-nginx configmap. This alone is not enough though because by default ingress-nginx tries to keep a long connection between the ingress-nginx pods and the backend services pods to reduce resource usage using keepalives.

To configure ingress-nginx not to use this, you can add another config upstream-keepalive-requests: "0" to the ingress-nginx configmap but beware that this might increase the resource of your ingress-nginx pods as it now needs to maintains a new connection for each requests coming in.

Conclusion

With the above steps, you should be able to avoid cross-AZ traffic in your AWS network for workloads hosted using NLB, EKS and ingress-nginx. Since this optimization might affect your production system, please make sure you test properly before rolling it out.

Exploring AWS ALB for EKS

Thu, 12 Dec 2024 09:07:00 GMT

Recently, my company got DDoS'ed and during the attack I noticed that the first thing that went down is the ingress-nginx pods. With this information, I was thinking if I could eliminate ingress-nginx and rely on AWS load balancers directly for routing the traffic to our services. AWS load balancer comes in two flavors: the L4 Network Load Balancer (NLB) and the L7 Application Load Balancer. The NLB doesn't support the same feature as ingress-nginx for routing HTTP requests as it operates on the L4 layer only so we'll be looking at the ALB in this article.

Current Setup

Our current setup uses a AWS NLB per country to accepts the connection from the internet. The NLB then forwards the requests to a target group consisting all the ingress-nginx controller pods. The ingress-nginx pods will then route the traffic from NLB to the backend services based on the configured Ingress configurations.

For arguments sake, let's assume that AWS NLB is reliable and can scale infinitely. Our bottleneck then is the ingress-nginx pods. We can setup the ingress-nginx pods to autoscale based on the traffic but I've seen issues with connections getting disrupted when the autoscaling happens. What if we can eliminate this bottleneck altogether and just route directly from AWS LB to our backend services?

Proposed Setup

Our proposed setup uses the ALB directly and route traffic to the backend services through its listener rules. By default the aws-load-balancer controller will provision one ALB for each Ingress resource but we can share the ALB for multiple Ingress by specifying the same alb.ingress.kubernetes.io/group.name. In my case there will be one group name for each country resulting in separate ALB created for each respectively. Each Ingress resource will create a separate listener rule based on the host and path configuration specified in the Ingress.

ALB Quotas and Limitations

Since we now rely on AWS ALB directly to route requests to our backend services, we must pay attention to the limitations set by AWS. For ALB there is a soft-limit of 100 rules per ALB. Assuming we need one listener rule per backend service, the maximum number of backend services that can be served by one ALB is 100. There are workarounds you can do to overcome this grouping separate backend services together by team or other attributes and each will get one ALB.

Pricing: ALB vs NLB

This is the part that I'm most interested in: how much more will it cost us if we migrate to ALB?

Based on the AWS ELB pricing page, the per-hour cost of the LB instance itself is the same for ALB and NLB. What differs is the (N)LCU-hour cost. ALB LCU-hour is around 30% more expensive than NLB. It might seem like that's the only difference but if you read further on the ELB pricing page you will notice that an LCU-hour for ALB is not the same as NLB.

Rule Evaluations

For ALB, the LCU-hour has an extra dimension measured which is the rule evaluations. You get 10 free rules per ALB. The formula given for calculating this dimension is Rule evaluations = Request rate * (Number of rules processed - 10 free rules). Since ALB adds new rule for each Ingress and having separate path for defined in the Ingress spec will also create new rule, your LCU-hour cost might increase the more Ingress and paths you have in your cluster.

New Connections

Another difference is the included new connections count per second. For NLB there are different values depending on whether you're using TCP, UDP, or TLS but for our comparison with ALB, lets look at the TLS pricing.

NLB with TLS includes 50 new TLS connections per second while ALB only includes half of that amount. We already calculated that ALB LCU-hour is already 30% more expensive than NLB but assuming you have the same amount of new connections, ALB will incur more LCU-hour than NLB.

To get a better comparison let's calculate the cost per connection for each LCU-hour. For NLB this is $0.006/50 connections = $0.00012 and for ALB it's $0.008/25 connections = $0.00032. So, comparing cost per connection, $0.00032/$0.00012, ALB is 2.7x more expensive than NLB.

LCU-hour pricing

You are charged only on the dimension with the highest usage.

LCU-hour is charged based on the highest dimension from all the dimensions measured so this means it depends on which is higher for both the rule evaluations and new connections dimension above. Let's say you don't have that many services so your listening rules also not that many causing you to not exceed the rule evaluations per second for ALB, you will still be charged 2.7x more compared to when you're using NLB.

Conclusion

I started this thought experiment thinking it might be better for us to migrate to ALB and eliminate ingress-nginx but after doing this research I think for our current workload we are better suited sticking with NLB + ingress-nginx. Hope this is useful for others too.

Docker's lesser known command: docker buildx bake

Fri, 15 Nov 2024 16:52:00 GMT

Have you seen someone use docker buildx bake before? Me neither... until I need to build a multi-platform image for our services. In this blog post I'll walk through the reason why I ended up being forced to use docker buildx bake.

Background

I am on a journey to run our services on Graviton on AWS and its using an arm64-based CPU. To make this migration smooth I decided there will be a transitionary period where there might be both architectures running on separate environments. This means I need to make sure that we're building both the linux/amd64 and linux/arm64 variant of the images.

We're using one monorepo per team for our services (don't ask me how we got there). For each monorepo there will be codebases for multiple services side by side and the Dockerfile are also managed together in one folder. To avoid code duplication, we have one base.Dockerfile that will generate a local base image which will be referred to when building other services.

For reference, this is how the files look like:

# ./cicd/docker/base.Dockerfile
FROM node:21-alpine as base
...
# installs all the common dependencies
RUN pnpm install --production
...

# ./cicd/docker/serviceA.Dockerfile
FROM node:21-alpine
WORKDIR /app

# copies the dependencies from base image into current folder
COPY --from=base /usr/src/app /app
...

# ./cicd/docker/docker-compose.yaml
services:
  base:
    build:
      dockerfile: ./cicd/docker/base.Dockerfile
  serviceA:
    build:
      dockerfile: ./cicd/docker/serviceA.Dockerfile
  serviceB:
    build:
      dockerfile: ./cicd/docker/serviceB.Dockerfile

When building the service we run the following command:

$ docker compose -f ./cicd/docker/docker-compose.yaml build base
$ docker compose -f ./cicd/docker/docker-compose.yaml build serviceA

This will first build the base image and then use that image to build serviceA. It works without issue.

Adding Multi-platform support

The Docker Compose specification supports specifying the we want to build for using the platforms key. So, adding that to our docker-compose.yaml, it'll now look like this with multi-platform support:

# ./cicd/docker/docker-compose.yaml
services:
  base:
    build:
      dockerfile: ./cicd/docker/base.Dockerfile
      platforms:
        - linux/amd64
        - linux/arm64
  serviceA:
    build:
      dockerfile: ./cicd/docker/serviceA.Dockerfile
      platforms:
        - linux/amd64
        - linux/arm64
  serviceB:
    build:
      dockerfile: ./cicd/docker/serviceB.Dockerfile
      platforms:
        - linux/amd64
        - linux/arm64

Now let's run the same docker-compose build command like before and we get...

[+] Building 0.0s (0/0)
Multi-platform build is not supported for the docker driver.
Switch to a different driver, or turn on the containerd image store, and try again.
Learn more at https://docs.docker.com/go/build-multi-platform/

Damn it.

Using the docker-container build driver

After some reading, I learnt that the default build driver when you use docker is the docker driver which doesn't have support for multi-platform images. You can read more on Docker build drivers on this page.

I need to use the docker-container driver which has support for building multi-platform images. To do that I need to create a new builder using the following command:

$ docker buildx create --driver docker-container --name multiplatform --use
$ docker buildx install

The first command creates the builder using the docker-container driver and sets it as the default while the second command creates a shell alias so that I can just use docker build instead of having to specify docker buildx build on the CLI.

Now, I should be able to run my docker-compose command right...?

# use `--builder multiplatform` to tell docker-compose to use the docker-container builder we just created
$ docker compose -f ./cicd/docker/docker-compose.yaml build --builder multiplatform serviceA
...
failed to solve: base: failed to resolve source metadata for docker.io/library/base:latest: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Now the builder cannot find the base image that we've built used before. Why? This section from the Docker build drivers page answered it:

Unlike when using the default docker driver, images built using other drivers aren't automatically loaded into the local image store. If you don't specify an output, the build result is exported to the build cache only.

I did use the --load and the --driver-opt default-load=true to automatically load the image into the local image store but long story short, it didn't work. So what's next?

Enter docker buildx bake

At this point, I've almost exhausted all my options and just browsing through the Docker documentation in hope of something and I found it!

At first the docker buildx bake command just looks like a different syntax for specifying the docker compose file to me but when my eyes caught on to one of the properties: target.contexts. It allows you to pass in more contexts in addition to the folder content context that we're used to with normal docker build that can be used in the Dockerfile.

These are the things you can specify under the target.contexts property:

Container image: docker-image://alpine@sha256:0123456789
Git URL: https://github.com/user/proj.git
HTTP URL: https://example.com/files
Local directory: ../path/to/src
Bake target: target:base

The first four are cool but the last one stood out to me: Bake target: target:bake.

Bake 101 crash course: remember in docker-compose.yaml we have services?

# ./cicd/docker/docker-compose.yaml
services:
  base:
    build:
      dockerfile: ./cicd/docker/base.Dockerfile
  serviceA:
    build:
      dockerfile: ./cicd/docker/serviceA.Dockerfile
  serviceB:
    build:
      dockerfile: ./cicd/docker/serviceB.Dockerfile

In Bake format, those services are called targets. Let's recall my original docker-compose.yaml file again, I have a base service used to build the shared image used in the Dockerfiles of serviceA and serviceB. In other words, the base service is also a target. This means, I can pass it on as extra contexts to my build!

Putting it all together

The Bake specification allows you to write the file in 3 languages: HCL (Terraform, anyone?), JSON, and YAML (through docker-compose.yaml syntax). To be less disruptive I chose YAML. To use Bake specification with YAML, the CLI can parse existing docker-compose.yaml files but for Bake-specific syntax you have to put it under the property x-bake. We'll also add the platform properties under the x-bake property. This is how the final docker-compose.yaml looks like in my case:

# ./cicd/docker/docker-compose.yaml
services:
  base:
    build:
      dockerfile: ./cicd/docker/base.Dockerfile
      x-bake:
        platforms:
        - linux/amd64
        - linux/arm64
  serviceA:
    build:
      dockerfile: ./cicd/docker/serviceA.Dockerfile
      x-bake:
        contexts:
            base: target:base
        platforms:
        - linux/amd64
        - linux/arm64
  serviceB:
    build:
      dockerfile: ./cicd/docker/serviceB.Dockerfile
      x-bake:
        contexts:
            base: target:base
        platforms:
        - linux/amd64
        - linux/arm64

To run the build we need to use our friend docker buildx bake:

$ docker buildx bake --file ./cicd/docker/docker-compose.yaml serviceA

DONE

Finally, we managed to build a multi-platform image using docker buildx bake!

well ackshually

you can just use docker multi-stage build

I know. You're right. I chose not do that to avoid making big changes to the code.

Just let me suffer.

Recap

TLDR here's what I had to do to get multi-platform build working when using multiple Dockerfiles:

Add the platforms section to your service in docker-compose.yaml
Setup a new Docker builder using the docker-container driver
Add the base image as extra contexts using Bake x-bake syntax
Build the image using docker buildx bake

This use case is quite niche tbh and like I've said it is avoidable by using multi-stage builds but it is what it is. There's other use cases for Bake as described in this guide that is more practical. It's worth a read IMO. Hope you've learnt something new as I have.

Using DuckDB to analyze NGINX logs

Sun, 27 Oct 2024 10:15:00 GMT

As part of my recent DDoS mitigation effort, I had to go through millions of nginx logs to identify patterns that I can use to further improve our custom WAF rules on Cloudflare. This article shows what I did to be able to run some analysis on the logs using DuckDB.

Making the hard things easy

Changing the log format

"To solve a problem that is difficult, you must first make it easy."

The default nginx access logs format looks something like this:

'$remote_addr - $remote_user [$time_local] ''"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"'

Parsing it is definitely possible using regex as shown by this article but why bother when you have the option to change the format and make it easier to parse using DuckDB. To do so we will be configuring ingress-nginx controller in our cluster to log in JSON format. You can set the custom log format using the log-format-upstream config and set log-format-escape-json to make sure that the variables are escaped properly for use in as JSON variables.

This is the log format that I'm currently using:

'{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName": "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus": "$upstream_status", "upstreamAddr": "$upstream_addr","method": "$request_method", "host": "$host", "uri": "$uri", "uriNormalized": "$uri_normalized", "uriWithParams": "$request_uri", "status": $status,"requestSize": "$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent", "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time s", "protocol":"$server_protocol"}'

Mapping customer IDs to generic placeholder

One thing I don't know how to do with DuckDB is normalizing URIs. Given a URI path containing user ID 12345678 like /users/1234567/info. How do I group by path where I ignore the middle section and group it as if all the URIs are like /users/:userId/info. I tried regex and patterns but couldn't get it to work. If you know how to do it please DM me on Twitter, I'd really appreciate it.

I found a way to do it in nginx instead. It works but its definitely not scalable. I use the ngx_http_map_module module to match URIs with certain paths and then convert it to a generic version of that path.

map $uri $uri_normalized {
    "~^/user/(.*)$" "/user/:ID";
    default $uri;
}

This snippet will do the following: for each of the variable $uri, map it to a new variable $uri_normalized. When $uri value matches the regex ^/user/(.*)$, the replace the value with new value /users/ID. If no matching regex found, then use default to the value of $uri.

With those configured, you can proceed to ship the logs to your logging backend of choice to retrieve later.

Fetching the logs from Loki

I'm using Loki as my logging backend. Loki provides an API endpoint you can use to fetch the logs and to make things easier they also have an CLI tool called logcli.

To fetch all nginx logs from my production cluster, this is the command that I used:

$ logcli query -oraw --from="2024-10-24T00:00:00+08:00" --to="2024-10-24T22:00:00+08:00" --part-path-prefix="logs" --parallel-max-workers=100 --parallel-duration=15m '{namespace="ingress-nginx", cluster="production"} | json | __error__=``'

-oraw is to set the output format of the logs. I'm using the raw format here so that I can get the same JSON input that I sent to Loki
--from and --to are self-explanatory but I did have some problem specifying the correct format that the tool will accept and the official docs was quite confusing
--part-path-prefix use this prefix to name the files when downloading multiple files in parallel
--parallel-max-workers sets the max parallel workers to be used. Note that the actual workers used depends also on the available tasks based on the parallel duration configured
--parallel-duration is the duration size to use for each file. combined with the --from and --to option, this option determines how many files will be created e.g. for 1 hour duration and you've specified the --parallel-duration there will be 4 files created, each containing logs from specific 15 minutes section.

Loading the logs into DuckDB

I'm using the CLI version of duckdb but you can also do this using the embedded library in other languages of your choice.

To read the files we've pulled from Loki and create a table with it I used this command:

create table logs as select * from read_json('logs_*.part', format='auto', columns={timestamp: 'TIMESTAMP', method: 'VARCHAR', host: 'VARCHAR', uri: 'VARCHAR', uriNormalized: 'VARCHAR', status:'INT', remoteIp: 'VARCHAR', is_authed: 'BOOLEAN', userAgent: 'VARCHAR'});

read_json can automatically create all the columns based on the keys in the JSON files but I want it to treat certain columns as specific data types so that's why I'm specifying the columns manually here.

After loading the columns into the table you can check the number of rows using this commands:

select count() from logs;

In my case, for a day's worth of logs from nginx, I have around 60 million rows. On idle, its using around 8GBs of RAM. If you have longer periods of logs to analyze, then definitely opt for a beefier machine or else DuckDB will crash.

Running queries

Its the fun part. Here's some queries that I find useful:

Highest requests per minute grouped by IP

I use this information to set the proper value to use in our Cloudflare rate limiting rule. By looking at existing request rate I'm reducing the chance of rate limiting our actual customers.

select
        (hour(timestamp) + 8) % 24 as hour,
        minute(timestamp) as minute,
        remoteIp,
        count() as total
from logs
where remoteIp not in ('X.X.X.X', 'Y.Y.Y.Y', 'Z.Z.Z.Z') --production NAT gateway IPs
and remoteIp not like '162.158.192.%'   --cloudflare IPs
group by all
order by total desc
limit 5;

IP address with the highest number of request to a particular host

select remoteIp, count() as total from logs where host = 'subdomain.example.com' group by all order by total desc limit 10;

Checking the paths an IP have been sending requests to

I'm including the result from the query here since it shows that this particular IP most likely are using a scanner to find vulnerable endpoints on our service.

select uri, count() as total from logs where remoteIp = '152.32.189.70' group by all order by total desc limit 20;
┌─────────────────────────────────────────────────────────────────┬───────┐
│                               uri                               │ total │
│                             varchar                             │ int64 │
├─────────────────────────────────────────────────────────────────┼───────┤
│ /                                                               │    10 │
│ /favicon.ico                                                    │     6 │
│ /api/user/ismustmobile                                          │     6 │
│ /h5/                                                            │     4 │
│ /m/                                                             │     4 │
│ /api                                                            │     4 │
│ /app/                                                           │     3 │
│ /api/config                                                     │     3 │
│ /leftDao.php?callback=jQuery183016740860980352856_1604309800583 │     2 │
│ /public/static/home/js/moblie/login.js                          │     2 │
│ /static/home/css/feiqi-ee5401a8e6.css                           │     2 │
│ /client/static/icon/hangqingicon.png                            │     2 │
│ /admin/webadmin.php?mod=do&act=login                            │     2 │
│ /static/images/auth/background.png                              │     2 │
│ /index/index/home?business_id=1                                 │     2 │
│ /stage-api/common/configKey/all                                 │     2 │
│ /Public/home/common/js/index.js                                 │     2 │
│ /ws/index/getTheLotteryInitList                                 │     2 │
│ /app/static/picture/star.png                                    │     2 │
│ /resource/home/js/common.js                                     │     2 │
├─────────────────────────────────────────────────────────────────┴───────┤
│ 20 rows                                                       2 columns │
└─────────────────────────────────────────────────────────────────────────┘

Conclusion

This has been an adhoc task and when I was told to do some analysis on the logs I immediately think of the tools I'm familiar with which is DuckDB to do the analysis. I'm aware there are better tools out there and we are currently evaluating using OpenSearch Security Analytics to automatically do this kind of detection in the future. Hopefully, I can talk about it soon. If any of you data analyst/data engineer out there got better ways to do the things I'm doing feel free to tweet me @pokgak73 on Twitter.

We got DDoSed

Sun, 27 Oct 2024 06:37:00 GMT

Recently at $work we've been hit by a series of DDoS attacks. In this post, I'm gonna describe the steps we've taken to protect our services from these attacks in the future and also what works and what don't.

Detection

The first attack was around 10PM on a Sunday. I was at home at the time and was notified that our ingress-nginx pods were repeatedly crashing. I've seen this happen before and my initial thought was that our service was getting more customers this night so I added more nodes into our cluster and increased the replica count for the ingress-nginx pods. That did nothing. Our pods are still crashing and customers still cannot use our service.

Then, one of my colleagues showed me the metrics for new connections to our load balancer. We're getting 10x our usual traffic in a minute. That's a DDoS for sure.

Rate limiting from the ingress-controller

Ingress-nginx supports a whole set of rate limiting features. So we decided to start there but we first need to know which endpoint is being hit. For this, the logs from the ingress-nginx pods was really helpful. The incoming logs was coming in fast but we can see that most of the log lines contain the same hostname. So we put the nginx.ingress.kubernetes.io/limit-rps annotation to the ingress that is used by that hostname. Then, we wait and monitor whether this annotation helps stabilize our crashing pods. It didn't.

From the logs, I can see that ingress-nginx is rate-limiting the request but because it still has to process the request first and keep counting it, it cannot keep up with the amount of incoming traffic. Rate limiting on the ingress-nginx level is useless because our infra is not enough to even receive and block all the request. We have an option to add more replicas to our ingress-nginx pods but thats gonna be costly. So, we started looking for other solutions.

Cloudflare to the rescue

I've always have known that people uses Cloudflare to protect against it I've never done it myself personally. After the initial response was proven not effective, I remember we're using Cloudflare to point the domain to our load balancer but why does Cloudflare not blocking all this DDoS traffic?

Our first mistake was that we didn't proxy the traffic through Cloudflare.

After proxying the traffic through Cloudflare though, there is a delay to when all the traffic will be routed to Cloudflare Anycast IP due to the TTL settings which is 300 seconds by default. So, again, we wait...and the request still coming in after a while. Meanwhile, I've enabled the Under Attack mode but honestly I'm not sure if this actually helps. After a while the requests was still not going down. It might take a while before Cloudflare can kick in to automatically mitigate the attack. In the meantime, we need to do something.

Using the WAF Rules

On the WAF page on Cloudflare, there's three types of rules you can configure: Custom rules, Rate limiting rules and Managed rules.

Managed rules

After enabling the Under Attack mode, we also enabled the Managed rules but it didn't help much in our case. Managed rules block commonly used attacks but it our case it doesn't block any of the DDoS attacks because the attack is targeting our API endpoint specifically with requests path that wasn't included in the managed ruleset. We leave it turned on regardless since it might protect against other attacks in the future.

Custom rules: blocking by country

After all the above doesn't seem to work, we need a way to differentiate DDoS traffic from valid traffic. I know that we only operate in several countries in Southeast Asia. This means that all the traffic that's coming from outside those countries are bots (read more below to see why this is not true). So, we added a Custom rule to only whitelist the traffic coming from the countries that we operate in and that works, sorta. The request hitting our LB reduced to around half but its still higher than usual and our ingress pods are still being overwhelmed.

Then, we noticed from the Security Analytics page in Cloudflare that the requests that are still hitting the LB passed the rule because its coming from one of the whitelisted countries. We can remove that country from our whitelist but that also means that we'll be blocking valid traffic from our customers from those countries. So, we need to come up with new rule to block the DDoS traffic. What does a valid request have that the requests from the attacker doesn't have?

Custom rules: blocking using query params, headers, and user agent

We went through nginx logs and noticed that the requests from the attacker are always using the same query params so we created a new rule blocking rqeuests with that query params and it worked! The requests hitting our LB dropped back to normal levels and we declared the incident finished. This doesn't last long tho. The next time we were hit with the attack, we noticed that the attacker now uses a different query param. Luckily, we had also added other rules in place.

After the first attack, we analysed our valid requests and came up with other rules based on the Referer and the X-Requested-With headers. We also check the User-Agent and block if its similar to the one that came from the attacker based on past attacks. So far this has been the most effective at blocking the attack. However, we know that this is not the final solution. If the attacker is determined enough, they can still look at valid requests and then spoof the values in their attack but so far we haven't seen this happening yet.

Custom rules: blocking known attacker IPs

After several rounds of attacks we noticed that the DDoS attacks were all coming from the same set of IPs, so we created a list of known attacker IPs on Cloudflare and block future requests coming from those IPs. Looking back, this rule was only effective for a little while. Once the attacker noticed that all their requests were blocked, they will change the IP so the process will just keep repeating over and over again.

Rate limiting rule

After we put in the custom rules, we also turned on the rate limiting rules. Unless you are on the Enterprise plan (its expensive XXXX), the rate limiting rule is pretty restricted. You can only rate limit by IP but I think it is good enough. The rate limiting rules will act as the last line of defense after the requests passes all your other configured custom rules. Here is the rule execution order for your reference.

On a free plan, you get 10 second counting period but if you pay for other plans you'll get more options. To me, the bigger counting period helps prevent from blocking valid requests. There might be a burst of activity from your users that causes the IP to hit the rate limit within 10 seconds but if measured within a longer period its still within a normal range. So, paying more is definitely worth it here.

I definitely think that rate limiting rule is a must if you're fighting against DDoS. So, make sure to configure this.

Bonus

Blocking our own IPs

This is one of those facepalm moments in my life. After the attacks passed, I spent some time exploring the events in Cloudflare Security Analytics to see if I can find any insights. From the requests that were not blocked by Cloudflare, I grouped the requests by IP and checked the requests. Requests from the top two IPs looks good, it has all the headers and referers we were expecting them to have but the requests coming from a datacenter in Singapore. What makes it more suspicious was that all the requests had user agents from mobile devices eventhough the IP shows that they're coming from a datacenter. So, I informed my team and proceed to block the requests. After a while complaints started coming in saying our customers requests were blocked. One of my colleagues suspects that those are actually our IPs.

We have NAT Gateways configured in our network which means that if the requests were actually from our own network it will have one of those IPs from the NAT Gateway...and after comparing, they are indeed our NAT Gateway IPs. Apparently, one of the services proxies all the requests from the customers back to another service, complete with all the headers and user-agents that why it was showing mobile device user agents eventhough the IP was coming from a datacenter.

After this incident, we created a list on Cloudflare containing all our known IPs and skip blocking to avoid confusion in the future.

Blocking accessibility bots (from US)

We also put in place rate limiting rule for all the requests that was categorized as Verified Bots by Cloudflare. After putting in all these rules, I try to regularly review the block requests to make sure they're not false positives - valid requests that was blocked - to further optimize our rules. There was a bunch of requests coming from the US, which we already put in custom rule to block but after looking at their user agent it seems a bit unusual - it contains the string Google-Read-Aloud. After some googling, I found out that Google uses that user agent for their text-to-speech feature for accessibility purposes.

Having this user-agent does not necessarily mean that the requests are 100% valid because attackers can still spoof their user agent. So, I'll leave it up to you to decide if this is something that should be blocked but I think this is worth mentioning since in our fervor to prevent attackers from bringing down our systems we might also be hurting valid customers and affecting the accessibility of our service.

Silly mistake: external-dns

Fast forward a few days, we got hit again with a DDoS attack but this time it was during lunch time, which was the peak hour for our customers. I thought, "Did the rule from last time not working anymore?". We checked the Security Analytics page on Cloudflare and noticed that Cloudlfare wasn't blocking any requests even though we already had the rule from last time.

I scratched my head for a bit wondering what we missed when my colleague pointed out that the DNS record for that domain wasn't proxied. So, I turned it back on but after a while it was turned back off. Then, I remember that we're using external-dns to automate the creation of our records on Cloudflare and it was configured to disable the proxy option. So, whenever we enabled the proxy option, external-dns reverted it back as its supposed to. We end up turning off external-dns to make sure the proxy option would not get reverted. Read more below to know how you can turn on the proxy option on a per-ingress basis.

Using annotation to proxy traffic through Cloudflare per Ingress

To enable proxying requests through Cloudflare on a per-Ingress basis when using external-dns, you can add the annotation external-dns.alpha.kubernetes.io/cloudflare-proxied: "true" to the Ingress. If you have multiple Ingress that all points to the same host, make sure to add the annotation to all of them. If not, external-dns will keep fighting itself, turning the proxy option onn and off forever.

Conclusion

I've outlined several steps you can take if you're facing DDoS attacks in the future. Despite the success in mitigating the attacks so far, we know that there is no forever solution to DDoS. We have to keep up with the attacker and play Whac-A-Mole until they are bored and stop the attacks. When fighting DDoS attacks, I find it helpful to log the requests and review it regularly to avoid false-positives. It has been an eye opening experience for me and next time I'm asked how to protect from a DDoS, I can definitely say more than just put it behind Cloudflare.

Scrape cAdvisor using Grafana Alloy

Wed, 10 Jul 2024 09:45:00 GMT

I was having some issues figuring out how to scrape cAdvisor metrics using Grafana Alloy. After googling I came across this k8s-monitoring helm chart and inside there is a configuration for scraping the built-in cAdvisor on the k8s kubelet.

I ran Alloy as a single pod Deployment and it'll scrape all the nodes in the cluster. Here's the config that I used to get the metrics:

prometheus.remote_write "default" {
  endpoint {
    url = "https://mimir.example.com/api/v1/push"
  }
}

discovery.kubernetes "nodes" {
  role = "node"
}

discovery.relabel "cadvisor" {
  targets = discovery.kubernetes.nodes.targets

  rule {
    replacement   = "/metrics/cadvisor"
    target_label  = "__metrics_path__"
  }
}

prometheus.scrape "cadvisor" {
  job_name   = "integrations/kubernetes/cadvisor"
  targets    = discovery.relabel.cadvisor.output
  scheme     = "https"
  scrape_interval = "60s"
  bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  tls_config {
    insecure_skip_verify = true
  }

  forward_to = [prometheus.remote_write.default.receiver]
}

Alloy cadvisor exporter

Alloy provides the prometheus.exporter.cadvisor components that can be used to start a new cadvisor on the nodes. This is not required if the kubelet running on your nodes already runs cadvisor. This is the case for me on EKS running on Bottlerocket.

The hidden cost of running your own observability stack

Mon, 24 Jun 2024 05:00:00 GMT

At my latest $job, I was tasked of setting up the LGTM stack (Loki, Grafana, Tempo, Mimir) for observability. Fast forward a few months, I noticed there's a hidden aspect to running the stack that I was not expecting before and that is the network cost, specifically the network transfer cost for cross AZ traffic. At one point we were paying more than $100 per day just for the cross AZ network traffic.

Update: since this article was written, I've found out that the official Loki helm chart have a section addressing the cross-az issue in the values file. It does not recommend running Loki across multiple AZs on the cloud.

Note: This can be used to run Loki over multiple cloud provider availability zones however this is not currently recommended as Loki is not optimized for this and cross zone network traffic costs can become extremely high extremely quickly. Even with zone awareness enabled, it is recommended to run Loki in a single availability zone.

Cross AZ Traffic Amplification

While investigating where does the traffic coming from I compared the load balancer "Processed Bytes" metrics with the Cost Explorer usage for cross AZ traffic and noticed that there's a 10x increase in the reported values by the load balancer to the actual charged traffic. It baffled me a bit and made me step back and take a deeper look at the possible points where I'm getting charged.

Collector to load balancer node
load balancer node to ingress controller pod
ingress controller pod to distributor
distributor to ingester

Collector to Load Balancer Node: client routing policy

In my setup, the services are exposed through a load balancer and given a DNS name like loki.example.com. The collectors are configured to send the telemetry data to that URL. Here is my fist mistake, I didnt' enable "Availability Zone affinity" for the client routing policy. When enabled, this will route traffic from the collector to the load balancer node in the same AZ avoiding being charged for cross AZ traffic.

The load balancer node will then forward the traffic to the ingress controller pod in the same AZ.

Ingress controller pod to distributor: Kubernetes Topology Aware Routing

From the load balancer node, the traffic will be forwarded to the k8s pod through the k8s service. The default behavior of service in k8s is it will route the traffic using the round-robin algorithm. This means that from the ingress controller pod to the distributor pod the traffic will go cross AZ. If you have 3 distributor pods, this means 2 out of 3 connections will be routed to pods in different AZ.

To avoid the traffic from crossing AZ, we can use kubernetes topology aware routing feature. Downside of using this is that we need to have at least 3 pods in each AZ but compute is cheaper in my use case since I'm using spot instances through Karpenter and getting up to 70% discount on the node price.

Distributor to Ingester: no workaround

This the only part I haven't solved. In the LGTM stack, the distributors uses an internal discovery mechanism to get the IP of the ingesters. This means that we cannot use kubernetes topology aware routing here.

To make things worse, depending on the replication_factor configuration, each distributor might be sending the logs to multiple ingesters, each one multiplying the cross AZ cost that we have to pay.

Special use case: getting logs from external source

Other than the above use case, our company also have other use case where the logs comes from external sources instead of from our internal network. In this case, I actually managed to eliminate the cross AZ cost completely by deploying the LGTM stack in just one AZ. The load balancer is also configured to use only one subnet that is in the same AZ.

Conclusion

All the above factor might explain why the amount I was charged for cross AZ traffic is 10x bigger than the amount that is received at the load balancer. I outlined some the possible points where the cross AZ charges are coming from and how to fix it. Hope it helps!

Tech Janitor: Linux Cookbook

Fri, 26 Jan 2024 19:14:00 GMT

In this blog post I'll be listing out the tools that I frequently use in my day to day work but in a cookbook format where it's not just a random list of CLI tools but it is grouped by the goals that I'm trying to achive.

Probing an instance from the outside
Checking which service is running
Checking the logs
Monitor the system

Probing an instance from the outside

Check DNS record exists

## dig <domain> <record-type>

# when the record exists
$ dig pokgak.xyz +short
185.199.108.153
185.199.110.153
185.199.109.153
185.199.111.153

# when the record doesn't exist
$ dig pokgak.abc +short
# <empty>

Check if a port is open

If you're having trouble connecting to an instance, you can check if the port is open by using telnet. This can be used for all protocols that use TCP ie HTTP(80/443), SSH(22), Postgres(5432), Redis(6379) etc.

## telnet <host> <port>

# when the port is open
$ telnet nuc 22
Trying 100.102.147.64...
Connected to nuc.hamster-nase.ts.net.
Escape character is '^]'.
SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
^]
telnet> Connection closed.

# when the port is closed
$ telnet nuc 80
Trying 100.102.147.64...
telnet: connect to address 100.102.147.64: Connection refused
telnet: Unable to connect to remote host

Checking which service is running

Check which process is listening on any port

Sometimes you want to know which process is listening on a port. This can be done using ss. Make sure to use sudo or run it as root because it won't show the process name if you don't.

$ sudo ss -ntlp
State    Recv-Q   Send-Q                     Local Address:Port        Peer Address:Port   Process
LISTEN   0        4096                      100.102.147.64:63177            0.0.0.0:*       users:(("tailscaled",pid=771,fd=26))
LISTEN   0        128                              0.0.0.0:22               0.0.0.0:*       users:(("sshd",pid=719,fd=3))
LISTEN   0        32                            10.45.81.1:53               0.0.0.0:*       users:(("dnsmasq",pid=1047,fd=7))
LISTEN   0        4096                       127.0.0.53%lo:53               0.0.0.0:*       users:(("systemd-resolve",pid=607,fd=14))
LISTEN   0        4096         [fd7a:115c:a1e0::a166:9340]:63177               [::]:*       users:(("tailscaled",pid=771,fd=28))
LISTEN   0        128                                 [::]:22                  [::]:*       users:(("sshd",pid=719,fd=4))

Check systemd service status

If you're running a service using systemd, you can check the status of the service using systemctl.

## systemctl status <service-name>

Check if a process is running

This might be useful if you know what process/program name you're looking for.

## ps aux | grep <process-name>

$ ps aux | grep tailscale
root         771  0.3  0.2 1259760 43092 ?       Ssl  Jan25  11:45 /usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock --port=41641
pokgak     30489  0.0  0.0   7008  2304 pts/0    S+   12:43   0:00 grep --color=auto tailscal

Checking the logs

Check systemd service logs

journalctl is a tool that can be used to check the logs of a systemd service. This is really useful when you're trying to debug a systemd service that is not running or keep on failing.

## journalctl -fu <service-name>
$ journalctl -fu tailscaled

Check nginx logs

Some applications will write their logs to a certain folder. Usually on linux, the logs will be written to /var/log/<app-name>. For example, nginx will write its logs to /var/log/nginx.

tail will get the last 10 lines of a file. You can use -f to follow the file and print out the new lines that are added to the file.

tail -f /var/log/nginx/access.log

Monitor the system

Check CPU & Memory usage

htop is a tool that can be used to monitor the CPU and memory usage of a system. It's like top but with a better UI.

Check disk usage

df is a tool that can be used to check the disk usage of a system. -h is used to make the output human readable. In linux, your hard drive usually will be represented as /dev/sda or /dev/sdb or other letters of alphabet if you have more disks in the system. The prefix number on the disk is the partition number. For example, /dev/sda1 is the first partition of the first disk.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.6G  1.5M  1.6G   1% /run
efivarfs        128K   87K   37K  71% /sys/firmware/efi/efivars
/dev/sda2       457G   15G  419G   4% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda1       1.1G  6.1M  1.1G   1% /boot/efi
tmpfs           1.6G  4.0K  1.6G   1% /run/user/1000

Using Steampipe + DuckDB for VPC Flow Logs Analysis

Fri, 26 Jan 2024 12:00:00 GMT

As a so called Tech Janitor, I've been tasked to clean up one of our AWS accounts at work and that account have a bunch of EC2 instances that no one knows what they all do. So, I've decided to use one of AWS features, VPC Flow Logs, to first identify which EC2 instances are still being used and which are not.

Setting up the VPC Flow Logs and query using DuckDB

For our purpose, I've setup VPC flow logs to send all the traffic data to a S3 bucket that we'll refer to as vpc-flow-logs-bucket in this post. The flow logs are stored in a Parquet format for querying later using DuckDB.

Once the flow logs file are sent to S3, I'll be able to query them using DuckDB. To do that we will need to install the aws and httpfs extensions.

From DuckDB shell:

> INSTALL aws;
> INSTALL httpfs;

We also need to load our AWS credentials into DuckDB. Luckily, DuckDB has a built-in command to do that:

> CALL load_aws_credentials();
┌──────────────────────┬──────────────────────────┬──────────────────────┬───────────────┐
│ loaded_access_key_id │ loaded_secret_access_key │ loaded_session_token │ loaded_region │
│       varchar        │         varchar          │       varchar        │    varchar    │
├──────────────────────┼──────────────────────────┼──────────────────────┼───────────────┤
│ <redacted>           │ <redacted>               │                      │ eu-west-1     │
└──────────────────────┴──────────────────────────┴──────────────────────┴───────────────┘

This will look for your AWS credentials based on the standard AWS credentials file location. If you have multiple profiles in your credentials file, you can specify which profile to use by passing the profile name as an argument to the load_aws_credentials function.

Now it's time to load our VPC flow logs from S3 into a table in DuckDB. You can replace the year/month/day/hour with the actual date and hour of the flow logs that you want to load or use * for any or all of them to load all the flow logs. I'll be loading all the flow log records into a table flow_logs in DuckDB.

This might take a while since DuckDB will have to download the Parquet files from S3 and load them into memory. It took several minutes to finish loading in my case.

> CREATE TABLE flow_logs AS SELECT * from read_parquet('s3://vpc-flow-logs-bucket/AWSLogs/<aws-account-id>/vpcflowlogs/<region>/<year>/<month>/<day>/<hour>/*.parquet')

Now we can see that the flow logs records only contains the network interface ID (ENI) of the EC2 instance but not the EC2 instance ID or name itself. That won't be enough for my use case since I want to identify which traffic is flowing to which EC2 instance. Therefore, we need to correlate the ENI with the EC2 instance ID and here's where Steampipe comes in.

Steampipe: directly query your APIs from SQL

Steampipe is a tool that allows you to query APIs from SQL. It supports a lot of different APIs from AWS, GCP, Azure, Github, etc. You can also write your own plugins to support other APIs. I'll be using it to query my AWS account for the EC2 instance ID and name based on the ENI ID from the VPC flow logs.

Life before Steampipe

Usually to do the things I'm about to show below, I'll pull the data from AWS using the aws-cli and then massage it using jq/yq/awk/sed, if I'm desperate maybe Python. Then I'll use some other tools to visualize it or export to CSV. With Steampipe, pulling the data from AWS is so simple and using SQL to correlate the data with other information source is a breeze.

Steampipe is just Postgresql

Under the hood, Steampipe is running PostgreSQL and it even allows you to run it as a standalone instance running in the background and allows connecting to it from any third-party tools that can connect to a Postgresql instance. Here's where it gets interesting, DuckDB has the capability to connect to any PostgreSQL database through the Postgresql extension and query it as if all the data inside that database is coming from the DuckDB. This means that we can use Steampipe as a data source for DuckDB and access all of the AWS resources data available in Steampipe.

Setting up Steampipe and DuckDB connection

To run Steampipe as a service mode, you'll need to run the following command to start the PostgreSQL instance and get the credentials for connecting to it:

$ steampipe service start
Database:

  Host(s):            127.0.0.1, ::1, 2606:4700:110:8818:e17b:f78c:6c52:dccb, 172.16.0.2, 2001:f40:909:8e2:207a:634a:2070:d99d, 2001:f40:909:8e2:1cdb:75da:2a70:4b05, 192.168.100.23, 127.0.2.3, 127.0.2.2
  Port:               9193
  Database:           steampipe
  User:               steampipe
  Password:           ********* [use --show-password to reveal]
  Connection string:  postgres://steampipe@127.0.0.1:9193/steampipe

Then inside DuckDB shell, you can connect to the Steampipe PostgreSQL instance using the following command:

> ATTACH 'dbname=steampipe user=steampipe password=23e2_4853_bd96 host=127.0.0.1 port=9193' AS steampipe (TYPE postgres);
> use steampipe.aws;
> SHOW tables;
show tables;
┌─────────────────────────────────────────────┐
│                    name                     │
│                   varchar                   │
├─────────────────────────────────────────────┤
│ aws_accessanalyzer_analyzer                 │
│ aws_account                                 │
│ aws_account_alternate_contact               │
...

Now we can see all the tables from the Steampipe Postgresql instance. For my use case I'll be using the aws_ec2_network_interface table which contains both the network interface ID (ENI) and the EC2 instance ID that I can use JOIN together with the VPC flow logs records to map the records to the EC2 instance ID.

JOIN-ing it all together

Here's an example query that will give me the count of all incoming traffic to the instances grouped by the port number:

select
    i.title,
    fl.dstport,
    count(fl.dstaddr) traffic
from aws_ec2_network_interface ni
left join memory.flow_logs fl on fl.interface_id = ni.network_interface_id
left join aws_ec2_instance i on i.instance_id = ni.attached_instance_id
where
    fl.dstaddr = i.private_ip_address
group by i.instance_name, fl.dstport
order by traffic desc, dstport asc

From this information I'll be able to guess which service is running on those instances and take the next step towards migrating or depecrating the instances.

Conclusion

It is kinda mindblowing that I can do all this using SQL. Both Steampipe and DuckDB are great products and the flexibility of those tools allows me to pick and choose the best tool for the job. I first came across Steampipe in one of the podcasts that I listen to but haven't really used it much. Now, after having the opportunity to use it to solve one of my problems, I'll definitely pay more attention to it to make my tech janitor life easier in the future ;)

Terraform modules: be opinionated

Fri, 08 Dec 2023 09:21:59 GMT

Modules are containers for multiple resources that are used together.

Terraform modules is a way to bundle a bunch of Terraform resources into one group. Although not explicitly mentioned in the definition, it is also a way to provide an abstraction to the resources inside the module and only expose inputs and outputs that are relevant to the users of the module.

Terraform Module as an Abstraction Layer

A Terraform resource usually tends to be generic in that it allows you to configure it multiple ways through the input variables that it can accept. For example, the aws_mskconnect_connector resource has 3 options for log delivery: CloudWatch Logs, Kinesis Data Firehose, or S3. In most cases, your module shouldn't expose all 3 options to your users. Your organization probably already has some standard place where you send you logs to e.g. S3 for example where it gets forwarded to another logs search service for later use.

Therefore, your module should only expose the S3 option and leave out CloudWatch Logs and Kinesis Data Firehose from your module. By doing this you eliminate the choice from the user and they don't have to think which one to use. Your Terraform code can be a lot simpler too.

Module Abstraction in Action

Here's an example module code when including all 3 options:

variable "log_delivery_s3" {
  type = object({
    bucket_arn = string
    prefix     = optional(string, "mskconnect-logs")
  })

  default = null
}

variable "log_delivery_cloudwatch_logs" {
  type = object({
    log_group = string
  })

  default = null
}

variable "log_delivery_firehose" {
  type = object({
    delivery_stream = string
  })

  default = null
}

resource "aws_mskconnect_connector" "example" {
  name = "example"

  log_delivery {
    worker_log_delivery {
      dynamic "s3" {
        for_each = var.log_delivery_s3 != null ? [1] : []

        enabled    = true
        bucket_arn = var.log_delivery_s3.bucket_arn
        prefix     = var.log_delivery_s3.prefix
      }

      dynamic "cloudwatch_logs" {
        for_each = var.log_delivery_cloudwatch_logs != null ? [1] : []

        enabled   = true
        log_group = var.log_delivery_cloudwatch_logs.log_group
      }

      dynamic "firehose" {
        for_each = var.log_delivery_firehose != null ? [1] : []

        enabled         = true
        delivery_stream = var.log_delivery_firehose.delivery_stream
      }
    }
  }
}

And here's an example when we only offer S3 option:

variable "log_delivery_s3" {
  type = object({
    bucket_arn = string
    prefix     = optional(string, "mskconnect-logs")
  })

  default = null
}

resource "aws_mskconnect_connector" "example" {
  name = "example"

  log_delivery {
    worker_log_delivery {
      s3 {
        enabled    = var.log_delivery_s3 != null
        bucket_arn = var.log_delivery_s3.bucket_arn
        prefix     = var.log_delivery_s3.prefix
      }
    }
  }
}

See how simple and shorter the code become? No need to use those dynamic blocks just to conditionally add or remove the log delivery option block anymore.

But I might need to use that other option in the future...

YAGNI.

Access internal kubernetes services from anywhere using Tailscale

Mon, 11 Sep 2023 18:05:00 GMT

I recently setup a local kubernetes in my home network to play with and one of the issues that I faced is that it is hard to access the services inside the cluster from my laptop. I don't have a load-balancer in my setup so everytime I want to access a service from my laptop, I'll have to run kubectl port-forward first before using the localhost address to access it. It works but it's annoying.

Usually in cloud environments like AWS, you would setup an ingress-controller that will provision a load balancer for you and use that load balancer to expose your services inside the cluster to the internet using Ingress resources. Your incoming traffic from the internet will then be routed through the load balancer into your cluster onto your pods. Unfortunately, you don't get the same thing when hosting your cluster locally outside of the cloud environment. You have to manually configure your network to allow access from the internet.

So what options do we have?

One way we can do it is to use a Service with type NodePort to use the host port and access the pods using the host IP, this will allow access to services inside the cluster to your local network but still not from the internet. To allow access from the internet, you'll have to open a port on your router to route to the host IP from the NodePort service.

I'm not a fan of opening my home network to the internet. Home routers is infamous for being vulnerable and easily exploitable. I don't want mine to be part of a new legion of botnets that will break a new record for biggest DDoS attack. Using the NodePort service type also is not that great. With NodePort service, you'll have to specify the node IP to access along with the port assigned and your traffic will always go to that node and the pods running on it. More reason on why NodePort is a bad idea on StackOverflow.

What other option do we have? Tailscale!

Tailscale Subnet Router

Tailscale is a mesh VPN built on top of Wireguard. I've been using it for a long time for accessing my personal servers at home while I'm outside and I love it. It is so simple to setup you don't have to know any networking magic to use it. Tailscale will create a peer-to-peer network from your client to your other Tailscale devices and it is also really smart in figuring out a way to punch a hole through your home network (see the Resources section) to connect to the internet so you don't have to open a port on your home router anymore. Bot legion problem solved!

One of the ways you can use Tailscale is by configuring a Tailscale node as a subnet router. Usually, when you have 10 devices in your network, you'll have to install Tailscale on each of those devices to connect it to your VPN network but with a subnet router only one Tailscale node in that network is enough, as long as that subnet router node have network access to all the devices in that network. You'll have to configure your subnet router to advertise the route of the internal cluster network that the subnet router is in using CIDR range e.g. 10.43.0.0/16 so that other devices outside of that network will know to look for the subnet router if they want to access the IP address from that CIDR range.

ELI5: subnet router advertisement

You're a postman trying to deliver a parcel. Your parcel destination is set to unit A-1-2-3 in the TRX Exchange 106 building. You've never been to TRX before so you don't know which floor the office actually is but you noticed there's a big signboard at the reception saying "Come here if you have parcel for unit A-1-0-0 to A-9-9-9". So, you went the reception and then the nice lady at the reception gave you the direction to reach the office unit A-1-2-3 for you to deliver your parcel.

The reception here is like our subnet router. All the traffic meant for the network have to go through the subnet router first, then they're passed through to the actual packet destination.

I don't want to remember all this IP addresses

With a subnet router, you can now reach any of the services inside the cluster using the ClusterIP of that service but IP address is not human-friendly and you don't want to (you can't!) memorize all the IP addresses for all the services inside the cluster. So, now we need something that will map our IP addresses to a human-friendly format. Sounds familiar? We can use DNS records.

You can definitely create DNS records manually and map it to each of the ClusterIP for your services. That's what I did for testing when validating this setup actually. At scale, that won't work tho. You don't want to be the one to manually go to your DNS registrar and create the records one by one. Luckily, in kubernetes there is an application called external-dns.

external-dns to the rescue

external-dns is an application that runs inside your kubernetes cluster and it periodically queries the kubernetes API for the list of all Service and Ingress resources. From the list, it checks whether it should create a DNS record for the resources based on the resource annotation. It supports a lot of DNS providers like AWS Route53, Cloudflare, Google Cloud DNS and more. For my setup I'm using Cloudflare.

By default, external-dns will only create DNS records for Ingress resource or Service with type LoadBalancer. For my setup, since I'm self-hosting the cluster inside my home network and don't have access to a load balancer, I have to add an extra configuration parameter to external-dns so that it will create DNS records for ClusterIP Service type. On the Service resource itself, usually external-dns searches for the external-dns.alpha.kubernetes.io/hostname annotation but since we're using it with ClusterIP, I have to change it to external-dns.alpha.kubernetes.io/internal-hostname.

Tailscale + external-dns = ❤️

➜ dig prometheus.k8s.pokgak.xyz +short
10.43.170.163

With those changes applied. All new Service resources in my kubernetes cluster that have the annotation will get one DNS record on Cloudflare. Now, if I try to resolve a name for a service inside the cluster, it will return me an internal ClusterIP. Combined with the Tailscale subnet-router we've configured earlier, now you can access services inside your cluster from any of your Tailscale devices from any part of the world.

With tailscale, you'll also have an additional layer of authentication. Only users in your Tailscale networks can access the exposed services. For others, they might be able to guess what you have running in your cluster from your DNS records but they won't be able to access it since all the IPs will be private IPs.

For the next part, I'm looking into exposing some service inside my cluster to the internet fully without having to be in the Tailscale network. Tailscale Funnel suppose to do just that but I still haven't tested if it's working with services inside kubernetes.

Resources

How Tailscale Works: explanation on how Tailscale uses Wireguard to create a mesh VPN network architecture.
How NAT traversal works: recommended read even if you're not a networking geek. You'll learn a thing or two about networking for sure.
Full cofiguration for external-dns helm chart
external-dns annotation for the services
Tailscale subnet router inside kubernetes
k8s manifest for my subnet router

On Public Speaking

Fri, 08 Sep 2023 13:37:00 GMT

I had the chance to give a talk at my local DevOps meetup last July and recently, did another one at a Tech Talk event hosted by my current employer. Both talks are related to OPA but this blog post will be more of a personal reflection for me just to document these moments in my career.

There are two common things that I noticed about myself from giving the talk and I'm gonna describe them here.

The First Common Thing

Firstly, on days leading to the event I would slowly get agitated, the realness of having to do that talk creeps on me. I tend to procrastinate when faced with a big assignment like this. I will be so productive and do everything from playing a new game, thinking about a new project to do, cleaning my house, cooking, and eating --- except preparing and finishing the slides for the talk.

On the day of the talk itself, my productivity will be out of the window. I'd be reading, or doing something else but the thought of having to do the talk later is always on the back of my head. A few hours before the talk, I'm still polishing my slides, making last-minute changes, and going through the slides a couple more times in my head but around two hours before, I start distracting myself and doing other stuff. I'll go talk to my colleagues, read up on unrelated topics, and even get the time to catch up on my novels. This is when things are getting real for me.

Usually at these events, there will be food served before the event starts. For the first one, they had pizza and for the second event, they served rice with chicken curry and some kuih. They look really good tbh but I definitely won't be touching those. I'm already struggling enough to keep my nerves together and the thought of having a taste of the food at the back of my throat when talking later will throw me into a full-blown panic mode. It's like my sense is heightened and I get so sensitive to my surrounding that even keeping a conversation with others made me feel overwhelmed. My best companion at this moment is a bottle of mineral water --- tasteless, and I can fiddle with the bottle cap to distract myself.

It might feel like I'm exaggerating here but I can tell you that me before and after finishing the talk is so different I feel like we're a totally different person.

The Second Common Thing

As the host of the event invited me to the stage, I'll bring my laptop, set it up, and connect it to the projector. Then, holding the mic, I'll take a moment to look over the audience and take a deep breath.

Once I start talking, it's like a switch flipped and all that nervousness is gone.

Just like how I'd practiced, I would go through the slides, following the flow that I've set when putting the slides together. I like my presentation to have a story. I think it is more engaging to the audience and it is also a lot easier for me to remember what to say. I probably need to work on my tempo still (I blazed through 40+ slides in 20+ minutes 😆) but at least I wasn't too nervous and blanked out mid-speaking. In the most recent talk that I gave, I even managed to slip in a joke during my introduction. Quite proud of that one. Hah!

What's the magic?

I'm not sure if there's any science behind this but one factor that I think contributed to this flip in the switch is my confidence in the topic. Before giving my first talk, I've been reading, thinking, and playing with the technology for about a year on and off. I won't say that I mastered the topic but at least for the topic that I'm sharing, I feel like I have something to give to that the audience can benefit.

For my second talk, I've been involved in the development process from day one --- together with my team members, of course. So, at this point, I can say that I know the topic quite well.

It's a journey

My journey with public speaking started a long time ago. I used to be so nervous waiting for my turn to introduce myself to the class on the first day of school. Having all the attention on me even for a brief moment when I barged in the conversation with my friend group during our lepak session at mamak can make my ears turn red.

My first time speaking in front of a large audience was in a high school public speaking competition when I was 13. I was chosen because my English was pretty good in the class, not because I was good at public speaking, mind you. Being the inexperienced public speaker that I am, I printed my speech text on a big A4 paper and brought it to the stage, holding it full like that, not folding the paper whatsoever.

I didn't manage to memorize the whole thing so I kept looking at the text during my speech. After I finished, my friends told me they noticed how nervous I was from how the A4 paper in my hand was shaking so badly during the whole speech. I'll always remember that one.

Final Note

I can gladly say that I've improved a lot since then. Being exposed to these situations countless times made me realize that it is natural to feel nervous and I know that I'll come out better after going through it.

For those who might struggle with the same issue, just know that others are going through the same thing too and most importantly, take the risk and put yourself out there and eventually you'll get to the point where you'll overcome that fear.

Special thanks to the organizers for giving me the chance to put myself out there and my colleagues and friends for supporting me.

Interview Series: Explain How Kubernetes DNS works

Wed, 24 May 2023 15:00:00 GMT

This will be the first in my interview questions series. I'll compile interesting questions that I got from my experience interviewing for DevOps/SRE role in Malaysia.

Calling a service by its cluster-internal DNS

We'll go from the highest to the lowest level in this journey. So let's go through the scenario a bit: you have two services, foo and bar. those two services live in the same namespace app in your cluster. Now, inside service foo code, it makes a HTTP request to service bar. Probably something like so:

http.get("https://bar/")

What happens behind the scene from when the request is made to service bar and until the response is received back by service foo?

What is that weird DNS format?

You might've noticed that we're just calling the service bar by the name using a weird name. Instead of the usual something.com domain, we're just using bar directly. How is this possible?

Kubernetes allows you to call other services by using the service resource name directly. It does this by automatically appending the full DNS domain to the given service name. So for example here, when you make a request to bar, the application will make a DNS request to the local DNS server. The DNS server then notices that the domain that it received is not "complete" so it automatically appends the rest of the domain name based on the configuration that was given to it. If the service is running inside the namespace app, it will turn bar into bar.app.svc.cluster.local.

This automatic appending to complete the domain name is called "search domain". In our example the seach domain is configured as app.svc.cluster.local. So, whenever the service makes a call to bar it will automatically try to append the search domain and tries to resolve the domain name.

How (and where) is this configured?

Every pods in kubernetes has a file /etc/resolv.conf that is configured by the kubelet when starting the pod. This file will contain the info where to find the DNS server inside the cluster and also what to use as the search domain. Here's an example of the file (source):

nameserver 10.32.0.10
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Which IP will be returned by the DNS query?

The DNS query will return us a virtual service IP. Why virtual? It's because this IP doesn't actually points to a pod that runs our services.

In kubernetes, pods can come and go at any time which also means that their IP will change all the time. How do we know then where to send our requests to? The Service resource is used to abstract dynamic nature of pod IPs and provide a consistent IP that your application can use to send requests to it.

How does the service IP maps to pod IPs?

The Service resource always comes with its pair, the Endpoint (or EndpointSlice) resource. This Endpoint resource tracks the pod IPs and also have information which pod IP is ready to receive traffic. This information can be queried using the kubernetes API.

On the node where the pod runs, there is a program called kube-proxy that runs and updates the routing to map from service IP to pod IP. This routing can be done in multiple ways but currently the default is using iptables.

When does this routing happens?

When a request is first sent from the application code, its destination will be set to the service IP but before the request is sent out over the network, iptables modifies the destination and changes the service IP to pod IP. If there are multiple pods that sits behind a service, the pod IP will be load balanced using a round-robin. Once the destination IP is changed, the packet is then sent out over the network.

How do you know which node to send the packet to?

A kubernetes cluster can contain a lot of nodes. Sending the packet to the correct node is important. To know which node to send the packet to, the router in your network will need to know which node to send this packet to. If you setup your own cluster ala kubernetes-the-hard-way, you might need to configure these routes yourself but if you're using kubernetes on top of any cloud providers, they usually will do these setup for you and you don't have to do anything here.

Once that is sorted, your packet now can reach the correct node and the packet is sent to the correct pod on the node based on the destination pod IP set in the packet header. The response then will be sent to the source pod IP in the request packet header.

Response now sent back to the source node. All done?

Not yet. There's one more last thing to do. Remember when we sent the request originally, iptables had rewrote the destination from service IP to pod IP? Now for the response packet to be received back by the pod, the pod IP that we rewrote before needs to be converted back to the service IP.

This is needed because as far as the application knows, it sends a request to the service IP and not the pod IP. If it suddenly receives a response from a pod IP that it doesn't know of, then it will just drop the response. So, here iptable will have to remember what it did before and convert pod IP on the response packet back to service IP. Finally, our foo service can receive the response that it wants from the bar service.

Analysis Ticketing System

Wed, 17 May 2023 16:37:00 GMT

I kinda have an idea how you can scale the backend service of a ticketing system but I'm still stuck on the seating choice locking.

As some Twitter users have highlighted in the thread, it's not as simple as infinitely scaling the backend service using serverless functions or Kubernetes. The bottleneck usually lies on the DB tier.

Let's review the scenario:

An announcement was made that users can start buying ticket at 10AM. Before reaching 10AM, a bunch of users already ready with their devices (most use more than one for higher probability of getting the ticket). Once the clock reached 10AM, they'll start accessing the link to buy the ticket (some may started spamming the link before 10AM). (Reminds me of DDoS).

Requirement: one seat can only be purchased by exactly one user. To simplify my analysis, one user can only buy maximum 1 seat/ticket.

Waiting Room Queue

Based on the tweets that I saw, the ticketing vendor implemented a waiting room queue before you can enter the seat selection page. User @zulhhandyplast suggested they use Cloudflare product, Waiting Room, instead of rolling out your own waiting room implementation. How does a waiting room works? Here's a snippet from the Cloudflare Waiting room landing page:

Cloudflare Waiting Room allows organizations to route excess users to a custom-branded waiting room, helping preserve customer experience and protect origin servers from being overwhelmed with requests.

"...protect origin servers from being overwhelmed with requests." I think this is the most important part, which leads to improved reliability and consequently, user happiness.

Implementing a waiting room yourself seems quite hard. Will be nice if I can revisit this in the future and write more about this.

How does a waiting room queue works?

From the whitepaper:

The Cloudflare Waiting Room limits the number of users allowed in the application while placing excess traffic into a virtual queue to provide a smoother, more predictable user experience.

The number of users allowed in the room is capped and tracked using session cookies. Once the user leaves the application or the session cookies expired, new users will be allowed in until the max cap is reached. So, instead of users accidentally DDoSing the origin servers by continously refreshing the page in the browser, the incoming traffic never hits the origin server. With experience handling large scale DDoS, it's a given that we can trust Cloudflare to be able to handle the traffic coming to our ticketing platform.

I got into the room, now what?

Once the user manage to get into the room, they now need to "fight" with other users who can get the best seats first. Now we need figure out how can we lock a seat for a user so that once the seat is selected, no other user can select the same seat. If the user completes the purchase, now the seat is no longer available. When should we release the lock on the seat?

In the frontend, should there be a difference between a seat that is locked but not purchased yet and seats that's already taken and paid for?

Real-time locking or refresh-based?

Should the user refresh the page to know the current status of the seat whether it is still locked or released already? As a customer, I would prefer if the status of the seats are updated in real-time without me having to refresh the page. If not real-time, there can be cases where the user selects a seat because it is shown as available but only after selecting the seat will the user be informed that the seat is no longer available or temporarily locked. Then it's a game of refreshing the page as fast as possilbe and randomly choosing a seat, which will significantly increase the load on the server.

Making this real-time means we will need to maintain a connection for each user accessing the page. Maintaining these connections is expensive and webservers usually have a limit how many connections it can keep track of at a time. Assuming the waiting room from before is working, we already set the max number of users that will be accessing the page, so there is already a hard cap in number of connections that we have to handle so we can already preprovision our hardware to handle the load.

How do we keep track status of the seats?

Instead of maintaining the status of seats in RDBMS like PostgreSQL or MySQL, I think it is better to use in-memory data store like Redis. The EXPIRE command seems like a perfect choice for locking the seat for a certain period and automatically releasing the lock after the period ended. How can the backend knows when a key expired so that it can push a new message to the frontend to update the seat status? Keyword: keyspace notification.

I'm leaning on having a separate service handling this real-time seat status update. Once a user loads the page, it'll start a websocket connection to this seat status service which will set the seat number as key in Redis with a expiry time. Once the key expired, this service will be notified by Redis and in turn it will send a message through the websocket connection that it maintained to update the status of the seat on the frontend.

Payment processor as the bottleneck?

What can we do if 3rd-party payment processor not behaving?

I manage to lock a seat!

Congrats! Now since the seat is locked, user will be given a fixed duration to finish the transaction and pay for the ticket. Once payment is confirmed, we can update the DB and then tell the seat locking service to update status of the seat lock to "taken".

Once we booked the user seat, we can dispatch a background job to send an email to the user as confirmation and also includes the ticket details. It's okay to use a background job here so that user can get instant response instead of having to wait longer. In the instant response that is sent to user, we should manage their expectations and inform them that it may take a few minutes for the ticket to be sent to their email.

The Plan

TLDR

waiting room queue
websocket for real-time connections to update seat status
redis for keeping track of seat status (available, locked, taken)
rdbms for persisting seating record after payment is done
background job for non-time sensitive tasks like sending email

Instrumenting CI Pipelines using otel-cli

Sat, 08 Apr 2023 04:18:00 GMT

Why?

Why not? Get the whole picture of what is happening in your pipeline. Get notified when something is taking longer than it should.

How?

Use otel-cli, a standalone Go binary that can create OpenTelemetry traces and sends to a tracing backend using the OTLP protocol.

OpenTelemetry?

https://opentelemetry.io/

Tracing Backend?

You collect traces from your application using the OpenTelemetry SDK. To visualize the relationship between the traces, you'll have to send the traces to a tracing backend, which will provide a UI for exploring your traces. Example of tracing backend:

Self-hosted:

Grafana Tempo
Jaeger
ElasticSearch

Paid:

Honeycomb
Datadog
Grafana Cloud
ElasticSearch Cloud

OTLP Protocol

The OpenTelemetry Protocol (OTLP) specification describes the encoding, transport, and delivery mechanism of telemetry data between telemetry sources, intermediate nodes such as collectors and telemetry backends.

https://opentelemetry.io/docs/reference/specification/protocol/otlp/

otel-cli?

OpenTelemetry (OTel) supports many SDK to create traces from your application but in CI pipelines, you're usually using a shell script language like Bash which is not supported by any OTel SDKs currently. Therefore, we need a tool create this traces for us.

otel-cli is a tool that will do that. It will generate a trace ID, span ID, and sends the traces in the expected format.

How to use otel-cli?

The simplest way to start using it is first to set the OTEL_EXPORTER_OTLP_ENDPOINT value to tell otel-cli which backend to send our traces to.

Starting a local tracing backend server

otel-cli has a server subcommand that you can use to run a simple tracing backend on your local. You can run the following command in another terminatl to start the server:

otel-cli server tui

Setting the tracing backend endpoint

Now that we have a server running locally to send our traces to, let's tell otel-cli to send all the traces that it generated to this local server:

export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317

Here we send it to localhost on port 4317. Port 4317 is the default port when sending traces using grpc.

Sending our first trace

You can use exec subcommand to wrap a command with otel-cli. It will automatically set the start and end time to calculate the run duration for the command:

otel-cli exec --service my-service --name "My First Trace" echo "HELLO WORLD"

Then you should be able to see the a new line in the other terminal that we ran otel-cli server tui just now.

Conclusion

In this article, I showed you the simplest way you can use otel-cli. To get more valuable information from your traces, you'll usually need to add nested spans to your trace. It'll help break down the execution of your program to more smaller unit that can be inspected. To get more advanced example, you should refer to the otel-cli examples.

How to use OPA policy from an S3 bucket when using Atlantis policy check

Wed, 16 Nov 2022 14:00:00 GMT

Atlantis is an application used for collaborating on a Terraform code base using pull requests and one of the feature that it has is to run conftest and test a set of defined OPA policies. At the moment I'm writing this article, Atlantis only supports using local sources i.e. local filesystem as the source of the policy. In this article, I'll show an example of how to use an S3 bucket instead as the source for the policies.

Custom workflow and `run` step

Atlantis supports using custom workflows to override the default commands that it runs and as part of that feature, it supports defining any custom commands to run as part of the steps for each stage. We will be ~~abusing~~using this feature to override the default conftest command that Atlantis uses and specify our policy through the --update flag of conftest

conftest `--update` flag

By using --update you can tell conftest to pull the policy first every time it wants to run the tests. We will be using an S3 bucket as our source but before we can pull from S3, you have to make sure that wherever the Atlantis server is running, it can access and have permission to pull objects from the bucket. In my case, Atlantis is running as a StatefulSet inside a Kubernetes cluster so I have already configured the IAM permission needed for it to access the bucket.

conftest is using the go-getter package underneath to pull these packages so technically it should be possible to also pull from other sources that go-getter supports, other than just S3.

Result

Combining both of the features described above, here's an example of a simplified repo config that I use:

# minimal config for brevity; you might need to configure more options to make atlantis works properly
repos:
  - id: github.com/$ORG/$REPO
    workflow: custom

workflows:
  custom:
    policy_check:
      steps:
        - show # important don't skip this step
        - run: conftest test $SHOWFILE --update s3::https://s3-us-east-1.amazonaws.com/$BUCKET_NAME/policy

policies:
  policy_sets:
    - name: policy-from-s3
      path: /home/atlantis/policy
      source: local

In the example above, under the workflows key, I'm defining a custom workflow named custom and inside that custom workflow, I'm overriding the default policy_check steps with my own. My custom policy_check steps consists of the show step and the custom run step. The show step is crucial since this is when Atlantis will run terraform show to convert your Terraform planfile to a JSON formatted file.

When using the custom run step, Atlantis will store the path to this JSON formatted file in variable $SHOWFILE so when I ran my conftest command you can see that I'm using $SHOWFILE to run conftest against the file. Optional: if you want to run conftest against the Terraform files too, you can add *.tf after $SHOWFILE and it will include all the *tf files in that project directory.

Next comes the --update flag, to specify the S3 bucket, I'm using a URL format that is specified by the go-getter package replacing $BUCKET_NAME with the bucket name that I have configured with the correct permission and network access. Inside the S3 bucket, this is how I structured the files. I put all the OPA policies inside a folder policy since conftest complains when I just put all the policies directly at the root level inside the bucket. YMMV.

$BUCKET_NAME/
├─ policy/
│  ├─ stop_it.rego
│  ├─ dont_kill_server.rego

After defining our custom workflow, we can specify the custom worklow as the default workflow for a repo. This is done by setting the repos[].workflow value to the name of our custom workflow, in my case it's custom.

Next, as part of the using the policy check feature in Atlantis, you are required to set the policies values. You can refer to the docs for the full configuration required. Inside the policies key, there is a required policy_check key that is used to specify where Atlantis can find the OPA policies to use when running conftest. Usually, this is a folder on a local filesystem already containing the policies but in our case, since we're using the --update flag, we just need to specify any folder on the local filesystem that will be writable by the Atlantis user. You can see in the example above that I'm using /home/atlantis/policy.

Conclusion

That's all you need to do configure to make Atlantis pulls policies from S3 (and include Terraform source code files in your contest run). Shoutout to a DoorDash engineering blog post which mentioned briefly that they pulled their policies from S3 and made me curios how to do the same using Atlantis. You can mention me on Twitter (@pokgak73) if this article has helped you. That would most definitely made my day :)

Getting started with OPA and conftest

Sat, 12 Nov 2022 05:00:00 GMT

I started using OPA at my $dayJob recently and there are some parts that I think is not intuitive to grok for beginners.

If you want to play around with the rules and input, you can use the Rego Playground. It's super useful and I used it to test out policies and test my hypothesis when playing around with Rego language.

How does OPA, Rego, and conftest related to each other?

Rego is a declarative language used to write OPA policies. Then, OPA is the engine that takes in the policies written in Rego and evaluates it, producing a set of documents called "rules". You can use OPA and the Rego language directly to write policies for your config files but using conftest will make the DX much better. conftest builds on top of OPA and provide some extra functionality that makes using OPA easier.

Rego Basics

The Rego language focuses on querying the input to look for a given condition. If the input satisfies the query, then it will produce the document.

Variable Assignments

Variable assignment in Rego works the same like in other language. The expression foo := "hello" will assign the value "hello" to the variable foo.

One difference in Rego is that it implicitly assigns value true to the document if the condition given evaluates to true. In the example below, there's two ways to write the Rego expression. Rego actually implicitly assigns the value true so we can also remove

foo := "hello"

# first way: explicitly assigns `true` to `result` when condition is satisfied
result := true if foo == "hello"

# Rego implicitly assigns `true` to `result` when condition is satisfied
result if foo == "hello"

Let's bring this up to next level. Most of the time, the condition you're checking is not as straight forward as checking the value against a static value. You might also need to evaluate expressions in between and save the intermediary values in a variable to help improve readability. In previous example we only used a one-liner for the rule body but you can also have more complex rule body like the following using curly braces.

Declarative Rego Language

The Rego language is declarative and useful to query data structures for any value. Consider the following example (Rego playground link):

Let's assume our input is an array of object, each containing the keys "id" and "name". In this policy we're checking that the objects doesn't have any forbiden value for "name".

forbidden_names := ["foobar", "john"]

user_forbidden if input.users[i].name == forbidden_names[j]

This code would look something like this in Python:

users = [{"name": "foo"}, {"name": "bar"}]
forbidden_names = ["foobar", "john"]

user_forbidden = []
for i in range(len(input.users):
  for j in range(len(forbidden_names)):
    user_forbidden.push(input.users[i].name == forbidden_names[j])

return any(user_forbidden)

For both codes, user_forbidden will evaluate to true if one of the user name is included in the forbidden_names list. In the Python code, we used for loops with the any() function to check that none of the value is true. In the Rego code, we don't have to use any for loop or iterate through the user list. forbidden_names[i] means "for any of the values in forbidden_names. So in our Rego code, we essentially tells OPA, if any of the value in input.users is the same as any of the value in forbidden_name, then return set the value of user_forbidden to true.

In this case, since we are not using the index i and j to reference the value at those index anywhere in the policy, we can simplify it more by using _ (underscore) instead for the index. _ is like a throwaway value and we don't care about the index, we just care if one of the values is the same in user.input and forbidden_names.

user_forbidden if input.users[_].name == forbidden_names[_]

More complex policies

Before this our policies are all simple one liner but Rego also supports writing the rule body in multiple lines. In the example below, we are adding an exception to the rule that the previous rule doesn't apply to user with id == 5. So if one our user name value is john but have id == 5 then user_forbidden won't evaluate to true. Note that we are using the same index i when accessing the name and id property. This means we are referring to the same user. If we use _ or a different index when accessing the name and id, the rule will evaluate to true.

If any of the expressions inside the rule body evaluates to false or undefined then it will stop evaluating the rule body and return undefined for user_forbidden.

forbidden_names := ["foobar", "john"]

user_forbidden if {
    input.users[i].name == forbidden_names[_]
    input.users[i].id != 5

    false
    print("this will not be printed")
}

Using conftest

Previously, we used arbitrary names for our rules but conftest introduces a few keywords that we must use so that it can detect any failed rules and includes it in the output. Conftest will pick up any rules with name deny, warn, or violation and the summary will be shown in conftest output.

➜ tree conftest
conftest
├── input.json
└── policy
    └── names.rego

# input.json
{
    "users": [
        {
            "id": 1,
            "name": "john"
        },
        {
            "id": 2,
            "name": "bar"
        },
        {
            "id": 3,
            "name": "foobar"
        }
    ]
}

# policy/names.rego
package main

import future.keywords.contains
import future.keywords.if

deny contains msg if {
    forbidden_names := ["john"]
    name := input.users[_].name
    name == forbidden_names[_]
    
    msg := sprintf("username %v is not allowed", [name])
}

warn contains msg if {    
    id := input.users[_].id
    id == 2
    
    msg := sprintf("id %v is not allowed", [id])
}

Run conftest against our input file:

➜ conftest test input.json --policy policy/ 
WARN - input.json - main - id 2 is not allowed
FAIL - input.json - main - username john is not allowed

2 tests, 0 passed, 0 warnings, 2 failures, 0 exceptions

Note the values output here, the deny rule will be output as FAIL if the rule passes while the warn rule is counted as WARN. Here, conftest takes the output values from the OPA engine and formats the output for us to make it easier to interpret or integrate with other tools. You can also change the output format of conftest by passing in the --output flag. I like the github output since it will automatically prints the output in a format that Github Actions understoods and will surface error in Github UI approriately. You can also output it as JSON, which is great if you want to process the result output using tools like jq.

➜ conftest test --help
[...]
  -o, --output string         Output format for conftest results - valid options are: [stdout json tap table junit github] (default "stdout")

JSON output:

➜ conftest test input.json --output json
[
  {
    "filename": "input.json",
    "namespace": "main",
    "successes": 0,
    "warnings": [
      {
        "msg": "id 2 is not allowed"
      }
    ],
    "failures": [
      {
        "msg": "username john is not allowed"
      }
    ]
  }
]

parsers: using other format as input files

Until now all our input has been in JSON format but conftest also has built-in parsers that can automatically detect the input format and converts it to JSON for us. As of this moment, here is the list of valid parsers: [cue dockerfile edn hcl1 hcl2 hocon ignore ini json jsonnet properties spdx toml vcl xml yaml dotenv].

Example is for HCL2 code used for Terraform:

# input.tf
resource "aws_imaginary_resource" "this" {
  name = "this"
  instance_type = "r5.4xlarge"
  security_groups = ["12345", "45678"]
}

resource "aws_imaginary_resource" "that" {
  name = "that"
  instance_type = "t3.medium"
  
  ingress {
    port = 1234
    cidr = ["0.0.0.0/0"]
  }
}

We can use conftest parse to see how conftest will parse the Terraform file and then write our policy based on the parsed input.

➜ conftest parse input.tf
{
  "resource": {
    "aws_imaginary_resource": {
      "that": {
        "ingress": {
          "cidr": [
            "0.0.0.0/0"
          ],
          "port": 1234
        },
        "instance_type": "t3.medium",
        "name": "that"
      },
      "this": {
        "instance_type": "r5.4xlarge",
        "name": "this",
        "security_groups": [
          "12345",
          "45678"
        ]
      }
    }
  }
}

Conclusion

I just got started with OPA but considering the flexibility of it when used with conftest, I feel like you can use this for a lot of use cases. The ability to separate the policy logic from your application is powerful and the declarative nature of the Rego language also helps simplify the policy a lot as demonstrated in my comparison of the Python code above.

OpenTelemetry Basics

Sat, 13 Aug 2022 09:43:00 GMT

I got to work on integrating OpenTelemetry in an application that our team maintains recently so I'm starting a series documenting my learnings throughout this journey.

A little background info on the application I'm working on: it's a Slack chatbot written in Typescript using BoltJS. Our goal is to know how many users are using our Slack bot with a breakdown of the percentage of successful and error interactions. When an error happened, we also want to know what exactly the user did and the current state of the application that caused it to error. Based on my reading, the last sentence is exactly what observability promises, So that's why we're giving it a try.

OpenTelemetry can be divided into three categories: tracing, logging, and metrics; but I'll be focusing on tracing in this series.

Tracing Primers

To get started you should know some basic concepts about tracing.

Traces, Spans

A trace consists of multiple spans and a span is a unit of work with a start and end time. In a span, you can create events that marks when something happened in the lifetime of the span.

A span can also have nested spans and these are called child spans. The parent span is usually representing some abstract unit of work, like the lifetime of a HTTP request when it from when it hits the application until the response is sent. Child spans can be used to get more details into the operations done during the lifetime of that parent span ie. API call to another service to fetch more informations.

Span attributes, Status, Errors

To add context to the spans, you can set custom attributes. Ideally, you want to send all the information that will help when debugging your application in the future so that later you don't have to modify the code and add more attribute when you noticed an issue and realized that you don't have enough information to debug the issue.

If your application encounters an error, you can set the span status to ERROR and also add the stack trace to the context for use in debugging. By default your span status will be set to OK.

Span Exporter

After the span ends, you'll want to send it to a backend service that will store and process it so that you can use it later. The sending is done by OTel Exporters. There are multiple backend available that accepts OTel traces as inputs but such as Jaeger, Zipkin but for my testing I'm using Honeycomb with the OLTP Collector.

Debugging

For debugging, there's also the ConsoleSpanExporter which will print out your spans in the console instead of sending it anywhere. I find this very useful to get fast response on what is being sent over but it's hard to do analysis with it so in production environment you should configure the exporter to use other backends instead.

Automatic vs Manual Instrumentation

Now we got the basics out of the way, let's look at how you can start adding spans to your application to build traces.

The easiest way to get started is to use auto instrumentation which will automatically injects code in the HTTP, requests, DNS, libraries that you're using to create spans and events. In nodejs, this can be done by installing the auto-instrumentations-node NPM package. This package pulls in several other packages to automatically instrument your application.

This is a nice onboarding experience but I get overwhelmed by the amount of data sent when by these auto instrumentation package. Therefore, I recommend to you to start with manual instrumentation instead.

With manual instrumentation, you're forced to be intentional with the data that you're sending to the backend. With this I get to decide which information I want to send over and already have in mind what I want to do with it and which information I would like to gain from it.

Initialization

Whatever approach you end up with for the instrumentation, you'll want to make sure that you're initializing the OTel libraries at the start of your application. This is required because if you starts it later, your application might already be handling request when your OTel libraries are not initialized yet, causing it to miss some requests, or worse encounter errors.

The recommended way to do it is to use the -r flag from the node command:

-r, --require module Preload the specified module at startup. Follows require()'s module resolution rules. module may be either a path to a file, or a Node.js module name.

So in your package.json you'll have to add that to your start command:

scripts: {
    "start": "node -r ./tracing.js app.js",
}

If you're using Typescript like me, you'll want to use the NODE_OPTIONS shell variable to specify the flag instead:

scripts: {
    "start": "NODE_OPTIONS='-r ./tracing.js' ts-node app.ts",
}

NodeSDK vs NodeTracerProvider Confusion

One thing that made me confused is how different the code for initializing auto instrumentation compared to manual instrumentation.

This is the code provided by Honeycomb to use auto instrumentation. The key there is the getNodeAutoInstrumentation() function which will register all the supported auto instrumentation libraries. One more thing is that it is using the NodeSDK class.

// tracing.js
("use strict");

const { NodeSDK } = require("@opentelemetry/sdk-node");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-proto");

// The Trace Exporter exports the data to Honeycomb and uses
// the environment variables for endpoint, service name, and API Key.
const traceExporter = new OTLPTraceExporter();

const sdk = new NodeSDK({
    traceExporter,
    instrumentations: [getNodeAutoInstrumentations()]
});

sdk.start()

On the other hand, this is the code example from opentelemetry.io to start manual instrumentation. Notice that it's not using the NodeSDK class anymore and you need to create the Resource and NodeTracerProvider objects and configure it yourself.

const opentelemetry = require("@opentelemetry/api");
const { Resource } = require("@opentelemetry/resources");
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
const { ConsoleSpanExporter, BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");

// Optionally register automatic instrumentation libraries
registerInstrumentations({
  instrumentations: [],
});

const resource =
  Resource.default().merge(
    new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: "service-name-here",
      [SemanticResourceAttributes.SERVICE_VERSION]: "0.1.0",
    })
  );

const provider = new NodeTracerProvider({
    resource: resource,
});
const exporter = new ConsoleSpanExporter();
const processor = new BatchSpanProcessor(exporter);
provider.addSpanProcessor(processor);

provider.register();

TBH I'm still not clear what is the difference betwen using NodeSDK vs manually configuring the NodeTracerProvider. When using NodeSDK does the NodeTracerProvider got configured automatically?

How and when to start tracing?

To start manually instrumenting your application, you'll have to create a root span. A root span is the first span you create once the request enters your application.

Now, if you have a normal HTTP request/response-based application, it is easy to figure out where to start and end your root spans. All your incoming requests will most likely be handled by a controller and each endpoint will be handled by a method. In this type of application, your root span can be started once the request hits the application in the method in your controller and ends before you send the response.

During the lifetime of that request, you can create child spans to track other works done while processing the request. There's only one entry point for requests and exiting the entry point means the request is finished. If your application encountered errors during the execution, it can set the span status to ERROR and add the stack trace info to the span.

Conclusion

Once you managed to create spans, set attributes, and then export it to a backend. You're pretty much done with the basics of instrumenting your application. Go ahead and add more traces to your application!

Instrumenting a Slack bot with OpenTelemetry

Sat, 13 Aug 2022 09:43:00 GMT

Note: I'm using pseudocode in the code example in this article to keep the article brief. Please refer to the official Slack and OpenTelemetry documentation for the actual code.

I've talked about the basics of OpenTelemetry in my previous article. In this one, I'll explain more on how we're integrating OpenTelemetry with our Slack-based application.

At the end of this article, this is roughly how the span lifetime and events created will look like:

Slack BoltJS Socket Mode

Compared to the a standard HTTP request/response, we're using BoltJS with socket mode. This gives us the advantage of not having the application exposed publicly to be able to accept requests from Slack but this also means that we cannot just use the auto-instrumentation for HTTP developed by the community.

Socket mode uses WebSocket to establish connection to Slack and exchange messages through that connection. There is no official auto-instrumentation support for the ws library that is used by BoltJS socket mode but I found opentelemetry-instrumentation-ws, a 3rd-party library for ws library auto-instrumentation.

Spent a few days integrating it into our application and in the end I concluded that the auto-instrumentation provided by the opentelemetry-instrumentation-ws is too low-level. Our goal is to track user interactions with the application - when they use the bot, which option they choose, what were they trying to do, and whether the interaction ends successfully or with an error. The library, however, created spans when a new connection is established between our application and Slack but no spans or events for user interactions.

So, the conclusion? We'll instrument the application manually.

Creating and Ending Spans

Since this application is used company-wide, it's highly likely that multiple users will be using it in parallel. To track user interactions independent from each other, we'll also need separate spans for each user.

I decided to go with an object spanStore storing the user spans. Like a singleton pattern, a new span will be created for that user if it doesn't exist yet in spanStore, otherwise it will just return the existing user span.

spanStore = {}

function getUserSpan(username) {
    if (user in spanStore) {
        return spanStore[username]
    }

    span = startSpan(username)
    spanStore[username] = span

    return span
}

Now that we have a function to create the span, when in the lifetime of the incoming event do we create the span? Ideally, as early as possible before anything else so that we can track everything. BoltJS supports setting a global middleware that will be called before the event handler function are called. This is where I call the getUserSpan() function above. For the first event for that user, it will create a new span and for the next events it will just return the existing spans that I can use.

Next, when do you end the span? Due to how the application works, we're assuming that each user can only have one session at one time and at the end there will always be an finishing event triggered when the user finished their interaction with the application. Based on that fact, I wrote an event listener that will respond to this finishing event by calling the OTel function to end the span and remove the span from the spanStore object above.

app.event({id: 'finishing_event'}, async ({username}) => {
    span = getUserSpan(username)
    span.end()
    removeSpanFromSpanStore(span)
})

Tracking User Actions with Span Events

With these, we have a separate span for each user for the whole duration of their interaction with the application. With only one span, we don't have insights yet into what the user are doing, which actions are taken by the user, so we'll need a way to track user actions.

With Slack BoltJS, we can trigger a listener function on every user interaction. I wrote a function that will create a new span event using the user input id as the event name. I also passed in the whole payload so that the we can see the payload of user actions later when debugging issues. Add this as another global middleware, now we're creating a new event for every user actions.

app.action('callback_id', async ({username, action_id, payload}) => {
    span = getUserSpan(username)
    span.addEvent(action_id, {payload})
})

Confession

I'm actually not convinced that my way of doing this is correct. One of the reason is that since I use one root span for the whole interaction for a user, I'm also tracking the duration taken by the user to do the next action. From our perpective, this made the duration of the span tracked is now kinda useless for us since it also includes factors that are not controllable by us (time taken for users to do the next action).

Instead of one root span and creating new span events for every user interaction, maybe a new span for each interaction, linked to the previous span would be better since we only track the duration that we are in control of, not how long the user takes to click a button.

Nevertheless, since I already implemented like this now, let's see how that will turn out. Like the saying, you either die a hero, or you live long enough to see yourself become the villain.

Aiman Ismail

Scale down your instances, cut down your AWS bills

Know your workload patterns

Know your tools

Setting up the cron scaler

Combining cron and prometheus scalers

Setting up the prometheus scalers

Issues arising from autoscaling

Connections being terminated prematurely

HPA config: stabilizationWindowSeconds

HPA config: pod scale down policy

Scaling down your nodes using Karpenter

Summary

Eliminating cross-AZ traffic cost on AWS

The Traffic Flow

Incoming traffic to Load Balancer Nodes

Resolving LB Nodes IP from the Internet

Resolving LB Nodes IP from within your AWS Network

Load Balancer Nodes to ingress-nginx pods

ingress-nginx pods to backend services pods

Kubernetes 1.31 Service traffic distribution

ingress-nginx default routing behaviour

Conclusion

Exploring AWS ALB for EKS

Current Setup

Proposed Setup

ALB Quotas and Limitations

Pricing: ALB vs NLB

Rule Evaluations

New Connections

LCU-hour pricing

Conclusion

Docker's lesser known command: docker buildx bake

Background

Adding Multi-platform support

Using the docker-container build driver

Enter docker buildx bake

Putting it all together

well ackshually

Recap

Using DuckDB to analyze NGINX logs

Making the hard things easy

Changing the log format

Mapping customer IDs to generic placeholder

Fetching the logs from Loki

Loading the logs into DuckDB

Running queries

Highest requests per minute grouped by IP

IP address with the highest number of request to a particular host

Checking the paths an IP have been sending requests to

Conclusion

We got DDoSed

Detection

Rate limiting from the ingress-controller

Cloudflare to the rescue

Using the WAF Rules

Managed rules

Custom rules: blocking by country

Custom rules: blocking using query params, headers, and user agent

Custom rules: blocking known attacker IPs

Rate limiting rule

Bonus

Blocking our own IPs

Blocking accessibility bots (from US)

Silly mistake: external-dns

Using annotation to proxy traffic through Cloudflare per Ingress

Conclusion

Scrape cAdvisor using Grafana Alloy

Alloy cadvisor exporter

The hidden cost of running your own observability stack

Cross AZ Traffic Amplification

Collector to Load Balancer Node: client routing policy

Ingress controller pod to distributor: Kubernetes Topology Aware Routing

Distributor to Ingester: no workaround

Special use case: getting logs from external source

Conclusion

Tech Janitor: Linux Cookbook

Table of Contents

Probing an instance from the outside

Check DNS record exists

Custom workflow and `run` step

conftest `--update` flag