This is my first blog of multiple blogs on developing a pod troubleshooter turned into a compiler as a service. What do you call it CaaS? COMPaaS is cooler though, lol. I've been learning a lot of things in this journey, so I decided to share about it with you all.

Let me start this blog with how it started, I wanted to make a super simple pod troubleshooter to provide for developers. Developers were having a hard time running migrations, checking application logs, and everything. I thought, okay, I'll create a simple tool which will do one simple thing; provide live tty session inside a running pod.

Although I used Weavescope for the project need, I thought it was always exciting to learn how these tools work and how can I develop one myself. So, I decided I'll build one live tty API for the pod myself.

REST API of Kube-API Server

Let me give you a little bit of background on what is this kube-apiserver thing.

Kube-API Server is one of the major services in the control plane which provides a REST API to change/configure the cluster's state. All the things (kubelets, controller manager) need to communicate with the API Server to perform any change in the cluster.

Now, it was clear. If I want a shell in the pod, I need to go through the kube-api server. But how? I wanted the REST API documentation or any reference on how can I send requests to the kube-API server.

Let me teach you a nice hack to find out what requests are you making when you're using kubectl. You can change the verbosity to level 10 to get the exact curl request you're making with the API Server.

kubectl get pods --v=10
...
curl -v -XGET -H "User-Agent: kubectl/v1.25.4 (linux/amd64) kubernetes/872a965" \
 'https://[redacted].com/api/v1/namespaces/default/pods?limit=500'
...

It's that simple to understand what request is kubectl making to fetch the list of pods in your cluster.

You can also get the OpenAPI schema of the REST API of Kubernetes.

curl --insecure -L "https://$API_URL/openapi/v2" \
    -H "Authorization: Bearer $TOKEN"

This will provide you with the complete open API schema of the kube-api server. You can then visualize this with tools like redoc, swagger, etc., and get a proper hold of the API.

REST Endpoint for pod/exec

Our goal is to extract the endpoint of exec-ing inside of the pod. You can either get it using the kubectl as I showed you or just explore through the schema. Either way, the endpoint is this.

${this.config.url}/api/v1/namespaces/${namespace}/pods/${podName}/exec?command=${cmd}&stdin=true&stdout=true&stderr=true&tty=true

But the catch is, that this endpoint doesn't work on HTTP, it upgrades your request to WebSockets. You can't do something like okay, I'll fire up an HTTP request with the command I wanna execute and show the stdout. The response to this endpoint is always a WebSocket upgrade, and the stdout/stderr is sent as a WebSocket message.

curl -XPOST -H "X-Stream-Protocol-Version: v4.channel.k8s.io" \
 -H "X-Stream-Protocol-Version: v3.channel.k8s.io" \
 -H "X-Stream-Protocol-Version: v2.channel.k8s.io" \
 -H "X-Stream-Protocol-Version: channel.k8s.io" \
 -H "User-Agent: kubectl/v1.25.4 (linux/amd64) kubernetes/872a965" \
 "https://$API_URL/api/v1/namespaces/default/pods/nginx-deployment-cbdccf466-qjfvw/exec?command=python&container=nginx&stdin=true&stdout=true&tty=true"

The response will contain the header Connection: Upgrade and Upgrade: SPDY/3.1 This means the connection is upgraded to WebSocket.

curl won't be able to take you long enough with WebSockets, we can use another tool similar to curl wscurl to send WebSockets requests with curl-like API.

If you're using curl, make sure to specify the subprotocol (i.e. X-Stream-Protocol-Version) with the flag -s .

wscat -H "Authorization: Bearer $TOKEN" \
   -c "$URL" \
   -s "v4.channel.k8s.io"

Message Format from Kube-API Server.

The message you receive from kube-api server over WebSockets is in a specific format. The first byte contains the messageType and the remaining bytes are the actual message from the TTY.

There can be three types of messages: stdin, stdout, stderr .

In this way, you need to filter out the messages whether they're in stdin, stdout, or stderr.

For example: if you establish a connection with websocket, wscat will print something like this:

You see the first byte in the message received from the server. It's 0x01 because it's sent in stdout by the pod when you open a shell. And, your terminal doesn't know how to render that.

Will continue in Part 2.