kloud/provider: add Kubernetes support #10776

This serves as a retainer for the design and implementation of a Kubernetes provider for Koding.

Overview

Kubernetes is a container orchestration technology that has gained critical traction in the world of development and production devops centered around container-based application workloads. You can even bring over your existing Dockerfiles and Kubernetes will happily accept your Docker images into its workload abstractions called pods.

You tell Kubernetes (with kubectl) what you want the topology of your application to look like desired state, usually by means of a YAML specification (deployment, replica set, job), and Kubernetes makes the actual state match the desired state in a safe and reliable way.

Interested in design docs for Kubernetes?

As a basic example of a deployment specification to Kubernetes for nginx:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

This will result in Kubernetes:

1. Pulling the ngingx:1.7.9 image from Docker Hub if it doesn't already have it
2. Creating and running 3 containers from that image all listening on their own port 80s
3. Creating a service endpoint in the Kubernetes proxy that all 3 of the nginx containers serve.

The service endpoint should persist through application updates (rolling), so it is ideal for configuration/environment variables for consumers. This would be analogous to the public endpoint that current Koding stacks serve once provisioned based on their specific application port(s).

Expected Behavior

What do we want/expect from Kubernetes?

As a Koding user I want to be able to configure the following for my Kubernetes Pod Stack:

• Images
• Scale
• Storage
• Network
• Quotas

NOTE: Modifications to deployments running on Kubernetes that are done outside of Koding's knowledge, shouldn't break Koding's implementation. Have some sort of discovery process, the Kubernetes API should be able to make this a simple task.

Once my Pod Stack has been provisioned on the existing Kubernetes cluster, that I specify at stack creation/initialization, I should be able to access my Pod Stack from a Kubernetes service endpoint.

I also want the ability to:

• Exec into any container that Koding is aware of in the Kubernetes cluster
• Scale in/out any Pod Stack

Current Behavior

As a close reference point, we are doing a small comparison with the existing Marathon provider for Koding. Currently the Marathon provider focuses on single container applications, given this restriction the implementation injects Koding's agent klient into the container via a startup script before running the user's application inside the container.

This isn’t great in terms of scale, and wrapping the application with an entrypoint script to install klient at container runtime is slow and hacky. Thus, we haven’t ruled out modifying klient.

While single container applications work great for the basic use cases, it might not be the most realistic view of how users expect to consume hosted container orchestration services. Perhaps a more realistic view is at least an application that has more than one process (multiple container images) and each of those processes needs to be scaled out. The number of klient processes that would be in play with the current approach, grows very rapidly.

Possible Solution

If we take a step back, and let's say we rely on Kubernetes to do it's job, trust it even. All the Koding provider needs to do at that point is call the right Kubernetes API operations with the correct data structures representing the desired state. Every API operation from Kubernetes results in a record of intent response. Meaning Kubernetes will make the actual state match the desired state as soon as possible.

A possible solution that we have discussed briefly is to have a dedicated klient container per deployment that the provider deploys to Kubernetes cluster. There are certain guarantees that Kubernetes provides that makes this an appealing solution per deployment:

• Co-located processes
• Shared storage (volumes)
• Shared network
• Shared namespaces
• Everyone fails or succeeds together (no half-functioning deployment)

In short, we are guaranteed that:

• if our deployment is up, our corresponding klient container is also up
• if our deployment is down for any reason, our corresponding klient container is also down

This would remove the need to inject an entrypoint script into each container image, and we would gain back any built-in Kubernetes features that the current approach would mask such as: healthchecking.

Possibly we should look into syncing Koding credentials to the provided Kubernetes cluster to leverage the benefits of Kubernetes providing credentials to user containers.

Kubernetes community maintains a Golang client so we can programmatically invoke the Kubernetes API to set desired state.

Do we still want to use Terraform under the covers here?

It might not make sense to. Terraform is about provisioning cloud resources, and in our proposed > solution we believe the Kubernetes provider shouldn't be involved with managing the Kubernetes
cluster (GCE, Bluemix Container Service, On-Prem), the provider should only
create/update/destroy deployments/pods via the Kubernetes API. Terraform also assumes it is
authoritative over cloud resources. If someone were to use kubectl outside of Koding
there would likely be a difference in state records.