Kustomize for managing Kubernetes manifests

With Kustomize we will be able to manage easily multiple K8S YML files for multiple configurations. It will act like a YML files merger which will be easier to manage a real world application or infrastructure.

Prerequisites and objectives

You should have followed the Kubernetes(K8S) Workshop - Part 1 or have basic knowledge of Kubernetes in a declarative way.

Let's do it

Install Kustomize

$ brew install kustomize

We may use a file structure like so :

.
└── k8s
    └── kustomize
        ├── base
        │   ├── kustomization.yml
        │   ├── deployment.yml
        │   └── service.yml
        └── overlays
            ├── dev
            │   ├── kustomization.yml
            │   └── additional.yml
            ├── int
            │   └── kustomization.yml
            ├── qa
            │   └── kustomization.yml
            └── prd
                └── kustomization.yml

Important part here is the concept of base and overlays, we can define default K8S object in base and customize them for our multiple environments. We can eventually add specific object to one environment, for example in dev we may need to handle more thing than on a real environment. Important files are kustomization.yml in each sub folders. It's important to respect the filename but It can be .yml or .yaml

Use kustomize

Create folders and file like the schema above (no need to create all env).

mkdir -p k8s/kustomize/base
mkdir -p k8s/kustomize/overlays/dev
mkdir -p k8s/kustomize/overlays/prd

touch k8s/kustomize/base/kustomization.yml
touch k8s/kustomize/base/deployment.yml
touch k8s/kustomize/base/service.yml
touch k8s/kustomize/overlays/dev/kustomization.yml
touch k8s/kustomize/overlays/dev/mongodb.yml
touch k8s/kustomize/overlays/prd/kustomization.yml

Base

# k8s/kustomize/base/kustomization.yml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yml
  - service.yml

# k8s/kustomize/base/deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.7.9
          ports:
            - containerPort: 80

# k8s/kustomize/base/service.yml

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Dev

# k8s/kustomize/overlays/dev/kustomization.yml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
  - mongodb.yml

# k8s/kustomize/overlays/dev/mongodb.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      name: mongodb
  template:
    metadata:
      labels:
        name: mongodb
    spec:
      containers:
        - name: mongodb
          image: mongo:4.2.8
          ports:
            - containerPort: 27017

Prd

# k8s/kustomize/overlays/prd/kustomization.yml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base

We define all our base ressources and we use kustomization.yml file to include them. For overlays we relatively include the base.

We can now build files with kustomize to see the result

$ kustomize build k8s/kustomize/overlays/dev

It outputs a single YML file with all our resources defined (separated with ---).

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      name: mongodb
  template:
    metadata:
      labels:
        name: mongodb
    spec:
      containers:
        - image: mongo:4.2.8
          name: mongodb
          ports:
            - containerPort: 27017
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: nginx:1.7.9
          name: nginx
          ports:
            - containerPort: 80

We can apply them if we want.

$ kustomize build k8s/kustomize/overlays/dev | kubectl apply -f -

Now let's try to override some configurations.

# k8s/kustomize/overlays/prd/kustomization.yml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base

patchesStrategicMerge:
  - deployment-prd.yml

Create a new file k8s/kustomize/overlays/prd/deployment-prd.yml and we will override some value of our base deployment.

# k8s/kustomize/overlays/prd/deployment-prd.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1024Mi"
              cpu: "1000m"

We changed for example the the number of replicas and we set some resources allocation and limits.

Build It again

kustomize build k8s/kustomize/overlays/prd

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: nginx:1.7.9
          name: nginx
          ports:
            - containerPort: 80
          resources:
            limits:
              cpu: 1000m
              memory: 1024Mi
            requests:
              cpu: 250m
              memory: 512Mi

We can also change the docker image version nginx:1.7.8 instead of nginx:1.7.9

# k8s/kustomize/overlays/prd/kustomization.yml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patchesStrategicMerge:
  - deployment-prd.yml
images:
  - name: nginx
    newTag: 1.7.8

This can be also done with command line (inside the overlay folder) kustomize edit set image nginx:1.7.8

$ kustomize build k8s/kustomize/overlays/prd

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: nginx:1.7.8
          name: nginx
          ports:
            - containerPort: 80
          resources:
            limits:
              cpu: 1000m
              memory: 1024Mi
            requests:
              cpu: 250m
              memory: 512Mi

Thanks to kustomize we can now handle and merge YML files easily, It offers other features that you may want like :

Another useful option is generatorOptions, like so, you can disable the suffix generation when applying files (eg: my-secret-34bfD will be my-secret)

kind: Kustomization
apiVersion: kustomize.config.k8s.io/v1beta1
# It's a good practice to add the namespace to kustomization files
# It can prevent you to apply accidentally namespaced object to the wrong one, in case of wrong context
# I usually do It once, in base/kustomization.yml
namespace: my-namespace
resources:
  - my-deployment.yaml

generatorOptions:
  disableNameSuffixHash: true