Open in app

Sign in

Write

Sign in

Using Kustomize and Confd Notes

Yunus Kılıç
CodeX
Published in
7 min readSep 28, 2021

--

This post was written to collect notes from different sources.

“Kustomize[1] is a standalone tool to customize Kubernetes objects through a kustomization file. Since 1.14, Kubectl also supports the management of Kubernetes objects using a kustomization file. To view Resources found in a directory containing a kustomization file, run the following command”[0]:

kubectl kustomize <kustomization_directory>

To apply those Resources, run kubectl apply with — kustomize or -k flag:

kubectl apply -k <kustomization_directory>

Kustomize is used for declarative object configuration.

What are imperative and declarative object configurations?

Imperative object configuration

In imperative object configuration, the kubectl command specifies the operation (create, replace, etc.), optional flags, and at least one file name. The file specified must contain a full definition of the object in YAML or JSON format.[2]

Examples

Create the objects defined in a configuration file:

kubectl create -f nginx.yaml

Delete the objects defined in two configuration files:

kubectl delete -f nginx.yaml -f redis.yaml

Trade-offs

Advantages compared to imperative commands:

Object configuration can be stored in a source control system such as Git.

Object configuration can integrate with processes such as reviewing changes before push and audit trails.

Object configuration provides a template for creating new objects.

Disadvantages compared to imperative commands:

Object configuration requires a basic understanding of the object schema.

Object configuration requires the additional step of writing a YAML file.

Advantages compared to declarative object configuration:

Imperative object configuration behavior is simpler and easier to understand.

As of Kubernetes version 1.5, imperative object configuration is more mature.

Disadvantages compared to declarative object configuration:

Imperative object configuration works best on files, not directories.

Updates to live objects must be reflected in configuration files, or they will be lost during the next replacement.

Declarative object configuration

When using declarative object configuration, a user operates on object configuration files stored locally, however, the user does not define the operations to be taken on the files. Create, update, and delete operations are automatically detected per object by kubectl. This enables working on directories, where different operations might be needed for different objects.

Trade-offs

Advantages compared to imperative object configuration:

Changes made directly to live objects are retained, even if they are not merged back into the configuration files.

Declarative object configuration has better support for operating on directories and automatically detecting operation types (create, patch, delete) per-object.

Disadvantages compared to imperative object configuration:

Declarative object configuration is harder to debug and understand results when they are unexpected.

Partial updates using diffs create complex merge and patch operations.

Kustomize Usage

  • Generating resources from other sources
  • Setting cross-cutting fields for resources
  • Composing and customizing collections of resources

Generating Resources

“ConfigMaps and Secrets hold configuration or sensitive data that are used by other Kubernetes objects, such as Pods. The source of truth of ConfigMaps or Secrets are usually external to a cluster, such as a .properties file or an SSH keyfile. Kustomize has secretGenerator and configMapGenerator, which generate Secret and ConfigMap from files or literals.[0]”

configMapGenerator

“To generate a ConfigMap from a file, add an entry to the files list in configMapGenerator[0]”

$ mkdir kustomizeDemo
$ cd kustomizeDemo
$ echo "ENV=DEMO" > demo.yaml
$ cat <<EOF >./kustomization.yaml
configMapGenerator:
- name: demo-configmap
  files:
- demo.yaml
  EOF
  $ kubectl kustomize ./
  apiVersion: v1
  data:
  demo.yaml: |
  ENV=DEMO
  kind: ConfigMap
  metadata:
  name: demo-configmap-47f78c8429
  Other approaches:
  configMapGenerator:
- name: example-configmap-1
  envs:
    - .env
      EOF
      cat <<EOF >./kustomization.yaml
      configMapGenerator:
- name: example-configmap-2
  literals:
    - FOO=Bar
      EOF

This is an example deployment that uses a generated ConfigMap:

# Create a application.properties file
cat <<EOF >application.properties
FOO=Bar
EOFcat <<EOF >deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: my-app
        volumeMounts:
        - name: config
          mountPath: /config
      volumes:
      - name: config
        configMap:
          name: example-configmap-1
EOFcat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
configMapGenerator:
- name: example-configmap-1
  files:
  - application.properties
EOF

Kustomize generates ConfigMap and ConfigMap is mapped to a volume.

secretGenerator can be found at [0].

Setting cross-cutting fields

It is quite common to set cross-cutting fields for all Kubernetes resources in a project. Some use cases for setting cross-cutting fields:

  • adding the same name prefix or suffix
  • Adding the same set of labels
  • Adding the same set of annotations
  • Setting the same namespace for all Resources

Here is an example:

# Create a deployment.yaml
cat <<EOF >./deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
EOFcat <<EOF >./kustomization.yaml
namespace: my-namespace
namePrefix: dev-
nameSuffix: "-001"
commonLabels:
  app: bingo
commonAnnotations:
  oncallPager: 800-555-1212
resources:
- deployment.yaml
EOF

Composing and Customizing Resources

Composing

Kustomize supports composition of different resources. The resources field, in the kustomization.yaml file, defines the list of resources to include in a configuration. Set the path to a resource’s configuration file in the resources list. Here is an example of an NGINX application comprised of a Deployment and a Service:

# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
EOF# Create a service.yaml file
cat <<EOF > service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    run: my-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    run: my-nginx
EOF# Create a kustomization.yaml composing them
cat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
- service.yaml
EOF

The Resources from kubectl kustomize ./ contain both the Deployment and the Service objects.

Customizing

Patches can be used to apply different customizations to Resources. Kustomize supports different patching mechanisms through patchesStrategicMerge and patchesJson6902. patchesStrategicMerge is a list of file paths. Each file should be resolved to a strategic merge patch. The names inside the patches must match Resource names that are already loaded. Small patches that do one thing are recommended. For example, create one patch for increasing the deployment replica number and another patch for setting the memory limit.

# Create a deployment.yaml file
cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
EOF# Create a patch increase_replicas.yaml
cat <<EOF > increase_replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  replicas: 3
EOF# Create another patch set_memory.yaml
cat <<EOF > set_memory.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  template:
    spec:
      containers:
      - name: my-nginx
        resources:
          limits:
            memory: 512Mi
EOFcat <<EOF >./kustomization.yaml
resources:
- deployment.yaml
patchesStrategicMerge:
- increase_replicas.yaml
- set_memory.yaml
EOF

Run kubectl kustomize ./ to view the Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      run: my-nginx
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - image: nginx
        name: my-nginx
        ports:
        - containerPort: 80
        resources:
          limits:
            memory: 512Mi

CONFD

confd is a lightweight configuration management tool focused on:

keeping local configuration files up-to-date using data stored in etcd, consul, dynamodb, redis, vault, zookeeper, aws ssm parameter store or env vars and processing template resources. reloading applications to pick up new config file changes[3].

AWS SSM

AWS Systems Manager allows you to centralize operational data from multiple AWS services and automate tasks across your AWS resources. You can create logical groups of resources such as applications, different layers of an application stack, or production versus development environments. With Systems Manager, you can select a resource group and view its recent API activity, resource configuration changes, related notifications, operational alerts, software inventory, and patch compliance status. You can also take action on each resource group depending on your operational needs. Systems Manager provides a central place to view and manage your AWS resources, so you can have complete visibility and control over your operations.[4]

Confd is related with configuration management. That’s why I am interested in SSM parameter store.

SSM Parameter Store

AWS Systems Manager provides a centralized store to manage your configuration data, whether plain-text data such as database strings or secrets such as passwords. This allows you to separate your secrets and configuration data from your code. Parameters can be tagged and organized into hierarchies, helping you manage parameters more easily. For example, you can use the same parameter name, “db-string”, with a different hierarchical path, “dev/db-string” or “prod/db-string”, to store different values. Systems Manager is integrated with AWS Key Management Service (KMS), allowing you to automatically encrypt the data you store. You can also control user and resource access to parameters using AWS Identity and Access Management (IAM). Parameters can be referenced through other AWS services, such as Amazon Elastic Container Service, AWS Lambda, and AWS CloudFormation.[4]

One such service is SSM Parameter Store which is a secured and managed key/value store perfect for storing parameters, secrets, and configuration information. However, in April of 2018, AWS also introduced another service called AWS Secrets Manager that offers similar functionality. Given that both services kind of do the same thing, which to choose isn’t clear. With that in mind, let us take a look at the similarities and differences of these two services to better understand which service will best fit your architectural needs.[5]

SSM Parameter Store is free, that’s why I prefer it instead of AWS Secrets Manager.

aws ssm put-parameter --name "/myapp/database/url" --type "String" --value "db.example.com"
aws ssm put-parameter --name "/myapp/database/user" --type "SecureString" --value "rob"

Above commands put two parameter to SSM Parameter Store

The confdir is where template resource configs and source templates are stored.

sudo mkdir -p /etc/confd/{conf.d,templates}

Template resources are defined in TOML config files under the confdir.

/etc/confd/conf.d/myconfig.toml

[template]
src = "myconfig.conf.tmpl"
dest = "/tmp/myconfig.conf"
keys = [
    "/myapp/database/url",
    "/myapp/database/user",
]

Source templates are Golang text templates.

/etc/confd/templates/myconfig.conf.tmpl

[myconfig]
database_url = {{getv "/myapp/database/url"}}
database_user = {{getv "/myapp/database/user"}}confd -onetime -backend ssm$ cat /tmp/myconfig.conf 
[myconfig]
database_url = db.example.com
database_user = rob

0-) https://github.com/kelseyhightower/confd

1-) https://aws.amazon.com/systems-manager/features/

2-) https://www.1strategy.com/blog/2019/02/28/aws-parameter-store-vs-aws-secrets-manager/

3-) https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/

4-) https://github.com/kubernetes-sigs/kustomize

5-) https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/

Kubernetes

--

--

Yunus Kılıç
CodeX

Written by Yunus Kılıç

107 Followers

I have 10 years of experience in high-quality software application development, implementation, and integration.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams