Understanding Kustomize Overlays

❌ The Problem: Without Overlays

Imagine managing 50+ components across 3 environments...

Without overlays: 50 × 3 = 150 nearly-identical YAML files
Change service name? Update 150 files!
Add new label? Update 150 files!
99% duplication, nightmare to maintain

# Without overlays, you'd have files like:
components/build-service-dev.yaml       # 500 lines
components/build-service-staging.yaml   # 500 lines (95% identical!)
components/build-service-prod.yaml      # 500 lines (95% identical!)

# And this for EVERY component... 😱
            

✅ The Solution: Kustomize Overlays

With overlays:

1 base with common configuration (shared by all environments)
3 small overlays with only environment-specific differences
Change service name? Update base once, all environments inherit it!
~95% reduction in duplication

📦 Base Layer

Common manifests: Deployment, Service, ServiceAccount, RBAC

🟢 Development Overlay

Patches: 1 replica, debug logging, latest image

🟡 Staging Overlay

Patches: 2 replicas, info logging, tested image

🔴 Production Overlay

Patches: 3 replicas, error logging, stable image, resource limits

📁 Real infra-deployments Structure

Here's the actual structure used for EVERY component in infra-deployments:

📁 components/build-service/

📁 base/ — Common resources for ALL environments

kustomization.yaml deployment.yaml service.yaml rbac.yaml

📁 development/ — Dev environment overlay

kustomization.yaml Patches base for dev

📁 staging/base/ — Staging environment overlay

kustomization.yaml Patches base for staging

📁 production/base/ — Production environment overlay

kustomization.yaml Patches base for production

Note: staging/ and production/ have an extra /base/ subdirectory because they have additional sub-overlays for stone-* clusters.

📌 Key Insight:

The base/ directory contains actual YAML manifests. The overlay directories contain only kustomization.yaml files that reference the base and apply patches. This is the DRY (Don't Repeat Yourself) principle in action!

🎯 Real Example: build-service

Base (Shared)

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- deployment.yaml
- service.yaml
- rbac.yaml

# Common labels for all environments
commonLabels:
  app: build-service
  app.kubernetes.io/name: build-service
                    

This base is referenced by ALL overlays. Changes here affect dev, staging, AND production.

Development Overlay

# development/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
- ../base  ← References common base!

# Dev-specific image (gets updated by Renovate)
images:
- name: quay.io/konflux-ci/build-service
  newTag: abc123def456...

# Dev-specific patches
patches:
- target:
    kind: Deployment
    name: build-service
  patch: |-
    - op: replace
      path: /spec/replicas
      value: 1  ← Only 1 replica in dev
                    

This overlay adds/modifies ONLY what's different for development. Everything else comes from base!

🔍 What Happens When You Run Kustomize Build?

$ kubectl kustomize components/build-service/development

# Kustomize process:
1. Read base/kustomization.yaml
2. Load all resources: deployment.yaml, service.yaml, rbac.yaml
3. Apply commonLabels from base
4. Read development/kustomization.yaml
5. Apply image tag patch (newTag: abc123...)
6. Apply replica patch (replicas: 1)
7. Output final, merged YAML

# Result: Full Kubernetes manifests ready to apply!
# The manifests have base content + dev-specific patches
                

🎨 Common Overlay Patterns in Konflux

✅ Image Tag Overrides

images:
- name: quay.io/konflux-ci/service
  newTag: different-per-env
                    

Why: Dev uses latest, staging uses tested, prod uses stable.

✅ Replica Count Patches

patches:
- target:
    kind: Deployment
  patch: |-
    - op: replace
      path: /spec/replicas
      value: 3  # prod: 3, staging: 2, dev: 1
                    

Why: Production needs more replicas for HA.

✅ Resource Limits

patches:
- target:
    kind: Deployment
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/resources
      value:
        limits:
          memory: 2Gi
          cpu: 1000m
                    

Why: Prod has stricter resource limits for stability.

✅ Environment Variables

patches:
- target:
    kind: Deployment
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/env/-
      value:
        name: LOG_LEVEL
        value: "debug"  # dev: debug, prod: error
                    

Why: Different logging verbosity per environment.

✅ Additional Resources

resources:
- ../base
- prod-only-monitoring.yaml
- prod-only-backup.yaml
                    

Why: Some resources only make sense in certain environments.

✅ Namespace Patches

namespace: build-service-dev

# Or for multi-tenant:
nameSuffix: -dev
                    

Why: Each environment deploys to different namespace.

⚠️ Common Mistakes & Anti-patterns

❌ Duplicating Entire Manifests

# DON'T DO THIS:
staging/deployment.yaml  ← Full 200-line file
prod/deployment.yaml     ← Another 200-line file
                    

Why wrong: Defeats the purpose of overlays. Use patches instead!

❌ Environment Logic in Base

# DON'T DO THIS in base/:
env:
- name: ENVIRONMENT
  value: "production"  ← Hardcoded!
                    

Why wrong: Base should be environment-agnostic. Put env-specific config in overlays!

❌ Not Using Bases Reference

# DON'T DO THIS:
resources:
- ../../base/deployment.yaml
- ../../base/service.yaml
- ../../base/rbac.yaml
                    

Why wrong: Use bases: [../base] to reference the entire base directory!

❌ Overly Complex Patches

patches:
- target:
    kind: Deployment
  patch: |-
    # 100 lines of JSON patch...
                    

Why wrong: If patches are huge, maybe that config belongs in base with strategic merge!

🛠️ How to Work with Overlays in infra-deployments

📝 Scenario 1: Adding a New Environment Variable

# Question: Should I add it to base or overlay?

✅ Add to BASE if:
- All environments need it
- It has the same value everywhere
- Example: API endpoint that's the same for all

✅ Add to OVERLAY if:
- Value differs per environment
- Only some environments need it
- Example: DEBUG_MODE (true in dev, false in prod)

# HOW to add to base:
1. Edit components/my-service/base/deployment.yaml
2. Add env var to container spec
3. All overlays automatically inherit it!

# HOW to add to overlay:
1. Edit components/my-service/development/kustomization.yaml
2. Add patch with env var
3. Only dev environment gets it!
                

📝 Scenario 2: Changing Container Image

# This is done by MintMaker (Renovate bot) automatically!

# What MintMaker does:
1. Detects new image in quay.io
2. Opens PR to infra-deployments
3. Updates image tag in ALL overlays (dev, staging, prod)
4. CI tests the change
5. After merge, ArgoCD deploys

# Files changed by MintMaker PR:
- components/my-service/development/kustomization.yaml
- components/my-service/staging/base/kustomization.yaml
- components/my-service/production/base/kustomization.yaml

# Each gets updated with new image tag:
images:
- name: quay.io/konflux-ci/my-service
  newTag: abc123def456...  ← NEW SHA!
                

📝 Scenario 3: Adding a New Component

# Follow the established pattern!

1. Create directory structure:
   components/my-new-service/
       base/
           kustomization.yaml
           deployment.yaml
           service.yaml
           rbac.yaml
       development/
           kustomization.yaml
       staging/base/
           kustomization.yaml
       production/base/
           kustomization.yaml

2. Write base/ manifests with common config

3. Write overlay kustomization.yaml files:
   - Reference base: bases: [../base]
   - Set image tag
   - Add environment-specific patches

4. Create ArgoCD Application pointing to overlays

5. Test: kubectl kustomize components/my-new-service/development
                

🧪 Testing Your Overlay Changes

Before Opening a PR, Always Test Locally!

# 1. Test kustomize build (does it produce valid YAML?)
kubectl kustomize components/build-service/development

# 2. Test kustomize build AND kubectl validation
kubectl kustomize components/build-service/development | kubectl apply --dry-run=client -f -

# 3. Check what changed compared to current deployment
kubectl kustomize components/build-service/development | kubectl diff -f -

# 4. Test all three overlays
kubectl kustomize components/build-service/development
kubectl kustomize components/build-service/staging/base
kubectl kustomize components/build-service/production/base

# 5. Validate with kubeval (if installed)
kubectl kustomize components/build-service/development | kubeval

# Common errors to catch:
✗ Error: no matches for kind "Deploymen" (typo!)
✗ Error: json: cannot unmarshal number into Go value of type string
✗ Error: accumulating resources: accumulation err='...
            

🤔 When Should You Improve the Overlay Structure?

✅ Consider Refactoring When:

You're copy-pasting the same patch across multiple overlays
Base has environment-specific logic (should be in overlays)
Overlays duplicate entire manifests instead of patching
Multiple components need the same patches (create component base)
Patches are becoming unmaintainably complex

✅ Signs of Good Structure:

Base contains only environment-agnostic resources
Overlays are small (< 50 lines typically)
Clear separation of concerns
Easy to understand what differs between environments
Changes to common config require editing only base

⚠️ Warning: Don't Over-Engineer!

It's tempting to create deeply nested overlay structures or complex patch systems. Keep it simple! The current infra-deployments pattern (base + 3 overlays) works well for 95% of cases. Only add complexity when you have a clear, repeated need.

📚 Quick Reference Card

╔══════════════════════════════════════════════════════════════╗
║                 KUSTOMIZE OVERLAYS CHEAT SHEET               ║
╚══════════════════════════════════════════════════════════════╝

📁 Structure:
components/my-service/
    base/                     ← Shared by all environments
        kustomization.yaml    ← Lists resources
        *.yaml                ← Actual manifests
    development/              ← Dev-specific
        kustomization.yaml    ← Patches base
    staging/base/             ← Staging-specific
        kustomization.yaml    ← Patches base
    production/base/          ← Prod-specific
        kustomization.yaml    ← Patches base

🔧 Common kustomization.yaml Stanzas:

# Reference base:
bases:
- ../base

# Override image tag:
images:
- name: quay.io/konflux-ci/service
  newTag: abc123def456

# Add/modify with patches:
patches:
- target:
    kind: Deployment
    name: my-service
  patch: |-
    - op: replace
      path: /spec/replicas
      value: 3

# Add additional resources:
resources:
- ../base
- extra-configmap.yaml

# Set namespace:
namespace: my-namespace

# Add labels to everything:
commonLabels:
  environment: production

🧪 Testing Commands:

kubectl kustomize components/my-service/development
kubectl kustomize components/my-service/development | kubectl apply --dry-run=client -f -
kubectl kustomize components/my-service/development | kubectl diff -f -

📖 Learn More:
- Kustomize Docs: https://kustomize.io
- Kubectl Kustomize: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/
- infra-deployments: https://github.com/redhat-appstudio/infra-deployments
            

🎨 Understanding Kustomize Overlays

🤔 What Are Kustomize Overlays?