Kustomize overlays when using a shared ConfigMap

I have an environment made of pods that address their target environment based on an environment variable called CONF_ENV that could be test, stage or prod.

The application running inside the Pod has the same source code across environments, the configuration file is picked according to the CONF_ENV environment variable.

I'v encapsulated this CONF_ENV in *.properties files just because I may have to add more environment variables later, but I make sure that each property file contains the expected CONF_ENV e.g.:

• test.properites has CONF_ENV=test,
• prod.properties has CONF_ENV=prod, and so on...

I struggle to make this work with Kustomize overlays, because I want to define a ConfigMap as a shared resource across all the pods within the same overlay e.g. test (each pod in their own directory, along other stuff when needed).

So the idea is:

• base/ (shared) with the definition of the Namespace, the ConfigMap (and potentially other shared resources
• base/pod1/ with the definition of pod1 picking from the shared ConfigMap (this defaults to test, but in principle it could be different)

Then the overlays:

• overlay/test that patches the base with CONF_ENV=test (e.g. for overlay/test/pod1/ and so on)
• overlay/prod/ that patches the base with CONF_ENV=prod (e.g. for overlay/prod/pod1/ and so on)

Each directory with their own kustomize.yaml.

The above doesn't work because when going into e.g. overlay/test/pod1/ and I invoke the command kubectl kustomize . to check the output YAML, then I get all sorts of errors depending on how I defined the lists for the YAML keys bases: or resources:.

I am trying to share the ConfigMap across the entire CONF_ENV environment in an attempt to minimize the boilerplate YAML by leveraging the patching-pattern with Kustomize.

The Kubernetes / Kustomize YAML directory structure works like this:

├── base
│   ├── configuration.yaml # I am trying to share this!
│   ├── kustomization.yaml
│   ├── my_namespace.yaml # I am trying to share this!
│   ├── my-scheduleset-etl-misc
│   │   ├── kustomization.yaml
│   │   └── my_scheduleset_etl_misc.yaml
│   ├── my-scheduleset-etl-reporting
│   │   ├── kustomization.yaml
│   │   └── my_scheduleset_etl_reporting.yaml
│   └── test.properties # I am trying to share this!
└── overlay
    └── test
        ├── kustomization.yaml # here I want tell "go and pick up the shared resources in the base dir"
        ├── my-scheduleset-etl-misc
        │   ├── kustomization.yaml
        │   └── test.properties # I've tried to share this one level above, but also to add this inside the "leaf" level for a given pod
        └── my-scheduleset-etl-reporting
            └── kustomization.yaml

The command kubectl with Kustomize:

• sometimes complains that the shared namespace does not exist:

error: merging from generator &{0xc001d99530 {  map[] map[]} {{ my-schedule-set-props merge {[CONF_ENV=test] [] [] } <nil>}}}: 
id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", isClusterScoped:false}, Name:"my-schedule-set-props", Namespace:""} 
does not exist; cannot merge or replace

• sometimes doesn't allow to have shared resources inside an overlay:

error: loading KV pairs: env source files: [../test.properties]: 
security; file '/my/path/to/yaml/overlay/test/test.properties' 
is not in or below '/my/path/to/yaml/overlay/test/my-scheduleset-etl-misc'

• sometimes doesn't allow cycles when I am trying to have multiple bases - the shared resources and the original pod definition:

error: accumulating resources: accumulation err='accumulating resources from '../': 
'/my/path/to/yaml/overlay/test' must resolve to a file': 
cycle detected: candidate root '/my/path/to/yaml/overlay/test' 
contains visited root '/my/path/to/yaml/overlay/test/my-scheduleset-etl-misc'

The overlay kustomization.yaml files inside the pod dirs have:

bases:
  - ../ # tried with/without this to share the ConfigMap
  - ../../../base/my-scheduleset-etl-misc/

The kustomization.yaml at the root of the overlay has:

bases:
  - ../../base

The kustomization.yaml at the base dir contains this configuration for the ConfigMap:

# https://gist.github.com/hermanbanken/3d0f232ffd86236c9f1f198c9452aad9
configMapGenerator:
  - name: my-schedule-set-props
    namespace: my-ss-schedules
    envs:
      - test.properties

vars:
  - name: CONF_ENV
    objref:
      kind: ConfigMap
      name: my-schedule-set-props
      apiVersion: v1
    fieldref:
      fieldpath: data.CONF_ENV

configurations:
  - configuration.yaml

With configuration.yaml containing:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

varReference:
- path: spec/confEnv/value
  kind: Pod

How do I do this?

How do I make sure that I minimise the amount of YAML by sharing all the ConfigMap stuff and the Pods definitions as much as I can?