Syncing secrets and configmaps in Kubernetes with Reflector

Reflector is a nice and small, easy-to-use K8s tool to sync secrets and configmaps between namespaces inside a K8s cluster - a feature which is quite useful if you dont want to - or simply cannot rely on externally managed secrets. It does not introduce new CRDs, but works solely by watching resource annotations on existing secrets and configmaps.

#Kubernetes #K8s #devops 

Author: Thomas Hikade

 

Emberstack's Reflector

Pros

  • A lightweight custom controller installed via a nicely configurable Helm chart.
  • No CRDs, only resource annotations on secrets & configmaps required.
  • push-sync: annotate a source-secret and automatically push it - and subsequent updates - into target namespace(s).
  • pull-sync: create empty destination secrets and annotate them to sync from a central master secret
  • Master secret can define list of namespaces by regex or enumeration. Only these are allowed to sync from master secret.

Cons

  • You need to be able to annotate your secrets and configmaps. In case these are managed by some other controller or operator, annotations on secrets/configmaps will most probably get removed by that operator.
    • proven to work for KafkaUsers and Keycloak user resources
  • Supports Secrets and Configmaps only

Docs

See reflector on Github

Installation

Install via Helm chart

helm repo add emberstack https://emberstack.github.io/helm-charts 
helm repo update
helm repo list
helm search repo emberstack/reflector  [-l]

helm upgrade --install reflector  emberstack/reflector  -n reflector

# --------------------------------------------------------------
# on Openshift 4.x, delete the default Helmchart values for the securityContext
# --------------------------------------------------------------
helm upgrade --install reflector  emberstack/reflector  -n reflector  \
  --set securityContext.runAsUser=null \
  --set podSecurityContext.fsGroup=null

Usecases and examples

Note: all testcases were evaluated using reflector v7.1.238 on Openshift 4.12.

I. Manual reflection (pull-sync)
II. Auto-reflection (push-sync)

I. Manual reflection (pull-sync)

Annotations used:

  • reflection-allowed="true"
  • reflection-allowed-namespaces="slave-.*"

  1. Create the master secret in namespace master:

    $ oc new-project master
$ oc create secret generic foo-secret --from-literal=foo=bar --dry-run=client -o yaml |  \
    oc annotate -f -  --local  --dry-run=client -o yaml  \
        reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
        reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="slave-.*"  \
    > fooSecret.yaml

    # cat fooSecret.yaml
apiVersion: v1
kind: Secret
metadata:
  annotations:
    reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
    reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "slave-.*"
  name: foo-secret
  namespace: master
data:
  foo: YmFy

    $ oc -n master apply -f fooSecret.yaml

  2. Create the reflected secret without data in target namespace slave-01:

    $ oc new-project slave-01

$ oc create secret generic slave-foo-secret --dry-run=client -o yaml |  \
    oc annotate -f -  --local  --dry-run=client -o yaml  \
        my-custom-annotation="foobar" \
        reflector.v1.k8s.emberstack.com/reflects="master/foo-secret" \
    > slave-fooSecret.yaml

    # cat slave-fooSecret.yaml  # NOTICE THE MISSING .data SECTION!!
apiVersion: v1
kind: Secret
metadata:
  annotations:
    my-custom-annotation: foobar
    reflector.v1.k8s.emberstack.com/reflects: master/foo-secret
  name: slave-foo-secret

    $ oc -n slave-01 apply -f slave-fooSecret.yaml

  3. Validate slave-secret data reflection:

    # verify that slave secret data was indeed synced!
$  oc -n slave-01 get secret slave-foo-secret -o yaml | yq -e .data

foo: YmFy

II. Auto-reflection (push-sync)

uses 2 additional annotations to auto-magically copy the entire source-secret to all namespaces in reflection-auto-namespaces annotation:

  • reflection-auto-enabled: "true"
  • reflection-auto-namespaces: slave-02

In this setup, a new target secret will be automatically created in - or pushed out to - namespace slave-02!

  1. Create another source secret in namespace master:

    $ oc project master
$ oc create secret generic foo-push-secret --from-literal=foo2=bar2 --dry-run=client -o yaml |  \
    oc annotate -f -  --local  --dry-run=client -o yaml  \
        reflector.v1.k8s.emberstack.com/reflection-allowed="true" \
                reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces="slave-.*"  \
                reflector.v1.k8s.emberstack.com/reflection-auto-enabled="true"  \
                reflector.v1.k8s.emberstack.com/reflection-auto-namespaces="slave-02"  \
         > foo-push-secret.yaml

    # cat foo-push-secret
apiVersion: v1
kind: Secret
metadata:
  annotations:
    reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
    reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "slave-.*"
    reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
    reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "slave-02"
  name: foo-push-secret
  namespace: master        
data:
  foo2: YmFyMg==

    $ oc -n master apply -f foo-push-secret.yaml

  2. Create namespace slave-02 and do not create any target secret!
    Be sure to tail reflector pod logs to verify what happends immediately after you create the new namespace!

    $ oc -n reflector logs deploy/reflector -f &

$ oc new-project slave-02
...


2024-01-29 15:55:52.112 +00:00 [INF] (SecretMirror) Created slave-02/foo-push-secret as a reflection of master/foo-push-secret

  3. Validate auto-reflection of secret

    # verify that a new secret `foo-push-secret` was created!
oc -n slave-02 get secret foo-push-secret -o yaml | yq -e .data

foo2: YmFyMg==

Troubleshooting

  • If things do not sync as expected, have a look the logs of the reflector controller pod. You might see hints like these to find the root-cause: 
    $ oc -n reflector logs deploy/reflector

2024-01-29 14:12:44.429 +00:00 [INF] () Starting host
2024-01-29 15:18:08.001 +00:00 [WRN] (ES.Kubernetes.Reflector.Core.SecretMirror) Could not update slave-01/slave-foo-secret - Source master/foo-secret does not permit it.
  • In pull-sync mode, if you had a typo in the reflection-allowed-namespaces annotation value, reflector wont try to update the target secret again, even if you fix the typo in the master secret. You will have to recreate the empty target secret to trigger a sync.

Summary

Reflector has got a quite specific and narrow use-case - but that it does pretty well!
You no longer need to (ab-)use Helm lookup function to copy secrets - which won't work if you use Helm via ArgoCD anyway!
So if you want to keep a central secret / configmap namespace and replicate those resources inside the cluster, give reflector a try!

zurück