Vault Agent with Kubernetes | Vault | HashiCorp Developer (2024)

Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret.

The applications running in a Kubernetes environment is no exception. Luckily, Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.

However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.

Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

• Automatic authentication
• Secure delivery/storage of tokens
• Lifecycle management of these tokens (renewal & re-authentication)

To perform the tasks described in this tutorial, you need:

• Docker
• Kubernetes command-line interface (CLI)
• minikube
• A running Vault environment reachable from your Kubernetes environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed

Install supporting tools

This tutorial was last tested 13 November 2023 on a macOS 13.5.2 using the following software versions.

$ docker versionClient: Cloud integration: v1.0.35+desktop.5 Version: 24.0.6...

$ kubectl version --shortClient Version: v1.28.3Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3...

$ minikube versionminikube version: v1.32.0commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

1. Retrieve the additional configuration by cloning the hashicorp/learn-vault-agent repository from GitHub.

   $ git clone https://github.com/hashicorp/learn-vault-agent.git

   This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found in a sub-directory.

2. Go into the learn-vault-agent/vault-agent-k8s-demo directory.

   $ cd learn-vault-agent/vault-agent-k8s-demo

Start a Vault server

1. To go through this tutorial, start a Vault dev server which listens for requestslocally at 0.0.0.0:8200 with root as the root token ID.

   $ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

   Setting the -dev-listen-address to 0.0.0.0:8200 overrides the default address of a Vault dev server (127.0.0.1:8200) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.

   Insecure operation

   Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.

2. Export an environment variable for the vault CLI to address the Vault server.

   $ export VAULT_ADDR=http://0.0.0.0:8200

1. Start a Kubernetes cluster running in minikube.

   $ minikube start --driver=docker

2. Wait a couple of minutes for the minikube environment to become fully available.

   $ minikube statusminikubetype: Control Planehost: Runningkubelet: Runningapiserver: Runningkubeconfig: Configured

3. In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server. Open the provided vault-auth-service-account.yaml file in your preferred text editor and examine its content for the service account definition to be used for this tutorial.

   See Also

   How to Capture Logs from Vault Agent running as a Service on WindowsClient-Side Response Caching Using Vault AgentVault Agent Windows service | Vault | HashiCorp DeveloperVault Agent and Vault Proxy quick start | Vault | HashiCorp Developer

   vault-auth-service-account.yaml

   apiVersion: v1kind: ServiceAccountmetadata: name: vault-auth namespace: default---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: role-tokenreview-binding namespace: defaultroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegatorsubjects:- kind: ServiceAccount name: vault-auth namespace: default

4. Create the vault-auth service account.

   $ kubectl apply --filename vault-auth-service-account.yaml

Kubernetes 1.24+ only

1. The service account generated a secret that is required for configuration automatically in Kubernetes 1.23. In Kubernetes 1.24+, you need to create the secret explicitly.

   vault-auth-secret.yaml

   apiVersion: v1kind: Secretmetadata: name: vault-auth-secret annotations: kubernetes.io/service-account.name: vault-authtype: kubernetes.io/service-account-token

2. Create a vault-auth-secret secret.

   $ kubectl apply --filename vault-auth-secret.yaml

Configure Kubernetes auth method

1. Create a read-only policy, myapp-kv-ro in Vault.

   $ vault policy write myapp-kv-ro - <<EOFpath "secret/data/myapp/*" { capabilities = ["read", "list"]}EOF

2. Create some test data at the secret/myapp path.

   $ vault kv put secret/myapp/config \ username='appuser' \ password='suP3rsec(et!' \ ttl='30s'

   Output:

   ====== Secret Path ======secret/data/myapp/config======= Metadata =======Key Value--- -----created_time 2022-03-24T06:09:49.99472Zcustom_metadata <nil>deletion_time n/adestroyed falseversion 1

3. Set the environment variables to point to the running minikube environment.

   Set the SA_SECRET_NAME environment variable value to the vault-auth service account secret.

   $ export SA_SECRET_NAME=$(kubectl get secrets --output=json \ | jq -r '.items[].metadata | select(.name|startswith("vault-auth-")).name')

   Set the SA_JWT_TOKEN environment variable value to the service account JWT used to access the TokenReview API

   $ export SA_JWT_TOKEN=$(kubectl get secret $SA_SECRET_NAME \ --output 'go-template={{ .data.token }}' | base64 --decode)

   Set the SA_CA_CRT environment variable value to the PEM encoded CA cert used to talk to Kubernetes API.

   $ export SA_CA_CRT=$(kubectl config view --raw --minify --flatten \ --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

   Set the K8S_HOST environment variable value to minikube IP address.

   $ export K8S_HOST=$(kubectl config view --raw --minify --flatten \ --output 'jsonpath={.clusters[].cluster.server}')

4. Now, enable and configure the Kubernetes auth method.

   Enable the Kubernetes auth method at the default path ("auth/kubernetes").

   $ vault auth enable kubernetesSuccess! Enabled kubernetes auth method at: kubernetes/

   Tell Vault how to communicate with the Kubernetes (minikube) cluster.

   $ vault write auth/kubernetes/config \ token_reviewer_jwt="$SA_JWT_TOKEN" \ kubernetes_host="$K8S_HOST" \ kubernetes_ca_cert="$SA_CA_CRT" \ issuer="https://kubernetes.default.svc.cluster.local"

   Output:

   Success! Data written to: auth/kubernetes/config

   You can validate the issuer name of your Kubernetes cluster found under Discovering the service account issuer.

5. Create a role named, example, that maps the Kubernetes Service Account to Vault policies and default token TTL.

   $ vault write auth/kubernetes/role/example \ bound_service_account_names=vault-auth \ bound_service_account_namespaces=default \ token_policies=myapp-kv-ro \ ttl=24h

   Output:

   Success! Data written to: auth/kubernetes/role/example

A service bound to all networks on the host, as you configured Vault, is addressable by pods within minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.

1. Start a minikube SSH session.

   $ minikube ssh## ... minikube ssh login

2. Within this SSH session, retrieve the value of the minikube host.TODO: is there a better way?

   $ dig +short host.docker.internal192.168.65.2

   Docker networking

   The host has a changing IP address (or none if you have no network access). It is recommended that you connect to the special DNS name host.docker.internal which resolves to the internal IP address used by the host. This is for development purpose and will not work in production. For more information, review the documentation for Mac, Windows.

3. Next, retrieve the status of the Vault server to verify network connectivity.

   $ dig +short host.docker.internal | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status | python3 -m json.tool{ "type": "shamir", "initialized": true, "sealed": false, "t": 1, "n": 1, "progress": 0, "nonce": "", "version": "1.4.1+ent", "migration": false, "cluster_name": "vault-cluster-3de6c2d3", "cluster_id": "10fd177e-d55a-d740-0c54-26268ed86e31", "recovery_seal": false, "storage_type": "inmem"}

   The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.

4. Next, exit the minikube SSH session.

   $ exit

5. Finally, create a variable named EXTERNAL_VAULT_ADDR to capture the minikube gateway address.

   $ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')

6. Verify that the variable contains the IP address you saw when executed in the minikube shell.

   $ echo $EXTERNAL_VAULT_ADDR192.168.65.2

Optional: Verify the Kubernetes auth method configuration

1. Define a Pod with a container.

   $ cat > devwebapp.yaml <<EOFapiVersion: v1kind: Podmetadata: name: devwebapp labels: app: devwebappspec: serviceAccountName: vault-auth containers: - name: devwebapp image: burtlo/devwebapp-ruby:k8s env: - name: VAULT_ADDR value: "http://$EXTERNAL_VAULT_ADDR:8200"EOF

   The Pod is named devwebapp and runs with the vault-auth service account.

2. Create the devwebapp pod in the default namespace

   $ kubectl apply --filename devwebapp.yaml --namespace defaultpod/devwebapp created

3. Display all the pods in the default namespace. Wait until the devwebapp pod is running and ready (1/1).

   $ kubectl get podsNAME READY STATUS RESTARTS AGEdevwebapp 1/1 Running 0 77s

4. Start an interactive shell session on the devwebapp pod.

   $ kubectl exec --stdin=true --tty=true devwebapp -- /bin/sh#

   Your system prompt is replaced with a new prompt #.

   Note

   The prompt within this section is shown as $ but the commands are intended to be executed within this interactive shell on the devwebapp container.

5. Set KUBE_TOKEN to the service account token.

   $ KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

6. Authenticate with Vault through the example role with the KUBE_TOKEN.

   $ curl --request POST \ --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \ $VAULT_ADDR/v1/auth/kubernetes/login | python3 -m json.tool

   Example output:

   { ...snip... "auth": { "client_token": "hvs.CAESIBxfvzds7M4tas017ls36Dl_kA-3YpCCBT9wczP1E41DGh4KHGh2cy5JYW1NMmFkb2gwTVBhZVhsN0pDM0tOaWM", "accessor": "1Miu5tbfuZzYuYWCntb4Ztke", "policies": [ "default", "myapp-kv-ro" ], "token_policies": [ "default", "myapp-kv-ro" ], "metadata": { "role": "example", "service_account_name": "vault-auth", "service_account_namespace": "default", "service_account_secret_name": "", "service_account_uid": "a293b5fb-96a2-43b0-acee-03d9ffce9423" }, "lease_duration": 86400, "renewable": true, "entity_id": "8925670b-d8d6-0792-ee56-eec1a9e740cb", "token_type": "service", "orphan": true, "mfa_requirement": null, "num_uses": 0 }}

   Notice that client_token is successfully generated and myapp-kv-ro policy is attached with the token. The metadata displays that its service account name (service_account_name) is vault-auth.

7. Lastly, exit the pod.

   $ exit

Start Vault Agent with Auto-Auth

Now that you have verified that the Kubernetes auth method has been configured on the Vault server, it is time to spin up a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.

1. First, open the provided configmap.yaml file in your preferred text editor and review its content.

   configmap.yaml

   apiVersion: v1data: vault-agent-config.hcl: | # Comment this out if running as sidecar instead of initContainer exit_after_auth = true pid_file = "/home/vault/pidfile" auto_auth { method "kubernetes" { mount_path = "auth/kubernetes" config = { role = "example" } } sink "file" { config = { path = "/home/vault/.vault-token" } } } template { destination = "/etc/secrets/index.html" contents = <<EOT <html> <body> <p>Some secrets:</p> {{- with secret "secret/data/myapp/config" }} <ul> <li><pre>username: {{ .Data.data.username }}</pre></li> <li><pre>password: {{ .Data.data.password }}</pre></li> </ul> {{ end }} </body> </html> EOT }kind: ConfigMapmetadata: name: example-vault-agent-config namespace: default

   This creates a Vault Agent configuration file, vault-agent-config.hcl. Notice that the Vault Agent Auto-Auth (auto_auth block) is configured to use the kubernetes auth method enabled at the auth/kubernetes path on the Vault server. The Vault Agent will use the example role which you created in Configure Kubernetes auth method.

   The sink block specifies the location on disk where to write tokens. Vault Agent Auto-Auth sink can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, the sink is set to /home/vault/.vault-token.

   The template block creates a file which retrieves username and password values at the secret/data/myapp/config path.

2. Create a ConfigMap containing a Vault Agent configuration.

   $ kubectl create --filename configmap.yamlconfigmap/example-vault-agent-config created

3. View the created ConfigMap.

   $ kubectl get configmap example-vault-agent-config --output yaml

4. An example Pod spec file is provided. Review the provided example Pod spec file, example-k8s-spec.yaml.

   example-k8s-spec.yaml

   apiVersion: v1kind: Podmetadata: name: vault-agent-example namespace: defaultspec: serviceAccountName: vault-auth volumes: - configMap: items: - key: vault-agent-config.hcl path: vault-agent-config.hcl name: example-vault-agent-config name: config - emptyDir: {} name: shared-data initContainers: - args: - agent - -config=/etc/vault/vault-agent-config.hcl - -log-level=debug env: - name: VAULT_ADDR value: http://EXTERNAL_VAULT_ADDR:8200 image: vault name: vault-agent volumeMounts: - mountPath: /etc/vault name: config - mountPath: /etc/secrets name: shared-data containers: - image: nginx name: nginx-container ports: - containerPort: 80 volumeMounts: - mountPath: /usr/share/nginx/html name: shared-data

   The example Pod spec (example-k8s-spec.yaml) spins up two containers in vault-agent-example pod. A Vault container which runs Vault Agent as an Init Container and an nginx container exposing port 80.

   

   The Vault address, VAULT_ADDR, is set to a placeholder value EXTERNAL_VAULT_ADDR.

5. Generate the Pod spec with EXTERNAL_VAULT_ADDR variable value in its place.

   $ cat example-k8s-spec.yaml | \ sed -e s/"EXTERNAL_VAULT_ADDR"/"$EXTERNAL_VAULT_ADDR"/ \ > vault-agent-example.yaml

6. Create the vault-agent-example pod defined in vault-agent-example.yaml.

   $ kubectl apply --filename vault-agent-example.yaml --record

   This takes a minute or so for the pod to become fully up and running.

Verification

1. In another terminal, launch the minikube dashboard.

   $ minikube dashboard

2. Under Workloads click Pods to verify that vault-agent-example pod has been created successfully.

   

3. Select vault-agent-example to see its details.

   

4. In another terminal, port forwardall requests made to http://localhost:8080 to port 80 on the vault-agent-example pod.

   $ kubectl port-forward pod/vault-agent-example 8080:80

5. In a web browser, go to localhost:8080

   

   Notice that the username and password values were successfully read from secret/myapp/config.

6. Optionally, you can view the HTML source.

   $ kubectl exec -it vault-agent-example --container nginx-container -- cat /usr/share/nginx/html/index.html<html><body><p>Some secrets:</p><ul><li><pre>username: appuser</pre></li><li><pre>password: suP3rsec(et!</pre></li></ul></body></html>

1. Stop the cluster.

   $ minikube stop✋ Stopping node "minikube" ...🛑 Powering off "minikube" via SSH ...🛑 1 node stopped.

2. Delete the cluster.

   $ minikube delete🔥 Deleting "minikube" in docker ...🔥 Deleting container "minikube" ...🔥 Removing /Users/mrken/.minikube/machines/minikube ...💀 Removed all traces of the "minikube" cluster

3. Delete the supporting code repository copied locally.

   $ cd ../.. && rm -rf learn-vault-agent

4. Stop the Vault server process.

   $ pkill vault

Help and reference

• Blog post: Why Use the Vault Agent for Secrets Management?
• Video: Streamline Secrets Management with Vault Agent and Vault 0.11
• Secure Introduction of Vault Clients
• Vault Agent Auto-Auth
• Kubernetes Auth Method
• Kubernetes Auth Method (API)
• Vault Kubernetes Workshop on Google Cloud Platform
• Kubernetes Tutorials
• Consul Template
• Vault Secrets Operator
• Vault Secrets Operator Tutorial

Other Vault Agent tutorials:

• Vault Agent Templates
• Vault Agent Caching
• Vault Agent Windows Service
• Read Secrets From Vault Using Vault Agent
• Using HashiCorp Vault Agent with .NET Core