Use Secretless with CyberArk AAM Dynamic Access Provider

If your database credentials are stored in CyberArk AAM Dynamic Access Provider (DAP), and your app is running in Kubernetes or OpenShift, you need a method to connect to the database without handling the credentials directly to avoid leaking the secrets.

This topic describes how to solve this problem using Secretless. The application receives the connection to the target system instead of access to secrets. The application connects to Secretless, which connects to the target. To do this, Secretless runs as a sidecar container next to your app, and your app connects to Secretless Broker over a local TCP or UNIX socket.

Prerequisites

Assumptions

• You have installed and configured your DAP Master cluster, and you have deployed a DAP Follower to your OpenShift / Kubernetes cluster that is configured to use the DAP Kubernetes authenticator.

• You have an application that requires a supported database. If you need a sample app to try, we have a pet store demo app that works with MySQL, PostgreSQL, and MSSQL.

• Your database is running and accessible to apps in your OpenShift / Kubernetes environment, it supports SSL, and the credentials for it are stored in DAP.

• You are using a supported version of OpenShift or Kubernetes.

OpenShift vs Kubernetes

In this topic, the code samples assume you are using Kubernetes. If you are using OpenShift instead:

• Replace kubectl with oc in all code snippets.

• Replace references to kubernetes: "true" in DAP host annotations with the OpenShift analogue, openshift: "true".

• References to namespaces may be equivalently considered references to OpenShift projects.

Required Information

To deploy Secretless, you need the following information about your DAP configuration and your application:

Sample code snippets

To assist you in deploying Secretless, copy the relevant code snippet below. It contains all of the variables defined in the table above. Modify the variables to reflect your environment and save the file as env.sh in your working directory. The env.sh file is sourced in other code snippets provided in this topic. Ensure that you execute all provided code snippets in a BASH shell or the equivalent.

Additional code snippets are provided that can be used to generate DAP policy and Kubernetes resource manifests, and to carry out actions.

#!/usr/bin/env bash

APP_NAME=my-app
APP_NAMESPACE=my-app-ns
APP_SERVICE_ACCOUNT_NAME=my-app-sa
SECRETLESS_CONTAINER_NAME=secretless

AUTHENTICATOR_ID="example-service-id"
APP_POLICY_BRANCH="conjur/authn-k8s/${AUTHENTICATOR_ID}/apps"

APP_SECRETS_POLICY_BRANCH="apps/prod-secrets"
APP_SECRETS_READER_LAYER="apps/prod-secrets/secret-users"

CONJUR_ACCOUNT="default"
CONJUR_SERVICE_NAME="conjur-follower"
CONJUR_SERVICE_ACCOUNT_NAME="conjur-cluster"
CONJUR_NAMESPACE="conjur-ns"
CONJUR_CLUSTERROLE_NAME="conjur-authenticator-${CONJUR_NAMESPACE}"

CONJUR_APPLIANCE_URL="https://${CONJUR_SERVICE_NAME}.${CONJUR_NAMESPACE}.svc.cluster.local"

Add your application to DAP policy

Submit a request to your DAP administrator to create a host identity for your application. You can define your host using a variety of Kubernetes resources. For details, see Application Identity in OpenShift/Kubernetes.

Enable the Kubernetes authenticator for your application

To enable your application to authenticate to DAP using the Kubernetes authenticator, your application must have a properly configured DAP host identity.

For example, if you have chosen service account-based authentication for your application, you can use the following BASH code snippet to generate a policy named app-policy.yml that defines the host and adds it to the layer that can access your Kubernetes authenticator’s webservice:

#!/usr/bin/env bash
. ./env.sh

cat << EOL > app-policy.yml
---
# Policy to define your application and enable the Kubernetes authenticator
- !host
  id: service-account-based-app
  annotations:
    authn-k8s/namespace: ${APP_NAMESPACE}
    authn-k8s/service-account: ${APP_SERVICE_ACCOUNT_NAME}
    authn-k8s/authentication-container-name: ${SECRETLESS_CONTAINER_NAME}
    kubernetes: "true"

# Add your host to the layer that has authenticate permission on the authenticator
# webservice
- !grant
  role: !layer
  members: !host service-account-based-app
EOL

To apply the policy, run the following command:

#!/usr/bin/env bash
. ./env.sh

conjur policy load ${APP_POLICY_BRANCH} app-policy.yml

Grant access to the database credentials

Now that the application host is defined in DAP policy, we can give the host access to the database credentials by adding it to the ${APP_SECRETS_READER_LAYER} layer.

You can use the following BASH code snippet to generate a policy named app-permissions.yml that grants the host access to database credentials in DAP:

#!/usr/bin/env bash
. ./env.sh

cat << EOL > app-permissions.yml
---
# Grant application's identity membership to the application secrets reader layer
# so the app identity inherits read / execute privileges on app's database secrets
- !grant
  role: !layer ${APP_SECRETS_READER_LAYER}
  members:
  - !host ${APP_POLICY_BRANCH}/service-account-based-app
EOL

To apply the policy run the command:

#!/usr/bin/env bash

conjur policy load root app-permissions.yml

Prepare the application namespace

Grant DAP access to pods in the application namespace

The Kubernetes authenticator uses the Kubernetes API to verify the identity of the authenticating application and to inject signed certificates and access tokens into the application pod. When you deploy DAP, you also create a ClusterRole with the permissions the DAP Kubernetes authenticator will need. Once you are ready to deploy an application in a new namespace, you must create a RoleBinding that grants the DAP service account the privileges defined in the DAP ClusterRole for all pods in the application namespace.

To create the manifest for the RoleBinding, run the following:

#!/usr/bin/env bash
. ./env.sh

cat << EOL > conjur-authenticator-role-binding.yml
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: conjur-authenticator-role-binding-${APP_NAMESPACE}
  namespace: ${APP_NAMESPACE}
subjects:
  - kind: ServiceAccount
    name: ${CONJUR_SERVICE_ACCOUNT_NAME}
    namespace: ${CONJUR_NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ${CONJUR_CLUSTERROLE_NAME}
EOL

To apply it, run:

kubectl create -f conjur-authenticator-role-binding.yml

Store the DAP SSL certificate in a ConfigMap

To secure the connection between Secretless and DAP, Secretless needs to send the DAP X.509 CA certificate chain with its requests to retrieve secrets. To make the certificate available to Secretless, you can store it in a ConfigMap in the application namespace. The certificate file may be provided to you, but if not, you can retrieve it as follows.

Run the following command to port-forward connections to the DAP follower:

#!/usr/bin/env bash
. ./env.sh

conjur_follower_pod="$(
  kubectl get pods \
  -n ${CONJUR_NAMESPACE} \
  --selector app="conjur-follower" \
  --no-headers \
  -o jsonpath='{.items[].metadata.name}'
)"
kubectl port-forward "${conjur_follower_pod}" 8443:443

With port-forwarding on, you can now retrieve the certificate:

openssl s_client -showcerts \
  -connect localhost:8443 </dev/null 2>/dev/null \
  | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > conjur.pem

Once you have the DAP SSL certificate in a file conjur.pem, you can turn off port forwarding and store it in a ConfigMap named conjur-ssl-cert:

#!/usr/bin/env bash
. ./env.sh

kubectl \
  --namespace "${APP_NAMESPACE}" \
  create configmap \
  conjur-ssl-cert \
  --from-file=ssl-certificate="conjur.pem"

Prepare the Secretless configuration

Secretless uses its configuration to determine where to listen for new connection requests, where to route those connections, and where to get the credentials for each connection.

Let’s assume that we’re using a PostgreSQL database and we’ve stored its host, port, username, and password in the DAP ${APP_SECRETS_POLICY_BRANCH} policy branch.

Secretless recognizes DAP by the secret provider name conjur and refers to the PostgreSQL using the connector identifier pg.

The Secretless configuration for this single PostgreSQL service connector that retrieves its credentials from DAP can be generated by running:

#!/usr/bin/env bash
. ./env.sh

cat << EOL > secretless.yml
version: "2"
services:
  postgres-sample:
    connector: pg
    listenOn: tcp://0.0.0.0:5432
    credentials:
      host:
        from: conjur
        get: ${APP_SECRETS_POLICY_BRANCH}/host
      port:
        from: conjur
        get: ${APP_SECRETS_POLICY_BRANCH}/port
      password:
        from: conjur
        get: ${APP_SECRETS_POLICY_BRANCH}/password
      username:
        from: conjur
        get: ${APP_SECRETS_POLICY_BRANCH}/username
EOL

The configuration above instructs Secretless to listen on port 5432 for an incoming PostgreSQL connection. The credentials come from the referenced variables stored in the ${APP_SECRETS_POLICY_BRANCH} DAP policy branch.

For your own application, update the individual secret references to refer to the actual secret names used in your DAP policy.

After generating the Secretless configuration, store it in a ConfigMap by running the following:

#!/usr/bin/env bash
. ./env.sh

kubectl \
  --namespace "${APP_NAMESPACE}" \
  create configmap \
  secretless-config \
  --from-file=secretless.yml

You may find it convenient to check the secretless.yml file in with your application code, since this configuration is specific to your application.

Deploy your application with the Secretless sidecar

You deploy Secretless as a sidecar in the same pod as your application. To do this, update your application to replace your database connection with a connection to Secretless, and update your manifest to include the Secretless container definition, mounting its configuration ConfigMap as a volume.

Update your application

You can delete any code in your application that fetches credentials from a vault. Instead, your database client can simply connect to the address on which Secretless is listening. No credentials are required. Secretless automatically injects the credentials into the connection request and authenticates on your behalf.

Following our example, you would configure your application’s database client to connect to localhost:5432 — the port on which Secretless is listening for incoming PostgreSQL connections.

Add Secretless to your app deployment manifest

The final step is to add the Secretless Broker container definition to your application manifest, including its DAP configuration and the mounted ConfigMaps for the DAP X.509 CA certificate chain and the application’s secretless.yml configuration file.

To help generate a valid manifest, you can run the following command to generate a Kubernetes Deployment manifest with an application called app and Secretless:

#!/usr/bin/env bash
. ./env.sh

cat << EOL > app-manifest.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ${APP_NAME}
  name: ${APP_NAME}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${APP_NAME}
  template:
    metadata:
      labels:
        app: ${APP_NAME}
    spec:
      serviceAccountName: ${APP_SERVICE_ACCOUNT_NAME}
      containers:
      # Insert your container definition here
      # Ensure it's configured to point to Secretless for the DB connection
      #
      # For example:
      # - name: ${my-app}
      #   image: ${my-app}:latest
      #   env:
      #     - name: DB_URL
      #       value: "postgresql://localhost:5432/my-db-name"
      - name: secretless
        image: cyberark/secretless-broker:latest
        imagePullPolicy: Always
        args: ["-f", "/etc/secretless/secretless.yml"]
        ports:
        - containerPort: 5432
        env:
          - name: CONJUR_APPLIANCE_URL
            value: ${CONJUR_APPLIANCE_URL}
          - name: CONJUR_ACCOUNT
            value: ${CONJUR_ACCOUNT}
          - name: CONJUR_AUTHN_URL
            value: "${CONJUR_APPLIANCE_URL}/authn-k8s/${AUTHENTICATOR_ID}"
          - name: CONJUR_AUTHN_LOGIN
            value: "host/${APP_POLICY_BRANCH}/service-account-based-app"
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: CONJUR_SSL_CERTIFICATE
            valueFrom:
              configMapKeyRef:
                key: ssl-certificate
                name: conjur-ssl-cert
        volumeMounts:
          - mountPath: /etc/secretless
            name: config
            readOnly: true
      volumes:
        - name: config
          configMap:
            name: secretless-config
            defaultMode: 420
EOL

In practice, you need to modify this manifest to include your application’s container definition.

After generating the application manifest, deploy the application by running:

#!/usr/bin/env bash
. ./env.sh

kubectl create -n ${APP_NAMESPACE} -f app-manifest.yml

Your application can now connect to PostgreSQL via Secretless without ever knowing your database credentials.

Next steps

• Learn about the supported Secret Providers (Secretless).

• Learn about the supported Secretless Service Connectors.