Use Secretless with CyberArk AAM Dynamic Access Provider

If your database credentials are stored in CyberArk AAM Dynamic Access Provider (DAP), and your app is running in Kubernetes or OpenShift, you need a method to connect to the database without handling the credentials directly to avoid leaking the secrets.

This topic describes how to solve this problem using Secretless. The application receives the connection to the target system instead of access to secrets. The application connects to Secretless, which connects to the target. To do this, Secretless runs as a sidecar container next to your app, and your app connects to Secretless Broker over a local TCP or UNIX socket.

Prerequisites

Assumptions

  • You have installed and configured your DAP Master cluster, and you have deployed a DAP Follower to your OpenShift / Kubernetes cluster that is configured to use the DAP Kubernetes authenticator.

  • You have an application that requires a supported database. If you need a sample app to try, we have a pet store demo app that works with MySQL, PostgreSQL, and MSSQL.

  • Your database is running and accessible to apps in your OpenShift / Kubernetes environment, it supports SSL, and the credentials for it are stored in DAP.

  • You are using a supported version of OpenShift or Kubernetes.

OpenShift vs Kubernetes

In this topic, the code samples assume you are using Kubernetes. If you are using OpenShift instead:

  • Replace kubectl with oc in all code snippets.

  • Replace references to kubernetes: "true" in DAP host annotations with the OpenShift analogue, openshift: "true".

  • References to namespaces may be equivalently considered references to OpenShift projects.

Required Information

To deploy Secretless, you need the following information about your DAP configuration and your application:

Required information

Description

How we refer to it

App Name

The Kubernetes name of the application.

${APP_NAME}

App Namespace

The Kubernetes namespace where the application pods will be deployed.

${APP_NAMESPACE}

App Service Account Name

The Kubernetes service account assigned to the application pods.

${APP_SERVICE_ACCOUNT_NAME}

Secretless Container Name

The name of the Secretless container within the application pod.

${SECRETLESS_CONTAINER_NAME}

Kubernetes authenticator ID

The ID of the Kubernetes authenticator configured in your DAP Follower.

${AUTHENTICATOR_ID}

Policy branch where kubernetes application identities are defined

The fully qualified DAP ID of a policy branch where new Kubernetes application host identities should be added.

${APP_POLICY_BRANCH}

Policy branch with database credentials

The fully qualified ID of the policy branch in DAP that contains your database credentials.

${APP_SECRETS_POLICY_BRANCH}

Layer/group with access to database credentials

The fully qualified DAP ID of a layer or group whose members have access to the database secrets in ${APP_SECRETS_POLICY_BRANCH}.

The examples in this topic refer to a layer; if you are provided with a group, replace all references to !layer ${APP_SECRETS_READER_LAYER} with !group ${APP_SECRETS_READER_LAYER}.

${APP_SECRETS_READER_LAYER}

DAP account

The account name configured for your DAP deployment.

${CONJUR_ACCOUNT}

DAP Kubernetes service name

The Kubernetes service name for the DAP instance deployed in your cluster.

${CONJUR_SERVICE_NAME}

DAP Kubernetes service account

The Kubernetes service account associated with your cluster’s DAP instance.

${CONJUR_SERVICE_ACCOUNT_NAME}

DAP Kubernetes namespace

The Kubernetes namespace in your cluster where the DAP instance is deployed. Secretless retrieves the database credentials from this DAP instance.

${CONJUR_NAMESPACE}

DAP ClusterRole name

The DAP Kubernetes authenticator ClusterRole defined with the required Kubernetes RBAC permissions.

${CONJUR_CLUSTERROLE_NAME}

X.509 CA Certificate Chain (optional)

The PEM encoded X.509 certificate chain for the DAP instance deployed to your cluster.

conjur.pem

Sample code snippets

To assist you in deploying Secretless, copy the relevant code snippet below. It contains all of the variables defined in the table above. Modify the variables to reflect your environment and save the file as env.sh in your working directory. The env.sh file is sourced in other code snippets provided in this topic. Ensure that you execute all provided code snippets in a BASH shell or the equivalent.

Additional code snippets are provided that can be used to generate DAP policy and Kubernetes resource manifests, and to carry out actions.

 
#!/usr/bin/env bash

APP_NAME=my-app
APP_NAMESPACE=my-app-ns
APP_SERVICE_ACCOUNT_NAME=my-app-sa
SECRETLESS_CONTAINER_NAME=secretless

AUTHENTICATOR_ID="example-service-id"
APP_POLICY_BRANCH="conjur/authn-k8s/${AUTHENTICATOR_ID}/apps"

APP_SECRETS_POLICY_BRANCH="apps/prod-secrets"
APP_SECRETS_READER_LAYER="apps/prod-secrets/secret-users"

CONJUR_ACCOUNT="default"
CONJUR_SERVICE_NAME="conjur-follower"
CONJUR_SERVICE_ACCOUNT_NAME="conjur-cluster"
CONJUR_NAMESPACE="conjur-ns"
CONJUR_CLUSTERROLE_NAME="conjur-authenticator-${CONJUR_NAMESPACE}"

CONJUR_APPLIANCE_URL="https://${CONJUR_SERVICE_NAME}.${CONJUR_NAMESPACE}.svc.cluster.local"

 

Add your application to DAP policy

Submit a request to your DAP administrator to create a host identity for your application. You can define your host using a variety of Kubernetes resources. For details, see Application Identity in OpenShift/Kubernetes.

Enable the Kubernetes authenticator for your application

To enable your application to authenticate to DAP using the Kubernetes authenticator, your application must have a properly configured DAP host identity.

For example, if you have chosen service account-based authentication for your application, you can use the following BASH code snippet to generate a policy named app-policy.yml that defines the host and adds it to the layer that can access your Kubernetes authenticator’s webservice:

 
#!/usr/bin/env bash
. ./env.sh

cat << EOL > app-policy.yml
---
# Policy to define your application and enable the Kubernetes authenticator
- !host
  id: service-account-based-app
  annotations:
    authn-k8s/namespace: ${APP_NAMESPACE}
    authn-k8s/service-account: ${APP_SERVICE_ACCOUNT_NAME}
    authn-k8s/authentication-container-name: ${SECRETLESS_CONTAINER_NAME}
    kubernetes: "true"

# Add your host to the layer that has authenticate permission on the authenticator
# webservice
- !grant
  role: !layer
  members: !host service-account-based-app
EOL

To apply the policy, run the following command:

 
#!/usr/bin/env bash
. ./env.sh

conjur policy load ${APP_POLICY_BRANCH} app-policy.yml

Grant access to the database credentials

Now that the application host is defined in DAP policy, we can give the host access to the database credentials by adding it to the ${APP_SECRETS_READER_LAYER} layer.

You can use the following BASH code snippet to generate a policy named app-permissions.yml that grants the host access to database credentials in DAP:

 
#!/usr/bin/env bash
. ./env.sh

cat << EOL > app-permissions.yml
---
# Grant application's identity membership to the application secrets reader layer
# so the app identity inherits read / execute privileges on app's database secrets
- !grant
  role: !layer ${APP_SECRETS_READER_LAYER}
  members:
  - !host ${APP_POLICY_BRANCH}/service-account-based-app
EOL

To apply the policy run the command:

 
#!/usr/bin/env bash

conjur policy load root app-permissions.yml

Prepare the application namespace

Grant DAP access to pods in the application namespace

The Kubernetes authenticator uses the Kubernetes API to verify the identity of the authenticating application and to inject signed certificates and access tokens into the application pod. When you deploy DAP, you also create a ClusterRole with the permissions the DAP Kubernetes authenticator will need. Once you are ready to deploy an application in a new namespace, you must create a RoleBinding that grants the DAP service account the privileges defined in the DAP ClusterRole for all pods in the application namespace.

To create the manifest for the RoleBinding, run the following:

 
#!/usr/bin/env bash
. ./env.sh

cat << EOL > conjur-authenticator-role-binding.yml
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: conjur-authenticator-role-binding-${APP_NAMESPACE}
  namespace: ${APP_NAMESPACE}
subjects:
  - kind: ServiceAccount
    name: ${CONJUR_SERVICE_ACCOUNT_NAME}
    namespace: ${CONJUR_NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ${CONJUR_CLUSTERROLE_NAME}
EOL

To apply it, run:

 
kubectl create -f conjur-authenticator-role-binding.yml

Store the DAP SSL certificate in a ConfigMap

To secure the connection between Secretless and DAP, Secretless needs to send the DAP X.509 CA certificate chain with its requests to retrieve secrets. To make the certificate available to Secretless, you can store it in a ConfigMap in the application namespace. The certificate file may be provided to you, but if not, you can retrieve it as follows.

Run the following command to port-forward connections to the DAP follower:

 
#!/usr/bin/env bash
. ./env.sh

conjur_follower_pod="$(
  kubectl get pods \
  -n ${CONJUR_NAMESPACE} \
  --selector app="conjur-follower" \
  --no-headers \
  -o jsonpath='{.items[].metadata.name}'
)"
kubectl port-forward "${conjur_follower_pod}" 8443:443

With port-forwarding on, you can now retrieve the certificate:

 
openssl s_client -showcerts \
  -connect localhost:8443 </dev/null 2>/dev/null \
  | sed -n '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > conjur.pem

Once you have the DAP SSL certificate in a file conjur.pem, you can turn off port forwarding and store it in a ConfigMap named conjur-ssl-cert:

 
#!/usr/bin/env bash
. ./env.sh

kubectl \
  --namespace "${APP_NAMESPACE}" \
  create configmap \
  conjur-ssl-cert \
  --from-file=ssl-certificate="conjur.pem"

Prepare the Secretless configuration

Secretless uses its configuration to determine where to listen for new connection requests, where to route those connections, and where to get the credentials for each connection.

Let’s assume that we’re using a PostgreSQL database and we’ve stored its host, port, username, and password in the DAP ${APP_SECRETS_POLICY_BRANCH} policy branch.

Secretless recognizes DAP by the secret provider name conjur and refers to the PostgreSQL using the connector identifier pg.

The Secretless configuration for this single PostgreSQL service connector that retrieves its credentials from DAP can be generated by running:

 
#!/usr/bin/env bash
. ./env.sh

cat << EOL > secretless.yml
version: "2"
services:
  postgres-sample:
    connector: pg
    listenOn: tcp://0.0.0.0:5432
    credentials:
      host:
        from: conjur
        get: ${APP_SECRETS_POLICY_BRANCH}/host
      port:
        from: conjur
        get: ${APP_SECRETS_POLICY_BRANCH}/port
      password:
        from: conjur
        get: ${APP_SECRETS_POLICY_BRANCH}/password
      username:
        from: conjur
        get: ${APP_SECRETS_POLICY_BRANCH}/username
EOL

The configuration above instructs Secretless to listen on port 5432 for an incoming PostgreSQL connection. The credentials come from the referenced variables stored in the ${APP_SECRETS_POLICY_BRANCH} DAP policy branch.

For your own application, update the individual secret references to refer to the actual secret names used in your DAP policy.

After generating the Secretless configuration, store it in a ConfigMap by running the following:

 
#!/usr/bin/env bash
. ./env.sh

kubectl \
  --namespace "${APP_NAMESPACE}" \
  create configmap \
  secretless-config \
  --from-file=secretless.yml
 

The example above is for the PostgreSQL service connector, but other database connectors have different connector identifiers and may require different credentials.

Refer to the individual Service Connector pages for more information.

You may find it convenient to check the secretless.yml file in with your application code, since this configuration is specific to your application.

Deploy your application with the Secretless sidecar

You deploy Secretless as a sidecar in the same pod as your application. To do this, update your application to replace your database connection with a connection to Secretless, and update your manifest to include the Secretless container definition, mounting its configuration ConfigMap as a volume.

Update your application

You can delete any code in your application that fetches credentials from a vault. Instead, your database client can simply connect to the address on which Secretless is listening. No credentials are required. Secretless automatically injects the credentials into the connection request and authenticates on your behalf.

Following our example, you would configure your application’s database client to connect to localhost:5432 — the port on which Secretless is listening for incoming PostgreSQL connections.

Add Secretless to your app deployment manifest

The final step is to add the Secretless Broker container definition to your application manifest, including its DAP configuration and the mounted ConfigMaps for the DAP X.509 CA certificate chain and the application’s secretless.yml configuration file.

To help generate a valid manifest, you can run the following command to generate a Kubernetes Deployment manifest with an application called app and Secretless:

 
#!/usr/bin/env bash
. ./env.sh

cat << EOL > app-manifest.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ${APP_NAME}
  name: ${APP_NAME}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${APP_NAME}
  template:
    metadata:
      labels:
        app: ${APP_NAME}
    spec:
      serviceAccountName: ${APP_SERVICE_ACCOUNT_NAME}
      containers:
      # Insert your container definition here
      # Ensure it's configured to point to Secretless for the DB connection
      #
      # For example:
      # - name: ${my-app}
      #   image: ${my-app}:latest
      #   env:
      #     - name: DB_URL
      #       value: "postgresql://localhost:5432/my-db-name"
      - name: secretless
        image: cyberark/secretless-broker:latest
        imagePullPolicy: Always
        args: ["-f", "/etc/secretless/secretless.yml"]
        ports:
        - containerPort: 5432
        env:
          - name: CONJUR_APPLIANCE_URL
            value: ${CONJUR_APPLIANCE_URL}
          - name: CONJUR_ACCOUNT
            value: ${CONJUR_ACCOUNT}
          - name: CONJUR_AUTHN_URL
            value: "${CONJUR_APPLIANCE_URL}/authn-k8s/${AUTHENTICATOR_ID}"
          - name: CONJUR_AUTHN_LOGIN
            value: "host/${APP_POLICY_BRANCH}/service-account-based-app"
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: CONJUR_SSL_CERTIFICATE
            valueFrom:
              configMapKeyRef:
                key: ssl-certificate
                name: conjur-ssl-cert
        volumeMounts:
          - mountPath: /etc/secretless
            name: config
            readOnly: true
      volumes:
        - name: config
          configMap:
            name: secretless-config
            defaultMode: 420
EOL

In practice, you need to modify this manifest to include your application’s container definition.

 

The ${AUTHENTICATOR_ID} in the CONJUR_AUTHN_URL environment variable (of the Secretless sidecar container) must be URL encoded.

 

 

The manifest above assumes you’ve already created the ${APP_SERVICE_ACCOUNT_NAME} service account.

After generating the application manifest, deploy the application by running:

 
#!/usr/bin/env bash
. ./env.sh

kubectl create -n ${APP_NAMESPACE} -f app-manifest.yml

Your application can now connect to PostgreSQL via Secretless without ever knowing your database credentials.

Next steps