Troubleshoot Secret Providers (Secretless)

This topic describes common troubleshooting issues for credential (secret) providers.

Error

Suggested resolution

Log output

Conjur provider not set up properly

Ensure that you have specified all of the required details for Conjur-based credential retrieval.

 
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:34 Instantiating provider 'conjur'
2019/10/30 15:06:34 ERROR: Provider 'conjur' could not be used! ERROR: Unable to construct a Conjur provider client from the available credentials
# Program exit

OR

...
2019/10/30 15:06:34 ERROR: Resolving variable 'simple/basic/variable' from provider 'conjur' failed: Post https://localhost/authn/myorg/admin/authenticate: x509: certificate signed by unknown authority

Sample client log output messages

ERROR: MySQL Error 2003 (HY000): Can't connect to MySQL server on 'localhost' (61)
ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'simple/basic/variable' from provider 'conjur' failed: Post https://localhost/authn/myorg/admin/authenticate: x509: certificate signed by unknown authority

Conjur server is unreachable

  • Ensure that the Conjur server is configured properly.

  • Ensure that the

    CONJUR_APPLIANCE_URL is set to the correct and accessible server destination.

 

2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:47 Instantiating provider 'conjur'
2019/10/30 15:06:47 Info: Conjur provider using API key-based authentication
2019/10/30 15:06:47 ERROR: Resolving variable 'simple/basic/variable' from provider 'conjur' failed: Post https://nopelocalhost/authn/myorg/admin/authenticate: dial tcp: lookup nopelocalhost: no such host

Sample client log output messages

ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'simple/basic/variable' from provider 'conjur' failed: Post https://nopelocalhost/authn/myorg/admin/authenticate: dial tcp: lookup nopelocalhost: no such host

Specified Conjur variable cannot be found

  • Ensure Conjur has the specified variable ID stored.

  • Ensure that the API key can retrieve the specified variable ID.

 
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:29 Instantiating provider 'conjur'
2019/10/30 15:06:29 Info: Conjur provider using API key-based authentication
2019/10/30 15:06:29 ERROR: Resolving variable 'simple/basic/variables' from provider 'conjur' failed: 404 Not Found. Variable 'simple/basic/variables' not found in account 'myorg'

Sample client log output messages

ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'simple/basic/variables' from provider 'conjur' failed: 404 Not Found. Variable 'simple/basic/variables' not found in account 'myorg'.


Insufficient permissions to read specified Conjur variable

  • Ensure that the user specified for Conjur has the appropriate permissions on the requested variable ID.

  • Connect directly to Conjur with the specified credentials to ensure that your backend configuration is correct.

 
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:44 Instantiating provider 'conjur'
2019/10/30 15:06:44 Info: Conjur provider using API key-based authentication
2019/10/30 15:06:44 ERROR: Resolving variable 'simple/basic/variables' from provider 'conjur' failed: 404 Not Found. Variable 'simple/basic/variables' not found in account 'myorg'

Sample client log output messages

ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'simple/basic/variables' from provider 'conjur' failed: 404 Not Found. Variable 'simple/basic/variables' not found in account 'myorg'.

Kubernetes authenticator has unreachable Conjur endpoint

  • Ensure CONJUR_APPLIANCE_URL and CONJUR_AUTHN_URL are valid parameters in the Secretless sidecar configuration.

  • Ensure network connectivity exists between sidecar containers and Conjur.

 
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:14 Instantiating provider 'conjur'
2019/10/30 15:06:14 Info: Conjur provider using Kubernetes authenticator-based authentication
2019/10/30 15:06:14 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:14 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:16 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:16 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:21 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:21 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:26 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:26 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:42 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:42 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:55 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:06:55 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:17 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:07:17 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:36 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:07:36 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:43 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:07:43 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:53 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:07:53 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:13 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:08:13 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:23 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 15:08:23 making login request to http://nopehttps://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:23 ERROR: Provider 'conjur' could not be used! ERROR: Conjur provider could not retrieve access token using the authenticator client: Error: Conjur provider unable to log in to Conjur: Post http://nopehttps//conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert: dial tcp: lookup nopehttps on 10.7.240.10:53: no such host

Bad Kubernetes authenticator auth details

  • Ensure that the authenticator authn-k8s/<id> is enabled on Conjur.

  • Ensure that the host is defined to log in through the Kubernetes authenticator.

  • Ensure that the CONJUR_AUTHN_LOGIN and CONJUR_ACCOUNT variables for Secretless match the expected values in Conjur.

 
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:49 Instantiating provider 'conjur'
2019/10/30 15:06:49 Info: Conjur provider using Kubernetes authenticator-based authentication
2019/10/30 15:06:49 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:06:49 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:50 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:06:50 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:06:54 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:06:54 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:03 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:07:03 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:19 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:07:19 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:36 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:07:36 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:07:56 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:07:56 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:05 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:05 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:16 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:16 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:25 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:25 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:45 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:45 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:56 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretlessnope.
2019/10/30 15:08:56 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 15:08:56 ERROR: Provider 'conjur' could not be used! ERROR: Conjur provider could not retrieve access token using the authenticator client: Error: Conjur provider unable to log in to Conjur:

Bad/invalid Kubernetes authenticator SSL certificate for Conjur

  • Ensure that the SSL certificate for Conjur exactly matches the certificate in CONJUR_SSL_CERTIFICATE or theCONJUR_CERT_FILE env variable.

  • If there are load balancers in front of the follower, make sure you use the balancer's SSL certificate in the Secretless configuration.

 
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 16:34:45 Instantiating provider 'conjur'
2019/10/30 16:34:45 Info: Conjur provider using Kubernetes authenticator-based authentication
2019/10/30 16:34:45 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:34:45 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:34:48 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:34:48 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:34:52 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:34:52 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:35:04 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:35:04 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:35:13 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:35:13 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:35:35 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:35:35 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:35:48 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:35:48 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:02 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:36:02 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:21 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:36:21 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:37 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:36:37 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:56 logging in as host/conjur/authn-k8s/dev/apps/sgnn7-demo/service_account/test-app-secretless.
2019/10/30 16:36:56 making login request to https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert
2019/10/30 16:36:56 ERROR: Provider 'conjur' could not be used! ERROR: Conjur provider could not retrieve access token using the authenticator client: Error: Conjur provider unable to log in to Conjur: Post https://conjur-follower.conjur-0xx9.svc.cluster.local/api/authn-k8s/dev/inject_client_cert: x509: certificate signed by unknown authority

OR

2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up...
2019/10/30 15:06:22 Initializing health check on :5335...
2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/30 15:06:22 [WARN]  Plugin hashes were not provided - tampering will not be detectable!
2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml
2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml
2019/10/30 15:06:22 Registering reload signal listeners...
2019/10/30 15:06:27 Instantiating provider 'literal'
2019/10/30 15:06:33 Instantiating provider 'conjur'
2019/10/30 15:06:33 Info: Conjur provider using Kubernetes authenticator-based authentication
2019/10/30 15:06:33 ERROR: Provider 'conjur' could not be used! ERROR: Conjur provider could not retrieve access token using the authenticator client: At least one of CONJUR_SSL_CERTIFICATE and CONJUR_CERT_FILE must be provided

Invalid Kubernetes secret ID format

Ensure that kubernetes is specified in the <secret_id>#<key> format.

 
2019/10/28 15:40:59 Secretless v1.2.0-906f9eb starting up...
2019/10/28 15:40:59 Initializing health check on :5335...
2019/10/28 15:40:59 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 15:40:59 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 15:40:59 k8s/crd: Using home dir config...
2019/10/28 15:40:59 k8s/crd: Registering CRD watcher...
2019/10/28 15:40:59 k8s/crd: Using home dir config...
2019/10/28 15:40:59 k8s/crd: Add configuration event
2019/10/28 15:40:59 secretless-example-config2
2019/10/28 15:40:59 WARN: v1 configuration is now deprecated and will be removed in a future release
2019/10/28 15:41:06 Instantiating provider 'kubernetes'
2019/10/28 15:41:06 ERROR: Resolving credential 'kube-password' from provider 'kubernetes' failed: Kubernetes secret id must contain secret name and field name in the format secretName#fieldName, received 'kube-password'

Sample client log output messages

ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'kube-password' from provider 'kubernetes' failed: Kubernetes secret id must contain secret name and field name in the format secretName#fieldName, received 'kube-password'

Kubernetes secret cannot be found

  • Ensure that the required Kubernetes secret and its identifier match the specified configuration in Secretless configuration.

  • Ensure Secretless has the correct permissions to list and read the specified secret.

  • Ensure Secretless is reading the secret from the correct namespace.

  • Connect to Kubernetes over kubectl to ensure that the secret is readable and accessible.

 
2019/10/28 15:43:49 Secretless v1.2.0-906f9eb starting up...
2019/10/28 15:43:49 Initializing health check on :5335...
2019/10/28 15:43:49 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 15:43:49 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 15:43:49 k8s/crd: Using home dir config...
2019/10/28 15:43:49 k8s/crd: Registering CRD watcher...
2019/10/28 15:43:49 k8s/crd: Using home dir config...
2019/10/28 15:43:49 k8s/crd: Add configuration event
2019/10/28 15:43:49 secretless-example-config2
2019/10/28 15:43:49 WARN: v1 configuration is now deprecated and will be removed in a future release
2019/10/28 15:43:52 Instantiating provider 'kubernetes'
2019/10/28 15:43:52 ERROR: Resolving credential 'mysql#password' from provider 'kubernetes' failed: could not find Kubernetes secret from 'mysql#password'

Sample client log output messages

ERROR: MySQL Error 
2000 (HY000): #HY000ERROR: Resolving variable 'mysql#password' from provider 'kubernetes' failed: could not find Kubernetes secret from 'mysql#password'

Kubernetes API server unreachable

  • Ensure Secretless can access the Kubernetes server.

  • Connect to Kubernetes over kubectl to ensure that the provider works.

 
2019/10/28 15:58:35 Secretless v1.2.0-906f9eb starting up...
2019/10/28 15:58:35 Initializing health check on :5335...
2019/10/28 15:58:35 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 15:58:35 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 15:58:35 Trying to load configuration file: secretless-k8s-secret.yml
2019/10/28 15:58:35 WARN: 'protocol' key found on service 'mysql-socket'. 'protocol' is now deprecated and will be removed in a future release.
2019/10/28 15:58:35 [INFO]  Waiting for new configuration...
2019/10/28 15:58:35 [DEBUG] Got new configuration
2019/10/28 15:58:35 Registering reload signal listeners...
2019/10/28 15:58:35 [INFO]  mysql-socket: Starting service
2019/10/28 15:58:35 [INFO]  Waiting for new configuration...
2019/10/28 15:58:37 Instantiating provider 'literal'
2019/10/28 15:58:37 Instantiating provider 'kubernetes'
2019/10/28 15:59:03 ERROR: Resolving credential 'mysql#password' from provider 'kubernetes' failed: Get https://192.168.99.100:8443/api/v1/namespaces/default/secrets/mysql: dial tcp 192.168.99.100:8443: connect: operation timed out
2019/10/28 15:59:03 [ERROR] mysql-socket: Failed on handle connection: failed on retrieve credentials: ERROR: Resolving credential 'mysql#password' from provider 'kubernetes' failed: Get https://192.168.99.100:8443/api/v1/namespaces/default/secrets/mysql: dial tcp 192.168.99.100:8443: connect: operation timed out

Sample client log output messages

ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'mysql#password' from provider 'kubernetes' failed: Get https://192.168.99.100:8443/api/v1/namespaces/default/secrets/mysql: dial tcp 192.168.99.100:8443: i/o timeout

Kubernetes secret does not contain the expected key

  • Ensure that the required Kubernetes secret and its identifier match the specified configuration in the Secretless configuration.

  • Connect to Kubernetes over kubectl to ensure that the secret has the appropriate key id and a value assigned.

 
2019/10/28 15:46:33 Secretless v1.2.0-906f9eb starting up...
2019/10/28 15:46:33 Initializing health check on :5335...
2019/10/28 15:46:33 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.
2019/10/28 15:46:33 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...
2019/10/28 15:46:33 k8s/crd: Using home dir config...
2019/10/28 15:46:33 k8s/crd: Registering CRD watcher...
2019/10/28 15:46:33 k8s/crd: Using home dir config...
2019/10/28 15:46:33 k8s/crd: Add configuration event
2019/10/28 15:46:33 secretless-example-config2
2019/10/28 15:46:33 WARN: v1 configuration is now deprecated and will be removed in a future release
2019/10/28 15:46:36 Instantiating provider 'kubernetes'
2019/10/28 15:46:36 ERROR: Resolving credential 'mysql#password' from provider 'kubernetes' failed: could not find field 'password' in Kubernetes secret 'mysql'

Sample client log output messages

ERROR: MySQL Error 2000 (HY000): #HY000ERROR: Resolving variable 'mysql#password' from provider 'kubernetes' failed: could not find field 'password' in Kubernetes secret 'mysql'