Troubleshoot Secretless Service Connectors
This topic describes common troubleshooting issues for service connectors.
Error |
Suggested resolution |
Log output |
---|---|---|
Server does not support TLS |
|
2020/04/21 17:34:36 [INFO] mssql: Starting service 2020/04/21 17:34:36 Registering reload signal listeners... 2020/04/21 17:34:45 Instantiating provider 'literal' 2020/04/21 17:34:45 [ERROR] mssql: Failed on handle connection: failed on connect: TLS Handshake failed: x509: certificate is valid for <HostName>, not <Given HostName> Received a stop signal |
Server requires TLS encryption |
Ensure that |
[00] 2020/04/21 17:43:37 [INFO] mssql: Starting service [00] 2020/04/21 17:43:37 [DEBUG] Waiting for new configuration... [00] 2020/04/21 17:43:43 Instantiating provider 'literal' [00] 2020/04/21 17:43:43 [DEBUG] mssql: New connection on X.X.X.X:2223. [00] 2020/04/21 17:43:58 [ERROR] mssql: Failed on handle connection: failed on connect: Unable to open tcp connection with host 'mssql:1433': dial tcp Y.Y.Y.Y:1433: i/o timeout |
Service connector is not found or is invalid |
|
22019/10/25 19:42:48 Secretless v1.2.0-906f9eb starting up... 2019/10/25 19:42:48 Initializing health check on :5335... 2019/10/25 19:42:48 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/25 19:42:48 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/25 19:42:48 [INFO] Adding 'my_service_connector.so' as a plugin... 2019/10/25 19:42:48 [INFO] Loading plugin 'my_service_connector'... 2019/10/25 19:42:48 [ERROR] my_service_connector: PluginInfo['type'] of 'not_connector.not_supported' is not supported 2019/10/25 20:01:17 [ERROR] my_service_connector: plugin 'my_service_connector' (API v0.1.2) is not a supported API version (v0.1.0) 2019/10/25 19:42:48 Trying to load configuration file: ./secretless.yml 2019/10/25 19:42:48 [INFO] Waiting for new configuration... 2019/10/25 19:42:48 [DEBUG] Got new configuration 2019/10/25 19:42:48 Registering reload signal listeners... 2019/10/25 19:42:48 [INFO] Validating services against available plugins: ssh,ssh-agent,pg,mysql,aws,basic_auth,conjur 2019/10/25 19:42:48 Failed to start services: services validation failed: my_service_name: missing service connector "my_service_connector". 2019/10/25 19:42:48 Registering reload signal listeners... |
Socket/port in use |
Ensure that the specified socket file or port is not in use by another program.
|
2019/10/25 14:01:16 Secretless v1.2.0-906f9eb starting up... 2019/10/25 14:01:16 Initializing health check on :5335... 2019/10/25 14:01:16 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/25 14:01:16 [WARN] Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins... 2019/10/25 14:01:16 Trying to load configuration file: /secretless.yml 2019/10/25 14:01:16 Registering reload signal listeners... 2019/10/25 14:01:16 [PANIC] unable to create TCP service 'backend_staging': listen tcp 0.0.0.0:2222: bind: address already in use |
Unable to open socket/port |
|
2019/10/25 13:04:09 Secretless v1.2.0-906f9eb starting up... 2019/10/25 13:04:09 Initializing health check on :5335... 2019/10/25 13:04:09 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/25 13:04:09 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/25 13:04:09 Trying to load configuration file: /secretless.yml 2019/10/25 13:04:09 [PANIC] unable to create TCP service 'backend_production': listen tcp 0.0.0.0:1: bind: permission denied panic: [PANIC] unable to create TCP service 'backend_production': listen tcp 0.0.0.0:1: bind: permission denied |
Unreachable PostgreSQL backend server |
|
2019/10/25 14:08:17 Secretless v1.2.0-906f9eb starting up... 2019/10/25 14:08:17 Initializing health check on :5335... 2019/10/25 14:08:17 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/25 14:08:17 [WARN] Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins... 2019/10/25 14:08:17 Trying to load configuration file: /secretless.yml 2019/10/25 14:08:17 Registering reload signal listeners... 2019/10/25 14:08:21 Instantiating provider 'literal' 2019/10/25 14:08:21 [ERROR] backend_production: Failed on handle connection: failed on connect: dial tcp 127.0.0.1:1234: connect: connection refused Sample PostgreSQL client log output messages: psql: FATAL: dial tcp 127.0.0.1:5436: connect: connection refused psql: FATAL: dial tcp: missing address |
Incompatible PostgreSQL client settings |
PostgreSQL connections to Secretless from the client must have sslmode set to disabled. For example: psql "host=pg-host port=5555 sslmode=disable dbname=postgres"). |
2019/10/30 11:53:19 Secretless v1.2.0-906f9eb starting up... 2019/10/30 11:53:19 Initializing health check on :5335... 2019/10/30 11:53:19 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/30 11:53:19 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/30 11:53:19 Trying to load configuration file: /secretless.yml 2019/10/30 11:53:19 Attaching filesystem notifier onto /secretless.yml 2019/10/30 11:53:19 Registering reload signal listeners... 2019/10/30 11:53:34 Instantiating provider 'literal' 2019/10/30 11:53:34 [ERROR] service_name: Failed on handle connection: failed on connect: SSL not supported Sample client log output messages: psql: FATAL: SSL not supported |
Incompatible backend PostgreSQL settings |
|
2019/10/30 11:53:19 Secretless v1.2.0-906f9eb starting up... 2019/10/30 11:53:19 Initializing health check on :5335... 2019/10/30 11:53:19 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/30 11:53:19 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/30 11:53:19 Trying to load configuration file: /secretless.yml 2019/10/30 11:53:19 Attaching filesystem notifier onto /secretless.yml 2019/10/30 11:53:19 Registering reload signal listeners... 2019/10/30 11:53:34 Instantiating provider 'literal' 2019/10/30 11:53:34 [ERROR] service_name: Failed on handle connection: failed on connect: the backend does not allow SSL connections Sample client log output messages: psql: FATAL: the backend does not allow SSL connections |
Bad PostgreSQL authentication details |
|
2019/10/30 11:53:19 Secretless v1.2.0-906f9eb starting up... 2019/10/30 11:53:19 Initializing health check on :5335... 2019/10/30 11:53:19 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/30 11:53:19 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/30 11:53:19 Trying to load configuration file: /secretless.yml 2019/10/30 11:53:19 Attaching filesystem notifier onto /secretless.yml 2019/10/30 11:53:19 Registering reload signal listeners... 2019/10/30 11:53:34 Instantiating provider 'literal' 2019/10/30 11:53:34 [ERROR] service_name: Failed on handle connection: failed on connect: pg: FATAL: password authentication failed for user "username" Sample client log output messages psql: FATAL: password authentication failed for user "username" |
Unreachable MySQL backend server |
Make sure that the host and port fields specified in Secretless configuration point to a reachable server. Make sure that Secretless outgoing connections are not blocked. |
2019/10/25 14:08:17 Secretless v1.2.0-906f9eb starting up... 2019/10/25 14:08:17 Initializing health check on :5335... 2019/10/25 14:08:17 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/25 14:08:17 [WARN] Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins... 2019/10/25 14:08:17 Trying to load configuration file: /secretless.yml 2019/10/25 14:08:17 Registering reload signal listeners... 2019/10/25 14:08:21 Instantiating provider 'literal' 2019/10/25 14:08:21 [ERROR] backend_production: Failed on handle connection: failed on connect: dial tcp 127.0.0.1:1234: connect: connection refused Sample client log output messages ERROR: MySQL Error 2000 (HY000): #HY000dial tcp: lookup localhosts: no such host ERROR: MySQL Error 2000 (HY000): #HY000dial tcp 127.0.0.1:5433: connect: connection refused |
Bad/incompatible MySQL client connection settings |
MySQL connections to Secretless from the client must have the ssl-mode set to disabled. For example:
|
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up... 2019/10/30 15:06:22 Initializing health check on :5335... 2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/30 15:06:22 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml 2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml 2019/10/30 15:06:22 Registering reload signal listeners... 2019/10/30 15:06:27 Instantiating provider 'literal' 2019/10/30 15:06:27 [ERROR] pg-tcp: Failed on handle connection: failed on connect: ERROR: 2026 (HY000): SSL connection error: SSL is required but the server doesn't support it Sample client log output messages ERROR: MySQL Error 2026 (HY000): SSL connection error: SSL is required but the server doesn't support it |
Incompatible backend MySQL settings |
|
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up... 2019/10/30 15:06:22 Initializing health check on :5335... 2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/30 15:06:22 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml 2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml 2019/10/30 15:06:22 Registering reload signal listeners... 2019/10/30 15:06:27 Instantiating provider 'literal' 2019/10/30 15:06:27 [ERROR] pg-tls: Failed on handle connection: failed on connect: dial tcp X.X.X.X:XXXX: connect: connection refused Sample client log output messages ERROR 2000 (HY000): #HY000dial tcp X.X.X.X:XXXX: connect: connection refused OR ERROR 2013 (HY000): Lost connection to MySQL server at 'waiting for initial communication packet', system error: 110 "Connection timed out" OR ERROR: MySQL Error 2000 (HY000): x509: certificate signed by unknown authority |
Bad MySQL authentication details |
|
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up... 2019/10/30 15:06:22 Initializing health check on :5335... 2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/30 15:06:22 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml 2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml 2019/10/30 15:06:22 Registering reload signal listeners... 2019/10/30 15:06:27 Instantiating provider 'literal' 2019/10/30 15:06:27 [ERROR] serivce_name: Failed on handle connection: failed on connect: ERROR: 1045 (28000): Access denied for user 'username'@'XX.XX.XX.XX' (using password: YES) Sample client log output messages ERROR 1045 (28000): Access denied for user 'username'@'XX.XX.XX.XX' (using password: YES) |
Unsupported MySQL version |
Make sure that your MySQL backend is supported by Secretless. |
2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up... 2019/10/30 15:06:22 Initializing health check on :5335... 2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/30 15:06:22 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml 2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml 2019/10/30 15:06:22 Registering reload signal listeners... 2019/10/30 15:06:27 Instantiating provider 'literal' 2019/10/30 15:06:27 [ERROR] serivce_name: Failed on handle connection: failed on connect: ERROR: 1045 (28000): Access denied for user 'username'@'XX.XX.XX.XX' (using password: YES) Sample client log output messages ERROR: MySQL Error 2000 (HY000): #HY000EOF |
Missing or Invalid Port for PostgreSQL |
Ensure that the address provided in the Secretless configuration is of the form |
|
Missing or Invalid Port for MySQL |
Ensure the |
Invalid port provided in configuration (e.g., host set to 2019/10/30 15:06:22 Secretless v1.2.0-906f9eb starting up... 2019/10/30 15:06:22 Initializing health check on :5335... 2019/10/30 15:06:22 Initialization of health check done. You can access the endpoint at `/live` and `/ready`. 2019/10/30 15:06:22 [WARN] Plugin hashes were not provided - tampering will not be detectable! 2019/10/30 15:06:22 Trying to load configuration file: /secretless.yml 2019/10/30 15:06:22 Attaching filesystem notifier onto /secretless.yml 2019/10/30 15:06:22 Registering reload signal listeners... 2019/10/30 15:06:27 Instantiating provider 'literal' 2019/10/30 15:06:27 [ERROR] service_name: Failed on handle connection: failed on connect: dial tcp HOST:INVALID_PORT: connect: connection refused Port credential provided in configuration, but set to empty value (e.g., host set to $ mysqlsh -h localhost -P 3306 -D my_db Creating a Session to 'myuser@localhost:3306/my_db' Enter password: ERROR: 2000 (HY000): #HY000dial tcp HOST:0: connect: can't assign requested address Port credential provided in configuration, but set to invalid value (e.g., host set to $ mysql -h localhost -P 3306 -D my_db Creating a Session to 'myuser@localhost:3306/my_db' Enter password: ERROR: 2000 (HY000): #HY000dial tcp HOST:INVALID_PORT: connect: operation timed out |
Bad/incompatible MSSQL client settings |
Verify that the MSSQL connection to Secretless from the client have the ssl-mode set to disabled. For example, do not use -N when using SQLCMD. |
[00] 2020/01/13 19:31:44 [WARN] Starting TCP listener on 0.0.0.0:2223... [00] 2020/01/13 19:31:44 [INFO] mssql: Starting service [00] 2020/01/13 19:31:44 [INFO] Waiting for new configuration... [00] 2020/01/13 19:44:11 Instantiating provider 'literal' [00] 2020/01/13 19:44:11 [INFO] mssql: New connection on x.x.x.x:2223. [00] 2020/01/13 19:44:11 [ERROR] mssql: Failed on handle connection: failed on connect: Unable to open tcp connection with host 'mssql:0': dial tcp x.x.x.x:0: connect: connection refused |
Bad MSSQL server settings |
Ensure that the database connection details for MSSQL in Secretless configuration match the connection details for your database. Attempt to connect directly to your database using the credentials to verify that the backend is working properly. |
[00] 2020/01/13 21:19:34 Secretless v1.4.2-dev starting up... ... [00] 2020/01/13 21:19:34 [WARN] Starting TCP listener on 0.0.0.0:2223... [00] 2020/01/13 21:19:34 [INFO] mssql: Starting service [00] 2020/01/13 21:19:34 [INFO] Waiting for new configuration... [00] 2020/01/13 21:19:34 Attaching filesystem notifier onto /secretless.yml [00] 2020/01/13 21:19:34 Registering reload signal listeners... [00] 2020/01/13 21:19:50 Instantiating provider 'literal' [00] 2020/01/13 21:19:50 [INFO] mssql: New connection on 192.168.16.3:2223. [00] 2020/01/13 21:19:50 [ERROR] mssql: Failed on handle connection: failed on connect: lookup <host> on x.x.x.x:53: no such host |
Bad MSSQL authentication details |
Verify that authentication details provided in the database connection details are valid. Attempt to connect directly to the database using those connection details to ensure that they are correct. |
[00] 2020/01/13 20:58:13 Secretless v1.4.2-dev starting up... ... [00] 2020/01/13 20:58:13 [WARN] Starting TCP listener on 0.0.0.0:2223... [00] 2020/01/13 20:58:13 [INFO] mssql: Starting service [00] 2020/01/13 20:58:13 [INFO] Waiting for new configuration... [00] 2020/01/13 20:58:13 Attaching filesystem notifier onto /secretless.yml [00] 2020/01/13 20:58:13 Registering reload signal listeners... [00] 2020/01/13 20:59:05 Instantiating provider 'literal' [00] 2020/01/13 20:59:05 [INFO] mssql: New connection on 172.27.0.3:2223. [00] 2020/01/13 20:59:05 [ERROR] mssql: Failed on handle connection: failed on connect: Login error: mssql: Login failed for user 'bad'. |
Invalid port for MSSQL |
Ensure that the port provided in the Secretless database connection details is valid for the MSSQL server |
[00] 2020/01/13 20:19:13 Secretless v1.4.2-dev starting up... ... [00] 2020/01/13 20:19:13 Registering reload signal listeners... [00] 2020/01/13 20:19:13 [WARN] Starting TCP listener on 0.0.0.0:2223... [00] 2020/01/13 20:19:13 [INFO] mssql: Starting service [00] 2020/01/13 20:19:13 [INFO] Waiting for new configuration... [00] 2020/01/13 20:21:13 Instantiating provider 'literal' [00] 2020/01/13 20:21:13 [INFO] mssql: New connection on x.x.x.x:2223. [00] 2020/01/13 20:21:13 [ERROR] mssql: Failed on handle connection: failed on connect: Unable to open tcp connection with host 'mssql:<invalid port>': dial tcp x.x.x.x:<invalid port>: connect: connection refused |
Your client receives a 405 Method Not Allowed response when attempting to connect to the target service using Secretless as an HTTPS proxy |
|
. curl: ``` * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 62160 (#0) * Establish HTTP proxy tunnel to httpbin.org:443 > CONNECT httpbin.org:443 HTTP/1.1 > Host: httpbin.org:443 > User-Agent: curl/7.54.0 > Proxy-Connection: Keep-Alive > < HTTP/1.1 405 Method Not Allowed < Content-Type: text/plain; charset=utf-8 < X-Content-Type-Options: nosniff < Date: Wed, 06 May 2020 17:39:19 GMT < Content-Length: 26 < * Received HTTP code 405 from proxy after CONNECT * Closing connection 0 curl: (56) Received HTTP code 405 from proxy after CONNECT ``` |
HTTPS certificate verification fails when |
This type of error can be broken into 2 categories:
|
curl: ``` * Rebuilt URL to: http://self-signed.badssl.com/ * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 62165 (#0) > GET http://self-signed.badssl.com/ HTTP/1.1 > Host: self-signed.badssl.com > User-Agent: curl/7.54.0 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 503 Service Unavailable < Content-Type: text/plain; charset=utf-8 < X-Content-Type-Options: nosniff < Date: Wed, 06 May 2020 17:39:20 GMT < Content-Length: 46 < { [46 bytes data] * Connection #0 to host 127.0.0.1 left intact x509: certificate signed by unknown authority |