==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: rcs-6f9fc5b96c-k4sxf Namespace: jnkns-pt-dbg-tsts-dbg-thr-tsts-463-rcs Priority: 0 Node: gke-perf-daily-default-pool-5cf934b0-4w1r/10.128.0.108 Start Time: Sun, 19 Mar 2023 21:42:11 +0000 Labels: app=rcs pod-template-hash=6f9fc5b96c skaffold.dev/run-id=941f59d7-16ec-44ff-8eb5-11610d0b9a19 Annotations: Status: Running IP: 10.108.13.6 IPs: IP: 10.108.13.6 Controlled By: ReplicaSet/rcs-6f9fc5b96c Containers: rcs: Container ID: containerd://6089ec4b796ce9efdafc1e2d6d3abd6bab49c9864d87adbef1217b53c5372923 Image: gcr.io/engineeringpit/lodestar-images/rcs:jnkns-pt-dbg-tsts-dbg-thr-tsts-463-rcs Image ID: gcr.io/engineeringpit/lodestar-images/rcs@sha256:038aefac3f986b7309ef0faadc2622e8a4937fb277d48fca5b178d689d708e41 Port: 8759/TCP Host Port: 0/TCP State: Running Started: Sun, 19 Mar 2023 23:07:42 +0000 Last State: Terminated Reason: Error Exit Code: 143 Started: Sun, 19 Mar 2023 23:07:14 +0000 Finished: Sun, 19 Mar 2023 23:07:42 +0000 Ready: True Restart Count: 15 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 500Mi Liveness: exec [/opt/openicf/rcs-probe.sh] delay=10s timeout=1s period=30s #success=1 #failure=1 Environment: CDM_HOST: jnkns-pt-dbg-tsts-dbg-thr-tsts-463.iam.perf-daily.engineeringpit.com RCS_LOG: /opt/openicf/logs/ConnectorServer.log RCS_LOG_ERR_PATTERN: inconsistent|Exception|No WebSocket connections Mounts: /opt/openicf/conf from config-vol (rw) /opt/openicf/lib/framework/logback.xml from config-logging-vol (rw,path="logback.xml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-56tz5 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config-vol: Type: ConfigMap (a volume populated by a ConfigMap) Name: rcsprops Optional: false config-logging-vol: Type: ConfigMap (a volume populated by a ConfigMap) Name: rcsprops Optional: false kube-api-access-56tz5: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 44m (x76 over 62m) kubelet Back-off restarting failed container Warning Unhealthy 39m (x14 over 64m) kubelet Liveness probe failed: URL is https://jnkns-pt-dbg-tsts-dbg-thr-tsts-463.iam.perf-daily.engineeringpit.com/openicf HTTP Return code is 405 Gateway not ready... Normal Pulling 34m (x15 over 119m) kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/rcs:jnkns-pt-dbg-tsts-dbg-thr-tsts-463-rcs" ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== VM settings: Max. Heap Size (Estimated): 1.60G Using VM: OpenJDK 64-Bit Server VM 23:07:44,984 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml] 23:07:44,984 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy] 23:07:44,985 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/opt/openicf/lib/framework/logback.xml] 23:07:45,069 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 23:07:45,071 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.FileAppender] 23:07:45,075 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SERVER-FILE] 23:07:45,078 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@13:83 - no applicable action for [rollingPolicy], current ElementPath is [[configuration][appender][rollingPolicy]] 23:07:45,078 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@14:30 - no applicable action for [fileNamePattern], current ElementPath is [[configuration][appender][rollingPolicy][fileNamePattern]] 23:07:45,079 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 23:07:45,102 |-INFO in ch.qos.logback.core.FileAppender[SERVER-FILE] - File property is set to [logs/ConnectorServer.log] 23:07:45,103 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.FileAppender] 23:07:45,103 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [TRACE-FILE] 23:07:45,103 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@23:83 - no applicable action for [rollingPolicy], current ElementPath is [[configuration][appender][rollingPolicy]] 23:07:45,104 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@24:30 - no applicable action for [fileNamePattern], current ElementPath is [[configuration][appender][rollingPolicy][fileNamePattern]] 23:07:45,104 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 23:07:45,104 |-INFO in ch.qos.logback.core.FileAppender[TRACE-FILE] - File property is set to [logs/ConnectorServerTrace.log] 23:07:45,104 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.FileAppender] 23:07:45,104 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [CONNECTOR-FILE] 23:07:45,104 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@33:83 - no applicable action for [rollingPolicy], current ElementPath is [[configuration][appender][rollingPolicy]] 23:07:45,105 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@34:30 - no applicable action for [fileNamePattern], current ElementPath is [[configuration][appender][rollingPolicy][fileNamePattern]] 23:07:45,105 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 23:07:45,105 |-INFO in ch.qos.logback.core.FileAppender[CONNECTOR-FILE] - File property is set to [logs/Connector.log] 23:07:45,105 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender] 23:07:45,106 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT] 23:07:45,107 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead. 23:07:45,107 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder. 23:07:45,107 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details 23:07:45,107 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.forgerock.openicf.misc] to INFO 23:07:45,107 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.forgerock.openicf.misc] to false 23:07:45,107 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [CONNECTOR-FILE] to Logger[org.forgerock.openicf.misc] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.forgerock.openicf.connectors] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.forgerock.openicf.connectors] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [CONNECTOR-FILE] to Logger[org.forgerock.openicf.connectors] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.identityconnectors] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.identityconnectors] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [CONNECTOR-FILE] to Logger[org.identityconnectors] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.forgerock.openicf.framework] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.forgerock.openicf.framework] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.forgerock.openicf.framework] 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SERVER-FILE] to Logger[org.forgerock.openicf.framework] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.forgerock.openicf.common] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.forgerock.openicf.common] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.forgerock.openicf.common] 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SERVER-FILE] to Logger[org.forgerock.openicf.common] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.forgerock.openicf.contract] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.forgerock.openicf.contract] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.forgerock.openicf.contract] 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SERVER-FILE] to Logger[org.forgerock.openicf.contract] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.forgerock.openicf.test] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.forgerock.openicf.test] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.forgerock.openicf.test] 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SERVER-FILE] to Logger[org.forgerock.openicf.test] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.identityconnectors.common] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.identityconnectors.common] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.identityconnectors.common] 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SERVER-FILE] to Logger[org.identityconnectors.common] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.identityconnectors.framework] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.identityconnectors.framework] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.identityconnectors.framework] 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SERVER-FILE] to Logger[org.identityconnectors.framework] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.identityconnectors.contract] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.identityconnectors.contract] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.identityconnectors.contract] 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SERVER-FILE] to Logger[org.identityconnectors.contract] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.identityconnectors.test] to INFO 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.identityconnectors.test] to false 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.identityconnectors.test] 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SERVER-FILE] to Logger[org.identityconnectors.test] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO 23:07:45,108 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT] 23:07:45,108 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 23:07:45,109 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@3fc9dfc5 - Registering current configuration as safe fallback point Mar 19, 2023 11:07:45 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.ssh-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.forgerock.openicf.connectors.ssh.SSHConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/ssh-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:45 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.groovy-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.forgerock.openicf.connectors.groovy.ScriptedPoolableConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/groovy-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:45 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.groovy-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.forgerock.openicf.connectors.groovy.ScriptedConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/groovy-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:45 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.databasetable-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.identityconnectors.databasetable.DatabaseTableConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/databasetable-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:45 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.scriptedsql-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.forgerock.openicf.connectors.scriptedsql.ScriptedSQLConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/scriptedsql-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:45 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.kerberos-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.forgerock.openicf.connectors.kerberos.KerberosConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/kerberos-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:45 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.scim-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.forgerock.openicf.connectors.scim.ScimConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/scim-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:45 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.ldap-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.identityconnectors.ldap.LdapConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/ldap-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:46 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.scriptedrest-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.forgerock.openicf.connectors.scriptedrest.ScriptedRESTConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/scriptedrest-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:46 PM INFO o.i.f.i.a.l.LocalConnectorInfoManagerImpl: Add ConnectorInfo ConnectorKey( bundleName=org.forgerock.openicf.connectors.csvfile-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.forgerock.openicf.csvfile.CSVFileConnector ) to Local Connector Info Manager from file:/opt/openicf/connectors/csvfile-connector-1.5.20.15-SNAPSHOT.jar Mar 19, 2023 11:07:46 PM INFO o.f.openicf.framework.server.Main: ConnectorAgent connecting to : wss://jnkns-pt-dbg-tsts-dbg-thr-tsts-463.iam.perf-daily.engineeringpit.com:443/openicf/0# Mar 19, 2023 11:07:46 PM INFO o.f.openicf.framework.server.Main: ConnectorAgent connecting to : wss://jnkns-pt-dbg-tsts-dbg-thr-tsts-463.iam.perf-daily.engineeringpit.com:443/openicf/1# RCS 1.5.20.15-SNAPSHOT started. Mar 19, 2023 11:08:21 PM INFO o.f.o.framework.ConnectorFramework: New local connector facade created for LDAP Connector Mar 19, 2023 11:08:21 PM INFO o.f.o.framework.ConnectorFramework: Connector Pool cleaner created for LDAP Connector Mar 19, 2023 11:08:21 PM INFO o.i.f.i.a.l.ConnectorPoolManager: Creating new pool: ConnectorKey( bundleName=org.forgerock.openicf.connectors.ldap-connector bundleVersion=1.5.20.15-SNAPSHOT connectorName=org.identityconnectors.ldap.LdapConnector ) Mar 19, 2023 11:16:04 PM INFO o.f.o.framework.ConnectorFramework: New local connector facade created for LDAP Connector Mar 19, 2023 11:16:04 PM INFO o.f.o.framework.ConnectorFramework: Connector Pool cleaner created for LDAP Connector Mar 19, 2023 11:22:46 PM INFO o.f.o.f.client.ConnectionManager: [connectorserver] GroupChecker executing for ConnectionManager: 1,715,680,292 Mar 19, 2023 11:22:46 PM INFO o.f.o.f.client.ConnectionManager: [connectorserver] Check ConnectionManager: 1,715,680,292, Session: 8E8CFD69B80742542612FD91CDB6FC4F7150D7B6, ConnectionGroup:connectorserver:8E8CFD69B80742542612FD91CDB6FC4F7150D7B6 - operational=true Mar 19, 2023 11:22:46 PM INFO o.f.o.f.client.ConnectionManager: [connectorserver] Check ConnectionManager: 1,715,680,292, Session: 3CF32154D865372ADB457F6FBEE53266C6E0C0E4, ConnectionGroup:connectorserver:3CF32154D865372ADB457F6FBEE53266C6E0C0E4 - operational=true Mar 19, 2023 11:37:46 PM INFO o.f.o.f.client.ConnectionManager: [connectorserver] GroupChecker executing for ConnectionManager: 1,715,680,292 Mar 19, 2023 11:37:46 PM INFO o.f.o.f.client.ConnectionManager: [connectorserver] Check ConnectionManager: 1,715,680,292, Session: 8E8CFD69B80742542612FD91CDB6FC4F7150D7B6, ConnectionGroup:connectorserver:8E8CFD69B80742542612FD91CDB6FC4F7150D7B6 - operational=true Mar 19, 2023 11:37:46 PM INFO o.f.o.f.client.ConnectionManager: [connectorserver] Check ConnectionManager: 1,715,680,292, Session: 3CF32154D865372ADB457F6FBEE53266C6E0C0E4, ConnectionGroup:connectorserver:3CF32154D865372ADB457F6FBEE53266C6E0C0E4 - operational=true