==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: am-b59c765b7-bgsxp Namespace: xlou Priority: 0 Service Account: default Node: gke-xlou-cdm-ds-0f601751-hmm7/10.142.0.41 Start Time: Wed, 17 Apr 2024 19:12:09 +0000 Labels: app=am app.kubernetes.io/component=am app.kubernetes.io/instance=am app.kubernetes.io/name=am app.kubernetes.io/part-of=forgerock pod-template-hash=b59c765b7 tier=middle Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true kubectl.kubernetes.io/restartedAt: 2024-04-17T19:12:09Z Status: Running IP: 10.106.45.135 IPs: IP: 10.106.45.135 Controlled By: ReplicaSet/am-b59c765b7 Init Containers: fbc-init: Container ID: containerd://c5e70a908f7eff78f170282d98c713a7df184cb595f5e14dbbdc30f061661c12 Image: gcr.io/engineeringpit/lodestar-images/am:xlou-affinityAll-noauditlog Image ID: gcr.io/engineeringpit/lodestar-images/am@sha256:0488d1f58b1bfb62dc04ff25b3bdf45580475705679ac11edf4a0407ca86f27f Port: Host Port: Command: /bin/bash -c if [ -d /fbc/config ]; then echo "Existing openam configuration found. Skipping copy" else echo "Copying docker image configuration files to the shared volume" cd /home/forgerock/openam cp -r .homeVersion * /fbc fi State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 17 Apr 2024 19:12:10 +0000 Finished: Wed, 17 Apr 2024 19:12:10 +0000 Ready: True Restart Count: 0 Environment: Mounts: /fbc from fbc (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jqcsj (ro) truststore-init: Container ID: containerd://84331c2d85a309180b50b7c1fdb53279d9b6863b12b0eb146d34f00744eb030d Image: gcr.io/engineeringpit/lodestar-images/am:xlou-affinityAll-noauditlog Image ID: gcr.io/engineeringpit/lodestar-images/am@sha256:0488d1f58b1bfb62dc04ff25b3bdf45580475705679ac11edf4a0407ca86f27f Port: Host Port: Command: /home/forgerock/import-pem-certs.sh State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 17 Apr 2024 19:12:11 +0000 Finished: Wed, 17 Apr 2024 19:12:11 +0000 Ready: True Restart Count: 0 Environment: TRUSTSTORE_PATH: /truststore/amtruststore TRUSTSTORE_PASSWORD: changeit AM_PEM_TRUSTSTORE: /var/run/secrets/truststore/cacerts AM_PEM_TRUSTSTORE_DS: /var/run/secrets/truststore/ca.crt Mounts: /truststore from new-truststore (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jqcsj (ro) /var/run/secrets/truststore from truststore (rw) Containers: openam: Container ID: containerd://667d1c8ed249bca02317a8833529cc3efe4b771ffc441b23aa4a925e11edaffd Image: gcr.io/engineeringpit/lodestar-images/am:xlou-affinityAll-noauditlog Image ID: gcr.io/engineeringpit/lodestar-images/am@sha256:0488d1f58b1bfb62dc04ff25b3bdf45580475705679ac11edf4a0407ca86f27f Port: 8080/TCP Host Port: 0/TCP Command: bash /home/forgerock/docker-entrypoint.sh State: Running Started: Wed, 17 Apr 2024 19:12:12 +0000 Ready: True Restart Count: 0 Limits: memory: 10Gi Requests: cpu: 11 memory: 10Gi Liveness: http-get http://:8080/am/json/health/live delay=30s timeout=5s period=30s #success=1 #failure=3 Readiness: http-get http://:8080/am/json/health/ready delay=20s timeout=5s period=10s #success=1 #failure=3 Startup: http-get http://:8080/am/json/health/live delay=0s timeout=1s period=10s #success=1 #failure=40 Environment Variables from: am-env-secrets Secret Optional: false platform-config ConfigMap Optional: false amster-env-secrets Secret Optional: false ds-env-secrets Secret Optional: false Environment: NAMESPACE: xlou (v1:metadata.namespace) AM_STORES_SSL_ENABLED: true TRUSTSTORE_PATH: /home/forgerock/amtruststore TRUSTSTORE_PASSWORD: changeit AM_STORES_USER_TYPE: LDAPv3ForForgeRockIAM Mounts: /home/forgerock/amtruststore from new-truststore (ro,path="amtruststore") /home/forgerock/logging from am-logback (rw) /home/forgerock/openam from fbc (rw) /var/run/secrets/am from am-secrets (rw) /var/run/secrets/amster from amster-key (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jqcsj (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: fbc: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: am-secrets: Type: Projected (a volume that contains injected data from multiple sources) SecretName: am-passwords SecretOptionalName: SecretName: am-keystore SecretOptionalName: amster-key: Type: Secret (a volume populated by a Secret) SecretName: amster Optional: false truststore: Type: Projected (a volume that contains injected data from multiple sources) SecretName: truststore-pem SecretOptionalName: SecretName: ds-ssl-keypair SecretOptionalName: new-truststore: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: am-logback: Type: ConfigMap (a volume populated by a ConfigMap) Name: am-logback Optional: false kube-api-access-jqcsj: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kubernetes.io/arch:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 49m default-scheduler Successfully assigned xlou/am-b59c765b7-bgsxp to gke-xlou-cdm-ds-0f601751-hmm7 Normal Pulling 49m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/am:xlou-affinityAll-noauditlog" Normal Pulled 49m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/am:xlou-affinityAll-noauditlog" in 992.574586ms (992.58086ms including waiting) Normal Created 49m kubelet Created container fbc-init Normal Started 49m kubelet Started container fbc-init Normal Pulling 49m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/am:xlou-affinityAll-noauditlog" Normal Pulled 49m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/am:xlou-affinityAll-noauditlog" in 204.797963ms (204.804739ms including waiting) Normal Created 49m kubelet Created container truststore-init Normal Started 49m kubelet Started container truststore-init Normal Pulling 49m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/am:xlou-affinityAll-noauditlog" Normal Pulled 49m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/am:xlou-affinityAll-noauditlog" in 231.452469ms (231.458946ms including waiting) Normal Created 49m kubelet Created container openam Normal Started 49m kubelet Started container openam Warning Unhealthy 48m (x2 over 49m) kubelet Startup probe failed: Get "http://10.106.45.135:8080/am/json/health/live": context deadline exceeded (Client.Timeout exceeded while awaiting headers) ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 10.106.45.129 - - [17/Apr/2024:19:56:59 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:19:57:09 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:19:57:09 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.41.131 - - [17/Apr/2024:19:57:14 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 361965 29ms 10.106.45.129 - - [17/Apr/2024:19:57:19 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:19:57:29 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:19:57:39 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:19:57:39 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.41.131 - - [17/Apr/2024:19:57:44 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 362008 29ms {"timestamp":"2024-04-17T19:57:53.125Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(923)","mdc":{"transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default","transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"} {"timestamp":"2024-04-17T19:57:54.125Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(892)","mdc":{"transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default","transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"} 10.106.45.129 - - [17/Apr/2024:19:57:49 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.44.133 - - [17/Apr/2024:19:57:52 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.44.133 - - [17/Apr/2024:19:57:53 +0000] "POST /am/json/authenticate?realm=/ HTTP/1.1" 200 167 7ms 10.106.44.133 - - [17/Apr/2024:19:57:53 +0000] "POST /am/json/realms/root/sessions?_action=getSessionInfo HTTP/1.1" 200 294 3ms 10.106.44.133 - - [17/Apr/2024:19:57:53 +0000] "GET /am/json/serverinfo/version HTTP/1.1" 200 276 3ms {"timestamp":"2024-04-17T19:57:55.125Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(847)","mdc":{"transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default","transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"} 10.106.45.129 - - [17/Apr/2024:19:57:59 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:19:58:09 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:19:58:09 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.41.131 - - [17/Apr/2024:19:58:14 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 361981 29ms 10.106.45.129 - - [17/Apr/2024:19:58:19 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms {"timestamp":"2024-04-17T19:58:26.031Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(919)","mdc":{"transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default","transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"} {"timestamp":"2024-04-17T19:58:27.125Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(1022)","mdc":{"transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default","transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"} 10.106.45.129 - - [17/Apr/2024:19:58:29 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms {"timestamp":"2024-04-17T19:58:43.155Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(971)","mdc":{"transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default","transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"} 10.106.45.129 - - [17/Apr/2024:19:58:39 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:19:58:39 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.41.131 - - [17/Apr/2024:19:58:44 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 362030 29ms {"timestamp":"2024-04-17T19:58:45.125Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(1032)","mdc":{"transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default","transactionId":"aa34e6bd-4080-49f8-9130-b3cc3f0b2ea1-0"} 10.106.45.129 - - [17/Apr/2024:19:58:49 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:19:58:59 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:19:59:09 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:19:59:09 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.41.131 - - [17/Apr/2024:19:59:14 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 361979 26ms 10.106.45.129 - - [17/Apr/2024:19:59:19 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:19:59:29 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:19:59:39 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:19:59:39 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.41.131 - - [17/Apr/2024:19:59:44 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 361973 26ms 10.106.45.129 - - [17/Apr/2024:19:59:49 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:19:59:59 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:20:00:09 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:20:00:09 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.41.131 - - [17/Apr/2024:20:00:14 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 361937 25ms 10.106.45.129 - - [17/Apr/2024:20:00:19 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:20:00:29 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:20:00:39 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:20:00:39 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.41.131 - - [17/Apr/2024:20:00:44 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 361943 26ms 10.106.45.129 - - [17/Apr/2024:20:00:49 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:20:00:59 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.45.129 - - [17/Apr/2024:20:01:09 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.45.129 - - [17/Apr/2024:20:01:09 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.41.131 - - [17/Apr/2024:20:01:14 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 361955 26ms 10.106.45.129 - - [17/Apr/2024:20:01:19 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms Copying docker image configuration files to the shared volume Copying /opt/java/openjdk/lib/security/cacerts to /truststore/amtruststore Found (2) certificates in am_combined_truststore Importing (2) certificates into /truststore/amtruststore Certificate was added to keystore Certificate was added to keystore Import complete!