==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: am-685d4f4864-7q8cw Namespace: xlou Priority: 0 Node: gke-xlou-cdm-default-pool-4f098948-tbmx/10.142.0.69 Start Time: Fri, 07 Apr 2023 22:21:11 +0000 Labels: app=am app.kubernetes.io/component=am app.kubernetes.io/instance=am app.kubernetes.io/name=am app.kubernetes.io/part-of=forgerock pod-template-hash=685d4f4864 tier=middle Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true Status: Running IP: 10.106.43.17 IPs: IP: 10.106.43.17 Controlled By: ReplicaSet/am-685d4f4864 Init Containers: fbc-init: Container ID: containerd://f36c18d65e53d4b8e1cccbf6e140f26ee322da102bb3b2ef3bfb42baec6af490 Image: gcr.io/engineeringpit/lodestar-images/am:xlou Image ID: gcr.io/engineeringpit/lodestar-images/am@sha256:fae86102dd8374e78ac5d98db56845fe98742e7aebebd5445a81ba84bfa5eed5 Port: Host Port: Command: /bin/bash -c if [ -d /fbc/config ]; then echo "Existing openam configuration found. Skipping copy" else echo "Copying docker image configuration files to the shared volume" cd /home/forgerock/openam cp -r .homeVersion * /fbc fi State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 07 Apr 2023 22:21:14 +0000 Finished: Fri, 07 Apr 2023 22:21:14 +0000 Ready: True Restart Count: 0 Environment: Mounts: /fbc from fbc (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-27bn5 (ro) truststore-init: Container ID: containerd://b9c59b9b28ab8fae1e6bd05d785975bed78b78e71e63c1aaef67702e95871770 Image: gcr.io/engineeringpit/lodestar-images/am:xlou Image ID: gcr.io/engineeringpit/lodestar-images/am@sha256:fae86102dd8374e78ac5d98db56845fe98742e7aebebd5445a81ba84bfa5eed5 Port: Host Port: Command: /home/forgerock/import-pem-certs.sh State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 07 Apr 2023 22:21:15 +0000 Finished: Fri, 07 Apr 2023 22:21:15 +0000 Ready: True Restart Count: 0 Environment: TRUSTSTORE_PATH: /truststore/amtruststore TRUSTSTORE_PASSWORD: changeit AM_PEM_TRUSTSTORE: /var/run/secrets/truststore/cacerts Mounts: /truststore from new-truststore (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-27bn5 (ro) /var/run/secrets/truststore from truststore (rw) Containers: openam: Container ID: containerd://ca4951e55144c777f332602700405370da2695e8603309d35a5e2c220ad04223 Image: gcr.io/engineeringpit/lodestar-images/am:xlou Image ID: gcr.io/engineeringpit/lodestar-images/am@sha256:fae86102dd8374e78ac5d98db56845fe98742e7aebebd5445a81ba84bfa5eed5 Port: 8080/TCP Host Port: 0/TCP Command: bash /home/forgerock/docker-entrypoint.sh State: Running Started: Fri, 07 Apr 2023 22:21:16 +0000 Ready: True Restart Count: 0 Limits: memory: 10Gi Requests: cpu: 11 memory: 10Gi Liveness: http-get http://:8080/am/json/health/live delay=30s timeout=5s period=30s #success=1 #failure=3 Readiness: http-get http://:8080/am/json/health/ready delay=20s timeout=5s period=10s #success=1 #failure=3 Startup: http-get http://:8080/am/json/health/live delay=0s timeout=1s period=10s #success=1 #failure=40 Environment Variables from: am-env-secrets Secret Optional: false platform-config ConfigMap Optional: false amster-env-secrets Secret Optional: false ds-env-secrets Secret Optional: false Environment: NAMESPACE: xlou (v1:metadata.namespace) AM_STORES_SSL_ENABLED: true TRUSTSTORE_PATH: /home/forgerock/amtruststore TRUSTSTORE_PASSWORD: changeit AM_STORES_USER_TYPE: LDAPv3ForForgeRockIAM Mounts: /home/forgerock/amtruststore from new-truststore (ro,path="amtruststore") /home/forgerock/openam from fbc (rw) /var/run/secrets/am from am-secrets (rw) /var/run/secrets/amster from amster-key (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-27bn5 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: fbc: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: am-secrets: Type: Projected (a volume that contains injected data from multiple sources) SecretName: am-passwords SecretOptionalName: SecretName: am-keystore SecretOptionalName: amster-key: Type: Secret (a volume populated by a Secret) SecretName: amster Optional: false truststore: Type: Secret (a volume populated by a Secret) SecretName: truststore-pem Optional: false new-truststore: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-27bn5: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 47m default-scheduler Successfully assigned xlou/am-685d4f4864-7q8cw to gke-xlou-cdm-default-pool-4f098948-tbmx Warning FailedMount 47m kubelet MountVolume.SetUp failed for volume "truststore" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 47m kubelet MountVolume.SetUp failed for volume "am-secrets" : failed to sync secret cache: timed out waiting for the condition Normal Pulling 47m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/am:xlou" Normal Pulled 47m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/am:xlou" in 211.756931ms Normal Created 47m kubelet Created container fbc-init Normal Started 47m kubelet Started container fbc-init Normal Pulling 47m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/am:xlou" Normal Pulled 47m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/am:xlou" in 205.525423ms Normal Created 47m kubelet Created container truststore-init Normal Started 47m kubelet Started container truststore-init Normal Pulling 47m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/am:xlou" Normal Pulled 47m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/am:xlou" in 202.217038ms Normal Created 47m kubelet Created container openam Normal Started 47m kubelet Started container openam Warning Unhealthy 47m (x2 over 47m) kubelet Startup probe failed: Get "http://10.106.43.17:8080/am/json/health/live": context deadline exceeded (Client.Timeout exceeded while awaiting headers) ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== Copying /opt/java/openjdk/lib/security/cacerts to /truststore/amtruststore Found (1) certificates in /var/run/secrets/truststore/cacerts Importing (1) certificates into /truststore/amtruststore Certificate was added to keystore Import complete! 10.106.43.1 - - [07/Apr/2023:23:07:03 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.42.23 - - [07/Apr/2023:23:07:11 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 357229 25ms 10.106.43.1 - - [07/Apr/2023:23:07:13 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.43.1 - - [07/Apr/2023:23:07:13 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms {"timestamp":"2023-04-07T23:07:29.265Z","level":"DEBUG","thread":"http-nio-8081-exec-25","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230150"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2023-04-07T23:07:29.265Z","level":"DEBUG","thread":"http-nio-8081-exec-25","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230150"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2023-04-07T23:07:29.269Z","eventName":"AM-LOGIN-MODULE-COMPLETED","transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230150","trackingIds":["34220ff6-3309-4e2a-9512-333948c285a4-11230153"],"userId":"id=amadmin,ou=user,ou=am-config","principal":["amadmin"],"entries":[{"moduleId":"DataStore","info":{"authControlFlag":"REQUIRED","moduleClass":"DataStore","ipAddress":"10.106.42.10","authLevel":"0"}}],"result":"SUCCESSFUL","realm":"/","component":"Authentication","source":"audit","topic":"authentication","level":"INFO","_eventId":"34220ff6-3309-4e2a-9512-333948c285a4-11230155"} {"timestamp":"2023-04-07T23:07:29.273Z","eventName":"AM-SESSION-CREATED","transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230150","trackingIds":["34220ff6-3309-4e2a-9512-333948c285a4-11230153"],"userId":"id=amadmin,ou=user,ou=am-config","runAs":"id=amadmin,ou=user,ou=am-config","objectId":"34220ff6-3309-4e2a-9512-333948c285a4-11230153","operation":"CREATE","realm":"/","component":"Session","source":"audit","topic":"activity","level":"INFO","_eventId":"34220ff6-3309-4e2a-9512-333948c285a4-11230158"} {"timestamp":"2023-04-07T23:07:29.273Z","eventName":"AM-LOGIN-COMPLETED","transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230150","trackingIds":["34220ff6-3309-4e2a-9512-333948c285a4-11230153"],"userId":"id=amadmin,ou=user,ou=am-config","principal":["amadmin"],"entries":[{"moduleId":"DataStore","info":{"ipAddress":"10.106.42.10","authLevel":"0"}}],"result":"SUCCESSFUL","realm":"/","component":"Authentication","source":"audit","topic":"authentication","level":"INFO","_eventId":"34220ff6-3309-4e2a-9512-333948c285a4-11230160"} {"timestamp":"2023-04-07T23:07:29.274Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230150","trackingIds":["34220ff6-3309-4e2a-9512-333948c285a4-11230153"],"userId":"id=amadmin,ou=user,ou=am-config","client":{"ip":"10.106.42.10","port":58952},"http":{"request":{"secure":true,"method":"POST","path":"https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate","headers":{"accept":["*/*"],"accept-api-version":["protocol=1.0,resource=2.0"],"content-type":["application/json"],"host":["xlou.iam.xlou-cdm.engineeringpit.com"],"user-agent":["python-requests/2.28.2"],"x-forwarded-for":["35.212.145.225"],"x-forwarded-host":["xlou.iam.xlou-cdm.engineeringpit.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-real-ip":["35.212.145.225"],"x-request-id":["95e432c644b1ea4252efd8e5feb28aa1"],"x-scheme":["https"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":9,"elapsedTimeUnits":"MILLISECONDS"},"realm":"/","component":"Authentication","source":"audit","topic":"access","level":"INFO","_eventId":"34220ff6-3309-4e2a-9512-333948c285a4-11230162"} {"timestamp":"2023-04-07T23:07:29.493Z","level":"DEBUG","thread":"http-nio-8081-exec-98","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230163"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2023-04-07T23:07:29.493Z","level":"DEBUG","thread":"http-nio-8081-exec-98","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230163"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2023-04-07T23:07:29.495Z","eventName":"AM-LOGIN-MODULE-COMPLETED","transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230163","trackingIds":["34220ff6-3309-4e2a-9512-333948c285a4-11230166"],"userId":"id=amadmin,ou=user,ou=am-config","principal":["amadmin"],"entries":[{"moduleId":"DataStore","info":{"authControlFlag":"REQUIRED","moduleClass":"DataStore","ipAddress":"10.106.42.10","authLevel":"0"}}],"result":"SUCCESSFUL","realm":"/","component":"Authentication","source":"audit","topic":"authentication","level":"INFO","_eventId":"34220ff6-3309-4e2a-9512-333948c285a4-11230168"} {"timestamp":"2023-04-07T23:07:29.499Z","eventName":"AM-SESSION-CREATED","transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230163","trackingIds":["34220ff6-3309-4e2a-9512-333948c285a4-11230166"],"userId":"id=amadmin,ou=user,ou=am-config","runAs":"id=amadmin,ou=user,ou=am-config","objectId":"34220ff6-3309-4e2a-9512-333948c285a4-11230166","operation":"CREATE","realm":"/","component":"Session","source":"audit","topic":"activity","level":"INFO","_eventId":"34220ff6-3309-4e2a-9512-333948c285a4-11230171"} {"timestamp":"2023-04-07T23:07:29.499Z","eventName":"AM-LOGIN-COMPLETED","transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230163","trackingIds":["34220ff6-3309-4e2a-9512-333948c285a4-11230166"],"userId":"id=amadmin,ou=user,ou=am-config","principal":["amadmin"],"entries":[{"moduleId":"DataStore","info":{"ipAddress":"10.106.42.10","authLevel":"0"}}],"result":"SUCCESSFUL","realm":"/","component":"Authentication","source":"audit","topic":"authentication","level":"INFO","_eventId":"34220ff6-3309-4e2a-9512-333948c285a4-11230173"} {"timestamp":"2023-04-07T23:07:29.499Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230163","trackingIds":["34220ff6-3309-4e2a-9512-333948c285a4-11230166"],"userId":"id=amadmin,ou=user,ou=am-config","client":{"ip":"10.106.42.10","port":58952},"http":{"request":{"secure":true,"method":"POST","path":"https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate","headers":{"accept":["*/*"],"accept-api-version":["protocol=1.0,resource=2.0"],"content-type":["application/json"],"host":["xlou.iam.xlou-cdm.engineeringpit.com"],"user-agent":["python-requests/2.28.2"],"x-forwarded-for":["35.212.145.225"],"x-forwarded-host":["xlou.iam.xlou-cdm.engineeringpit.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-real-ip":["35.212.145.225"],"x-request-id":["4bac2d4e843293ac3f918913e034f130"],"x-scheme":["https"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":6,"elapsedTimeUnits":"MILLISECONDS"},"realm":"/","component":"Authentication","source":"audit","topic":"access","level":"INFO","_eventId":"34220ff6-3309-4e2a-9512-333948c285a4-11230175"} {"timestamp":"2023-04-07T23:07:29.709Z","level":"DEBUG","thread":"http-nio-8081-exec-57","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230176"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2023-04-07T23:07:29.709Z","level":"DEBUG","thread":"http-nio-8081-exec-57","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230176"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2023-04-07T23:07:29.711Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11230176","trackingIds":["34220ff6-3309-4e2a-9512-333948c285a4-11230166"],"userId":"id=amadmin,ou=user,ou=am-config","client":{"ip":"10.106.42.10","port":39944},"server":{"ip":"10.106.43.17","port":8081},"http":{"request":{"secure":true,"method":"GET","path":"https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version","headers":{"accept":["*/*"],"accept-api-version":["protocol=2.1"],"host":["xlou.iam.xlou-cdm.engineeringpit.com"],"user-agent":["python-requests/2.28.2"],"x-forwarded-for":["35.212.145.225"],"x-forwarded-host":["xlou.iam.xlou-cdm.engineeringpit.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-real-ip":["35.212.145.225"],"x-request-id":["5095bd3a8cc9d0c19e8431f6ce756179"],"x-scheme":["https"]}}},"request":{"protocol":"CREST","operation":"READ"},"response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":3,"elapsedTimeUnits":"MILLISECONDS","detail":{"revision":"-560502796"}},"realm":"/","component":"Server Info","source":"audit","topic":"access","level":"INFO","_eventId":"34220ff6-3309-4e2a-9512-333948c285a4-11230180"} 10.106.43.1 - - [07/Apr/2023:23:07:23 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.42.10 - - [07/Apr/2023:23:07:29 +0000] "POST /am/json/authenticate?realm=/ HTTP/1.1" 200 167 10ms 10.106.42.10 - - [07/Apr/2023:23:07:29 +0000] "POST /am/json/authenticate?realm=/ HTTP/1.1" 200 167 7ms 10.106.42.10 - - [07/Apr/2023:23:07:29 +0000] "GET /am/json/serverinfo/version HTTP/1.1" 200 275 4ms 10.106.43.1 - - [07/Apr/2023:23:07:33 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.42.23 - - [07/Apr/2023:23:07:41 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 357257 26ms 10.106.43.1 - - [07/Apr/2023:23:07:43 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.43.1 - - [07/Apr/2023:23:07:43 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms {"timestamp":"2023-04-07T23:07:59.510Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(30)","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11229140"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default"} {"timestamp":"2023-04-07T23:07:59.968Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(30)","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11229140"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default"} 10.106.43.1 - - [07/Apr/2023:23:07:53 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.43.1 - - [07/Apr/2023:23:08:03 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms {"timestamp":"2023-04-07T23:08:15.973Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(30)","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11229140"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default"} {"timestamp":"2023-04-07T23:08:16.968Z","level":"WARN","thread":"OpenDJ LDAP SDK Client Worker(30)","mdc":{"transactionId":"34220ff6-3309-4e2a-9512-333948c285a4-11229140"},"logger":"com.sun.identity.idm.IdRepoListener","message":"objectChanged called with an empty name","context":"default"} 10.106.42.23 - - [07/Apr/2023:23:08:11 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 357252 25ms 10.106.43.1 - - [07/Apr/2023:23:08:13 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 2ms 10.106.43.1 - - [07/Apr/2023:23:08:13 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.43.1 - - [07/Apr/2023:23:08:23 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms 10.106.43.1 - - [07/Apr/2023:23:08:33 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 2ms 10.106.42.23 - - [07/Apr/2023:23:08:41 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 357269 26ms 10.106.43.1 - - [07/Apr/2023:23:08:43 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 1ms 10.106.43.1 - - [07/Apr/2023:23:08:43 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 1ms Copying docker image configuration files to the shared volume