Name: am-f66db87f4-f75d2 Namespace: fr-platform Priority: 999999 Priority Class Name: medium-priority-preempting Node: gke-default-cluster-1-489b2599-55tc/10.40.0.6 Start Time: Tue, 19 Mar 2024 18:35:56 +0000 Labels: app=am app.kubernetes.io/component=am app.kubernetes.io/instance=am app.kubernetes.io/name=am app.kubernetes.io/part-of=forgerock component=am forgerock.io/am.tag=7.5.0-2024-2-2-cbdf0302eac3978c68f50b853a4495a622999003 pod-template-hash=f66db87f4 tier=middle vendor=forgerock Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true cni.projectcalico.org/containerID: eb38270dbbe3f84ad38f3b3cd51bf56853d5abf000530cf66a2e50e8d62318dd cni.projectcalico.org/podIP: cni.projectcalico.org/podIPs: forgerock.io/restartedAt: 2024-03-18T14:37:11Z idcloud.forgerock.com/cookie-domains: openam-ema-perf-benchmark.forgeblocks.com idcloud.forgerock.com/custom-domains: com.sun.identity.server.fqdnMap[openam-ema-perf-benchmark.forgeblocks.com]=openam-ema-perf-benchmark.forgeblocks.com kubectl.kubernetes.io/restartedAt: 2024-03-16T04:14:27Z promcat.sysdig.com/omit: true prometheus.io/path: /am/json/metrics/prometheus prometheus.io/port: 8080 prometheus.io/scrape: am-metrics Status: Failed Reason: Evicted Message: The node was low on resource: memory. Threshold quantity: 100Mi, available: 616Ki. Container openam was using 6588236Ki, request is 4Gi, has larger consumption of memory. IP: 10.100.1.208 IPs: IP: 10.100.1.208 Controlled By: ReplicaSet/am-f66db87f4 Init Containers: workload-identity-init: Container ID: containerd://7084c7ae462621e167ba6b08504db6bb99011da695bfe783b98476de59d0d0fc Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/workload-identity-init:master-759f12d369898b631e46281c36dfe24871a93059 Image ID: us-docker.pkg.dev/fr-pre-prod-registry/container-images/workload-identity-init@sha256:0a11adeefef7c40f1a1ddd119bafb305c5c7c79125f9dd78f7d071dcd993a6f7 Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 19 Mar 2024 18:35:57 +0000 Finished: Tue, 19 Mar 2024 18:35:58 +0000 Ready: True Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) fbc-init: Container ID: containerd://69c1a73a92324dde1b277883f03e50b701626e87475370d59fbeafa6d843fca8 Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/am:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1 Image ID: us-docker.pkg.dev/fr-pre-prod-registry/container-images/am@sha256:0c8fbd37c325903d560e28ad88815e2372a188d0428573255b23af9df9960656 Port: Host Port: Command: /bin/sh Args: -c rm -rf /fbc/* /fbc/.git* && cp -a /home/forgerock/openam/config/* /home/forgerock/openam/config/.git* /fbc State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 19 Mar 2024 18:35:59 +0000 Finished: Tue, 19 Mar 2024 18:36:18 +0000 Ready: True Restart Count: 0 Environment: Mounts: /fbc from fbc (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) load-config-clone: Container ID: containerd://1f373a6c886297dadfef63e1330263667a66b7d4d9ccdd006bfe252d6b1133e1 Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/config-loader:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1 Image ID: us-docker.pkg.dev/fr-pre-prod-registry/container-images/config-loader@sha256:a3c2c0f53d1d926027446a75ba9455f08bd35739ab27c618e2a6594ecd7d0e61 Port: Host Port: Args: clone-and-copy State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 19 Mar 2024 18:36:19 +0000 Finished: Tue, 19 Mar 2024 18:36:23 +0000 Ready: True Restart Count: 0 Environment: SKIP_COMMIT_VERIFICATION: true ACTIVE_TAG: master CONFIG_SRC_PATH: am COMMIT_MESSAGE: Customer config DESTINATION_PATH: /home/forgerock/openam/config GIT_PATH: /home/forgerock/customer-config ADD_PATHS: services ENGINEERS_KEYRING: /home/forgerock/pgp/fraas-engineers.keyring.asc CONFIG_LOAD_STRATEGY: JSON_MERGE LOG_LEVEL: INFO Mounts: /home/forgerock/customer-config from customer-config (rw) /home/forgerock/openam/config from fbc (rw) /home/forgerock/pgp from engineers-keyring (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) upgrade-init: Container ID: containerd://e34219c351768b54f6530425583c726ac8635692017063287949b69e3f6a3483 Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/fbc-tools:master-759f12d369898b631e46281c36dfe24871a93059 Image ID: us-docker.pkg.dev/fr-pre-prod-registry/container-images/fbc-tools@sha256:4783690c75b00e2715626c4b62ed2023b71b6b4efe8c808fb51808af1d2ca15e Port: Host Port: Command: /bin/sh Args: -c rm -rf /customer-config-upgrade/* && cp -a /customer-config/am/services /customer-config-upgrade || mkdir -p /customer-config-upgrade/services/realm/root State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 19 Mar 2024 18:36:25 +0000 Finished: Tue, 19 Mar 2024 18:36:25 +0000 Ready: True Restart Count: 0 Environment: Mounts: /customer-config from customer-config (rw) /customer-config-upgrade from customer-config-upgraded (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) am-config-upgrader: Container ID: containerd://7dbf8caf2c4c8b0b7377932b3029e6211ad9919e670e61571a1918b6aee6fc2d Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/am-config-upgrader:master-759f12d369898b631e46281c36dfe24871a93059 Image ID: us-docker.pkg.dev/fr-pre-prod-registry/container-images/am-config-upgrader@sha256:854d00d538802ef081b08d1e308c2f2ebf7331d2b41a017f628469d557cd1fa4 Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 19 Mar 2024 18:36:27 +0000 Finished: Tue, 19 Mar 2024 18:37:01 +0000 Ready: True Restart Count: 0 Environment: Mounts: /am-config/config from customer-config-upgraded (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) upgraded-config: Container ID: containerd://6f5200fd5774296dd888bb4b4061609a9ce902eb5ca8c59befbf420440d6e05a Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/am:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1 Image ID: us-docker.pkg.dev/fr-pre-prod-registry/container-images/am@sha256:0c8fbd37c325903d560e28ad88815e2372a188d0428573255b23af9df9960656 Port: Host Port: Command: /bin/bash Args: -c echo "debugging_enabled=$UPGRADED_CONFIG_DEBUG" && if [ "$UPGRADED_CONFIG_DEBUG" == "true" ]; then echo "Upgraded config" && find /customer-config-upgrade -type f ; fi && rm -rf /fbc/services && if [ "$UPGRADED_CONFIG_DEBUG" == "true" ]; then echo "Empty" && find /fbc -type f ; fi && cp -a /home/forgerock/openam/config/services /fbc && if [ "$UPGRADED_CONFIG_DEBUG" == "true" ]; then echo "Image config" && find /fbc -type f ; fi && cp -a /customer-config-upgrade/services /fbc && if [ "$UPGRADED_CONFIG_DEBUG" == "true" ]; then echo "Upgraded customer config config" && find /fbc -type f ; fi State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 19 Mar 2024 18:37:05 +0000 Finished: Tue, 19 Mar 2024 18:37:07 +0000 Ready: True Restart Count: 0 Environment: UPGRADED_CONFIG_DEBUG: Mounts: /customer-config-upgrade from customer-config-upgraded (rw) /fbc from fbc (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) commit-upgraded-config: Container ID: containerd://5df8db3744f11863186cecbf85718d6df451935397b9b5df648a46926013be31 Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/config-loader:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1 Image ID: us-docker.pkg.dev/fr-pre-prod-registry/container-images/config-loader@sha256:a3c2c0f53d1d926027446a75ba9455f08bd35739ab27c618e2a6594ecd7d0e61 Port: Host Port: Args: commit State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 19 Mar 2024 18:37:09 +0000 Finished: Tue, 19 Mar 2024 18:37:11 +0000 Ready: True Restart Count: 0 Environment Variables from: am-upgraded-config ConfigMap Optional: false Environment: SKIP_COMMIT_VERIFICATION: true CONFIG_SRC_PATH: am DESTINATION_PATH: /home/forgerock/openam/config GIT_PATH: /home/forgerock/customer-config ADD_PATHS: services ENGINEERS_KEYRING: /home/forgerock/pgp/fraas-engineers.keyring.asc CONFIG_LOAD_STRATEGY: JSON_MERGE LOG_LEVEL: INFO Mounts: /home/forgerock/openam/config from fbc (rw) /home/forgerock/pgp from engineers-keyring (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) custom-cacerts: Container ID: containerd://ada3691cea579152bf21a8a194be01a9c3496e16b1a05a2fc9f3556662663343 Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/am:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1 Image ID: us-docker.pkg.dev/fr-pre-prod-registry/container-images/am@sha256:0c8fbd37c325903d560e28ad88815e2372a188d0428573255b23af9df9960656 Port: Host Port: Command: /home/forgerock/add-custom-cacerts.sh Args: /opt/java/openjdk/lib/security /home/forgerock/cacerts /home/forgerock/custom-cacerts JKS State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 19 Mar 2024 18:37:13 +0000 Finished: Tue, 19 Mar 2024 18:37:21 +0000 Ready: True Restart Count: 0 Environment: GOOGLE_PROJECT_ID: Optional: false Mounts: /home/forgerock/cacerts from cacerts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) wait-for-ds: Container ID: containerd://0c308170001918bea74da4141bab886e0d68e40b920042f84becf2a3d776b0ec Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/util:master-759f12d369898b631e46281c36dfe24871a93059 Image ID: us-docker.pkg.dev/fr-pre-prod-registry/container-images/util@sha256:05e9bc36e3ce7e915b397cd6ea785d24b9187d1939b74b1ee94bd92b3bb6b227 Port: Host Port: Args: waitForPort ctsstore-0.ctsstore:1389/ctsstore-1.ctsstore:1389/ctsstore-2.ctsstore:1389 userstore-0.userstore:1389/userstore-1.userstore:1389/userstore-2.userstore:1389 State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 19 Mar 2024 18:37:22 +0000 Finished: Tue, 19 Mar 2024 18:37:22 +0000 Ready: True Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) Containers: openam: Container ID: Image: us-docker.pkg.dev/fr-pre-prod-registry/container-images/am:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1 Image ID: Port: 8080/TCP Host Port: 0/TCP State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was terminated Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Last State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was deleted. The container used to be Running Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Ready: False Restart Count: 1 Limits: cpu: 16 memory: 16Gi Requests: cpu: 4 memory: 4Gi Liveness: http-get http://:8080/am/json/health/live delay=0s timeout=15s period=5s #success=1 #failure=3 Readiness: http-get http://:8080/am/json/health/ready delay=0s timeout=5s period=2s #success=1 #failure=2 Startup: http-get http://:8080/am/json/health/live delay=15s timeout=2s period=10s #success=1 #failure=40 Environment Variables from: platform-config ConfigMap Optional: false platform-config-dynamic-k5dh7d9589 ConfigMap Optional: false am-configmap ConfigMap Optional: false am-config-overrides ConfigMap Optional: true am-generated-secrets Secret Optional: false am-custom-domains ConfigMap Optional: true auto-access-hmac-key Secret Optional: true Environment: RSFILTER_PROVISIONING_SECRET: Optional: false REPLICATION_CONTROLLER: (v1:metadata.labels['pod-template-hash']) AUTONOMOUS_ACCESS_SIGNATURE_SECRET_KEY: Optional: true RSFILTER_RESOURCE_SERVER_SECRET: Optional: false CONTINUE_ON_UNRESOLVED_PLACEHOLDERS: false CONTINUE_ON_SECRETS_LOADER_ERROR: false NAMESPACE: fr-platform (v1:metadata.namespace) AM_STORES_SSL_ENABLED: false TRUSTSTORE_PATH: /var/run/secrets/truststore/cacerts TRUSTSTORE_PASSWORD: changeit COOKIE_NAME: 0e26551c0104b88 AM_STORES_CTS_PASSWORD: Optional: false AM_STORES_CTS_SERVERS: Optional: false AM_STORES_USER_PASSWORD: Optional: false AM_STORES_USER_SERVERS: Optional: false AM_STORES_USER_TYPE: LDAPv3ForOpenDS AM_PROMETHEUS_USERNAME: Optional: false AM_PROMETHEUS_PASSWORD: Optional: false IDM_USERNAME: Optional: false IDM_PASSWORD: Optional: false ESV_USERNAME: Optional: false ESV_PASSWORD: Optional: false ENABLE_AUTO_ACCESS: false AUTO_ACCESS_TENANT_ID: TOMCAT_ACCESS_LOGGING: DISABLED GIT_PATH: /home/forgerock/openam/config/services SVCACCT_TOKEN_ENCRYPTION_KEY: Optional: false FRAAS_SVCACCT_ALLOWED_SCOPES: fr:am:* fr:idc:analytics:* fr:autoaccess:* fr:idc:certificate:* fr:idc:certificate:read fr:idc:content-security-policy:* fr:idc:custom-domain:* fr:idc:esv:* fr:idc:esv:read fr:idc:esv:restart fr:idc:esv:update fr:idm:* fr:iga:* fr:idc:promotion:* fr:idc:release:* fr:idc:sso-cookie:* Mounts: /home/forgerock/openam/config from fbc (rw) /home/forgerock/web-xml-transform.xslt from am-web-xml (rw,path="transform.xslt") /opt/java/openjdk/lib/security from cacerts (ro) /usr/local/tomcat/webapps/am/WEB-INF/classes/logback.xml from am-logging-config (rw,path="logback.xml") /var/run/secrets/am from openam-keys (rw) /var/run/secrets/am-rootrealm/keystore from am-rootrealm-keystore (rw) /var/run/secrets/am-rootrealm/passwords from am-rootrealm-passwords (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qjsl (ro) Conditions: Type Status DisruptionTarget True Initialized True Ready False ContainersReady False PodScheduled True Volumes: openam-keys: Type: Secret (a volume populated by a Secret) SecretName: openam-keys Optional: false am-rootrealm-keystore: Type: Secret (a volume populated by a Secret) SecretName: am-rootrealm-keystore Optional: false am-rootrealm-passwords: Type: Secret (a volume populated by a Secret) SecretName: am-rootrealm-passwords Optional: false fbc: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: customer-config: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: customer-config-upgraded: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: cacerts: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: am-logging-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: am-logging-config Optional: false am-web-xml: Type: ConfigMap (a volume populated by a ConfigMap) Name: am-web-xml Optional: false engineers-keyring: Type: ConfigMap (a volume populated by a ConfigMap) Name: engineers-keyring Optional: false kube-api-access-9qjsl: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 49m gke.io/optimize-utilization-scheduler Successfully assigned fr-platform/am-f66db87f4-f75d2 to gke-default-cluster-1-489b2599-55tc Normal Pulled 49m kubelet Container image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/workload-identity-init:master-759f12d369898b631e46281c36dfe24871a93059" already present on machine Normal Created 49m kubelet Created container workload-identity-init Normal Started 49m kubelet Started container workload-identity-init Normal Pulling 49m kubelet Pulling image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/am:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1" Normal Pulled 49m kubelet Successfully pulled image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/am:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1" in 284.017084ms (284.030625ms including waiting) Normal Created 49m kubelet Created container fbc-init Normal Started 49m kubelet Started container fbc-init Normal Pulling 48m kubelet Pulling image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/config-loader:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1" Normal Pulled 48m kubelet Successfully pulled image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/config-loader:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1" in 265.252605ms (265.276613ms including waiting) Normal Created 48m kubelet Created container load-config-clone Normal Started 48m kubelet Started container load-config-clone Normal Pulling 48m kubelet Pulling image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/fbc-tools:master-759f12d369898b631e46281c36dfe24871a93059" Normal Pulled 48m kubelet Successfully pulled image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/fbc-tools:master-759f12d369898b631e46281c36dfe24871a93059" in 397.645064ms (397.660344ms including waiting) Normal Created 48m kubelet Created container upgrade-init Normal Started 48m kubelet Started container upgrade-init Normal Pulling 48m kubelet Pulling image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/am-config-upgrader:master-759f12d369898b631e46281c36dfe24871a93059" Normal Pulled 48m kubelet Successfully pulled image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/am-config-upgrader:master-759f12d369898b631e46281c36dfe24871a93059" in 285.535793ms (285.56785ms including waiting) Normal Created 48m kubelet Created container am-config-upgrader Normal Started 48m kubelet Started container am-config-upgrader Normal Pulling 48m kubelet Pulling image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/am:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1" Normal Pulled 48m kubelet Successfully pulled image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/am:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1" in 426.743425ms (426.768182ms including waiting) Normal Created 48m kubelet Created container upgraded-config Normal Started 48m kubelet Started container upgraded-config Normal Pulling 48m kubelet Pulling image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/config-loader:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1" Normal Pulled 48m kubelet Successfully pulled image "us-docker.pkg.dev/fr-pre-prod-registry/container-images/config-loader:master-e773dd3dc345c4add554add4a31eb3d7c254d1b1" in 204.192334ms (204.207453ms including waiting) Warning Unhealthy 39m kubelet Readiness probe failed: Get "http://10.100.1.208:8080/am/json/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Evicted 36m kubelet The node was low on resource: memory. Threshold quantity: 100Mi, available: 616Ki. Container openam was using 6588236Ki, request is 4Gi, has larger consumption of memory.