--Task--
name: Discover_all_components
enabled: True
class_name: DiscoverComponentsTask
source_name: controller
source_namespace: >default<
target_name: controller
target_namespace: >default<
start: 0.0
stop: None
timeout: no timeout
loop: False
interval: None
dependencies: []
wait_for: []
preceding_task: None
options: {}
group_name: None
Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock
________________________________________________________________________________
[2024-04-12 20:58:35] Discover_all_components pre : Initialising task parameters
________________________________________________________________________________
task will be executed on controller (localhost)
________________________________________________________________________________
[2024-04-12 20:58:35] Discover_all_components step1 : Discover components
________________________________________________________________________________
2024-04-12 20:58:35,350 INFO Deployment was skipped.
2024-04-12 20:58:35,684 INFO
2024-04-12 20:58:35,684 INFO ************************************* Creating deployment *************************************
2024-04-12 20:58:35,684 INFO Creating normal (forgeops) type deployment for deployment: stack
2024-04-12 20:58:35,700 INFO
2024-04-12 20:58:35,700 INFO ------- Custom component configuration present. Loading values -------
2024-04-12 20:58:35,700 INFO
2024-04-12 20:58:35,701 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker ds cts
2024-04-12 20:58:35,775 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:58:35,776 DEBUG --- stdout ---
2024-04-12 20:58:35,776 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/cts
2024-04-12 20:58:35,776 DEBUG --- stderr ---
2024-04-12 20:58:35,776 DEBUG
2024-04-12 20:58:35,776 INFO
2024-04-12 20:58:35,776 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/cts/Dockerfile: gcr.io/forgerock-io/ds/pit1:7.6.0-d49ad6a5879fab3530729d8773297783d042f903
2024-04-12 20:58:35,777 INFO
2024-04-12 20:58:35,777 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker ds idrepo
2024-04-12 20:58:35,851 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:58:35,851 DEBUG --- stdout ---
2024-04-12 20:58:35,851 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/idrepo
2024-04-12 20:58:35,851 DEBUG --- stderr ---
2024-04-12 20:58:35,851 DEBUG
2024-04-12 20:58:35,851 INFO
2024-04-12 20:58:35,851 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/idrepo/Dockerfile: gcr.io/forgerock-io/ds/pit1:7.6.0-d49ad6a5879fab3530729d8773297783d042f903
2024-04-12 20:58:35,852 INFO
2024-04-12 20:58:35,852 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker am
2024-04-12 20:58:35,928 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:58:35,928 DEBUG --- stdout ---
2024-04-12 20:58:35,928 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/am
2024-04-12 20:58:35,928 DEBUG --- stderr ---
2024-04-12 20:58:35,928 DEBUG
2024-04-12 20:58:35,928 INFO
2024-04-12 20:58:35,929 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/am/Dockerfile: gcr.io/forgerock-io/am-cdk/pit1:7.6.0-0449af26230b3dbde39b69d27b76a420998ce4a5
2024-04-12 20:58:35,929 INFO
2024-04-12 20:58:35,929 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker amster
2024-04-12 20:58:36,004 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:58:36,005 DEBUG --- stdout ---
2024-04-12 20:58:36,005 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/amster
2024-04-12 20:58:36,005 DEBUG --- stderr ---
2024-04-12 20:58:36,005 DEBUG
2024-04-12 20:58:36,005 INFO
2024-04-12 20:58:36,005 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/amster/Dockerfile: gcr.io/forgerock-io/amster/pit1:7.6.0-0449af26230b3dbde39b69d27b76a420998ce4a5
2024-04-12 20:58:36,005 INFO
2024-04-12 20:58:36,005 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker idm
2024-04-12 20:58:36,079 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:58:36,079 DEBUG --- stdout ---
2024-04-12 20:58:36,079 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/idm
2024-04-12 20:58:36,079 DEBUG --- stderr ---
2024-04-12 20:58:36,079 DEBUG
2024-04-12 20:58:36,079 INFO
2024-04-12 20:58:36,080 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/idm/Dockerfile: gcr.io/forgerock-io/idm-cdk/pit1:7.6.0-8554e0d1c5044f4f6c6504cd366196848874e695
2024-04-12 20:58:36,080 INFO
2024-04-12 20:58:36,080 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base end-user-ui
2024-04-12 20:58:36,152 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:58:36,152 DEBUG --- stdout ---
2024-04-12 20:58:36,153 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/end-user-ui
2024-04-12 20:58:36,153 DEBUG --- stderr ---
2024-04-12 20:58:36,153 DEBUG
2024-04-12 20:58:36,153 INFO
2024-04-12 20:58:36,156 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/end-user-ui/kustomization.yaml: gcr.io/forgerock-io/platform-enduser-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-12 20:58:36,156 INFO
2024-04-12 20:58:36,156 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base login-ui
2024-04-12 20:58:36,229 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:58:36,229 DEBUG --- stdout ---
2024-04-12 20:58:36,229 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/login-ui
2024-04-12 20:58:36,229 DEBUG --- stderr ---
2024-04-12 20:58:36,229 DEBUG
2024-04-12 20:58:36,229 INFO
2024-04-12 20:58:36,232 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/login-ui/kustomization.yaml: gcr.io/forgerock-io/platform-login-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-12 20:58:36,232 INFO
2024-04-12 20:58:36,232 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base admin-ui
2024-04-12 20:58:36,304 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:58:36,304 DEBUG --- stdout ---
2024-04-12 20:58:36,304 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/admin-ui
2024-04-12 20:58:36,304 DEBUG --- stderr ---
2024-04-12 20:58:36,304 DEBUG
2024-04-12 20:58:36,304 INFO
2024-04-12 20:58:36,307 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/admin-ui/kustomization.yaml: gcr.io/forgerock-io/platform-admin-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-12 20:58:36,307 INFO Loading external components from config.yaml
2024-04-12 20:58:36,419 INFO No external components defined.
2024-04-12 20:58:36,536 INFO Setting context for components to: gke_engineeringpit_us-east1-d_xlou-cdm
2024-04-12 20:58:36,536 INFO
2024-04-12 20:58:36,536 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-12 20:58:36,536 INFO
2024-04-12 20:58:36,536 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:58:36,536 INFO
2024-04-12 20:58:36,536 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:58:36,536 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:36,931 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:36,931 DEBUG --- stdout ---
2024-04-12 20:58:36,931 DEBUG 3
2024-04-12 20:58:36,931 DEBUG --- stderr ---
2024-04-12 20:58:36,932 DEBUG
2024-04-12 20:58:36,932 INFO
2024-04-12 20:58:36,932 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:58:36,932 INFO
2024-04-12 20:58:36,932 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-cts --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-12 20:58:36,932 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:58:37,383 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-12 20:58:37,383 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:37,383 DEBUG --- stdout ---
2024-04-12 20:58:37,384 DEBUG ds-cts-0 ds-cts-1 ds-cts-2
2024-04-12 20:58:37,384 DEBUG --- stderr ---
2024-04-12 20:58:37,384 DEBUG
2024-04-12 20:58:37,384 INFO
2024-04-12 20:58:37,384 INFO -------------------- Check pod ds-cts-0 is running --------------------
2024-04-12 20:58:37,384 INFO
2024-04-12 20:58:37,384 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-0 -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:58:37,384 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:37,758 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:37,758 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:37,758 DEBUG --- stdout ---
2024-04-12 20:58:37,759 DEBUG Running
2024-04-12 20:58:37,759 DEBUG --- stderr ---
2024-04-12 20:58:37,759 DEBUG
2024-04-12 20:58:37,759 INFO
2024-04-12 20:58:37,759 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-0 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:58:37,759 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:38,128 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:38,128 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:38,128 DEBUG --- stdout ---
2024-04-12 20:58:38,128 DEBUG true
2024-04-12 20:58:38,128 DEBUG --- stderr ---
2024-04-12 20:58:38,128 DEBUG
2024-04-12 20:58:38,129 INFO
2024-04-12 20:58:38,129 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-0 --output jsonpath={.status.startTime}
2024-04-12 20:58:38,129 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:38,489 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:38,489 DEBUG --- stdout ---
2024-04-12 20:58:38,489 DEBUG 2024-04-12T16:44:43Z
2024-04-12 20:58:38,489 DEBUG --- stderr ---
2024-04-12 20:58:38,489 DEBUG
2024-04-12 20:58:38,490 INFO
2024-04-12 20:58:38,490 INFO ------------- Check pod ds-cts-0 filesystem is accessible -------------
2024-04-12 20:58:39,321 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:39,322 INFO
2024-04-12 20:58:39,323 INFO ------------------ Check pod ds-cts-0 restart count ------------------
2024-04-12 20:58:39,323 INFO
2024-04-12 20:58:39,323 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-0 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:58:39,323 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:39,679 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:39,679 DEBUG --- stdout ---
2024-04-12 20:58:39,679 DEBUG 0
2024-04-12 20:58:39,679 DEBUG --- stderr ---
2024-04-12 20:58:39,679 DEBUG
2024-04-12 20:58:39,679 INFO Pod ds-cts-0 has been restarted 0 times.
2024-04-12 20:58:39,679 INFO
2024-04-12 20:58:39,679 INFO -------------------- Check pod ds-cts-1 is running --------------------
2024-04-12 20:58:39,680 INFO
2024-04-12 20:58:39,680 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-1 -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:58:39,680 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:40,036 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:40,036 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:40,036 DEBUG --- stdout ---
2024-04-12 20:58:40,036 DEBUG Running
2024-04-12 20:58:40,036 DEBUG --- stderr ---
2024-04-12 20:58:40,036 DEBUG
2024-04-12 20:58:40,037 INFO
2024-04-12 20:58:40,037 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-1 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:58:40,037 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:40,395 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:40,395 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:40,395 DEBUG --- stdout ---
2024-04-12 20:58:40,395 DEBUG true
2024-04-12 20:58:40,395 DEBUG --- stderr ---
2024-04-12 20:58:40,395 DEBUG
2024-04-12 20:58:40,395 INFO
2024-04-12 20:58:40,395 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-1 --output jsonpath={.status.startTime}
2024-04-12 20:58:40,395 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:40,749 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:40,749 DEBUG --- stdout ---
2024-04-12 20:58:40,749 DEBUG 2024-04-12T16:45:11Z
2024-04-12 20:58:40,749 DEBUG --- stderr ---
2024-04-12 20:58:40,749 DEBUG
2024-04-12 20:58:40,749 INFO
2024-04-12 20:58:40,750 INFO ------------- Check pod ds-cts-1 filesystem is accessible -------------
2024-04-12 20:58:41,594 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:41,594 INFO
2024-04-12 20:58:41,594 INFO ------------------ Check pod ds-cts-1 restart count ------------------
2024-04-12 20:58:41,594 INFO
2024-04-12 20:58:41,594 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-1 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:58:41,594 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:41,970 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:41,970 DEBUG --- stdout ---
2024-04-12 20:58:41,970 DEBUG 0
2024-04-12 20:58:41,970 DEBUG --- stderr ---
2024-04-12 20:58:41,970 DEBUG
2024-04-12 20:58:41,970 INFO Pod ds-cts-1 has been restarted 0 times.
2024-04-12 20:58:41,971 INFO
2024-04-12 20:58:41,971 INFO -------------------- Check pod ds-cts-2 is running --------------------
2024-04-12 20:58:41,971 INFO
2024-04-12 20:58:41,971 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:58:41,971 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:42,363 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:42,363 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:42,363 DEBUG --- stdout ---
2024-04-12 20:58:42,364 DEBUG Running
2024-04-12 20:58:42,364 DEBUG --- stderr ---
2024-04-12 20:58:42,364 DEBUG
2024-04-12 20:58:42,364 INFO
2024-04-12 20:58:42,364 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:58:42,364 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:42,727 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:42,728 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:42,728 DEBUG --- stdout ---
2024-04-12 20:58:42,728 DEBUG true
2024-04-12 20:58:42,728 DEBUG --- stderr ---
2024-04-12 20:58:42,728 DEBUG
2024-04-12 20:58:42,728 INFO
2024-04-12 20:58:42,729 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-2 --output jsonpath={.status.startTime}
2024-04-12 20:58:42,729 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:43,085 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:43,086 DEBUG --- stdout ---
2024-04-12 20:58:43,086 DEBUG 2024-04-12T16:45:38Z
2024-04-12 20:58:43,086 DEBUG --- stderr ---
2024-04-12 20:58:43,086 DEBUG
2024-04-12 20:58:43,086 INFO
2024-04-12 20:58:43,087 INFO ------------- Check pod ds-cts-2 filesystem is accessible -------------
2024-04-12 20:58:43,957 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:43,957 INFO
2024-04-12 20:58:43,957 INFO ------------------ Check pod ds-cts-2 restart count ------------------
2024-04-12 20:58:43,957 INFO
2024-04-12 20:58:43,957 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:58:43,957 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:44,320 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:44,320 DEBUG --- stdout ---
2024-04-12 20:58:44,320 DEBUG 0
2024-04-12 20:58:44,320 DEBUG --- stderr ---
2024-04-12 20:58:44,320 DEBUG
2024-04-12 20:58:44,320 INFO Pod ds-cts-2 has been restarted 0 times.
2024-04-12 20:58:44,320 INFO
2024-04-12 20:58:44,320 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-12 20:58:44,320 INFO
2024-04-12 20:58:44,320 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:58:44,320 INFO
2024-04-12 20:58:44,321 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:58:44,321 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:44,683 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:44,683 DEBUG --- stdout ---
2024-04-12 20:58:44,683 DEBUG 3
2024-04-12 20:58:44,683 DEBUG --- stderr ---
2024-04-12 20:58:44,683 DEBUG
2024-04-12 20:58:44,683 INFO
2024-04-12 20:58:44,683 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:58:44,683 INFO
2024-04-12 20:58:44,683 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-idrepo --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-12 20:58:44,683 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:58:45,115 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-12 20:58:45,115 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:45,115 DEBUG --- stdout ---
2024-04-12 20:58:45,115 DEBUG ds-idrepo-0 ds-idrepo-1 ds-idrepo-2
2024-04-12 20:58:45,115 DEBUG --- stderr ---
2024-04-12 20:58:45,115 DEBUG
2024-04-12 20:58:45,115 INFO
2024-04-12 20:58:45,115 INFO ------------------ Check pod ds-idrepo-0 is running ------------------
2024-04-12 20:58:45,115 INFO
2024-04-12 20:58:45,115 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-0 -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:58:45,115 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:45,472 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:45,472 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:45,472 DEBUG --- stdout ---
2024-04-12 20:58:45,472 DEBUG Running
2024-04-12 20:58:45,472 DEBUG --- stderr ---
2024-04-12 20:58:45,472 DEBUG
2024-04-12 20:58:45,473 INFO
2024-04-12 20:58:45,473 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-0 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:58:45,473 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:45,827 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:45,827 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:45,827 DEBUG --- stdout ---
2024-04-12 20:58:45,827 DEBUG true
2024-04-12 20:58:45,827 DEBUG --- stderr ---
2024-04-12 20:58:45,828 DEBUG
2024-04-12 20:58:45,828 INFO
2024-04-12 20:58:45,828 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-0 --output jsonpath={.status.startTime}
2024-04-12 20:58:45,828 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:46,184 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:46,184 DEBUG --- stdout ---
2024-04-12 20:58:46,184 DEBUG 2024-04-12T16:44:43Z
2024-04-12 20:58:46,184 DEBUG --- stderr ---
2024-04-12 20:58:46,184 DEBUG
2024-04-12 20:58:46,185 INFO
2024-04-12 20:58:46,185 INFO ----------- Check pod ds-idrepo-0 filesystem is accessible -----------
2024-04-12 20:58:47,041 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:47,041 INFO
2024-04-12 20:58:47,041 INFO ----------------- Check pod ds-idrepo-0 restart count -----------------
2024-04-12 20:58:47,041 INFO
2024-04-12 20:58:47,041 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-0 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:58:47,041 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:47,416 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:47,417 DEBUG --- stdout ---
2024-04-12 20:58:47,417 DEBUG 0
2024-04-12 20:58:47,417 DEBUG --- stderr ---
2024-04-12 20:58:47,417 DEBUG
2024-04-12 20:58:47,417 INFO Pod ds-idrepo-0 has been restarted 0 times.
2024-04-12 20:58:47,417 INFO
2024-04-12 20:58:47,417 INFO ------------------ Check pod ds-idrepo-1 is running ------------------
2024-04-12 20:58:47,417 INFO
2024-04-12 20:58:47,417 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-1 -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:58:47,417 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:47,789 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:47,789 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:47,789 DEBUG --- stdout ---
2024-04-12 20:58:47,789 DEBUG Running
2024-04-12 20:58:47,789 DEBUG --- stderr ---
2024-04-12 20:58:47,789 DEBUG
2024-04-12 20:58:47,790 INFO
2024-04-12 20:58:47,790 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-1 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:58:47,790 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:48,161 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:48,162 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:48,162 DEBUG --- stdout ---
2024-04-12 20:58:48,162 DEBUG true
2024-04-12 20:58:48,162 DEBUG --- stderr ---
2024-04-12 20:58:48,162 DEBUG
2024-04-12 20:58:48,162 INFO
2024-04-12 20:58:48,162 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-1 --output jsonpath={.status.startTime}
2024-04-12 20:58:48,162 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:48,535 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:48,535 DEBUG --- stdout ---
2024-04-12 20:58:48,535 DEBUG 2024-04-12T16:45:21Z
2024-04-12 20:58:48,535 DEBUG --- stderr ---
2024-04-12 20:58:48,535 DEBUG
2024-04-12 20:58:48,536 INFO
2024-04-12 20:58:48,536 INFO ----------- Check pod ds-idrepo-1 filesystem is accessible -----------
2024-04-12 20:58:49,380 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:49,380 INFO
2024-04-12 20:58:49,380 INFO ----------------- Check pod ds-idrepo-1 restart count -----------------
2024-04-12 20:58:49,380 INFO
2024-04-12 20:58:49,380 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-1 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:58:49,380 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:49,734 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:49,734 DEBUG --- stdout ---
2024-04-12 20:58:49,734 DEBUG 0
2024-04-12 20:58:49,734 DEBUG --- stderr ---
2024-04-12 20:58:49,734 DEBUG
2024-04-12 20:58:49,734 INFO Pod ds-idrepo-1 has been restarted 0 times.
2024-04-12 20:58:49,734 INFO
2024-04-12 20:58:49,734 INFO ------------------ Check pod ds-idrepo-2 is running ------------------
2024-04-12 20:58:49,735 INFO
2024-04-12 20:58:49,735 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:58:49,735 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:50,131 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:50,131 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:50,131 DEBUG --- stdout ---
2024-04-12 20:58:50,131 DEBUG Running
2024-04-12 20:58:50,131 DEBUG --- stderr ---
2024-04-12 20:58:50,131 DEBUG
2024-04-12 20:58:50,132 INFO
2024-04-12 20:58:50,132 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:58:50,132 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:50,503 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:50,503 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:50,503 DEBUG --- stdout ---
2024-04-12 20:58:50,503 DEBUG true
2024-04-12 20:58:50,503 DEBUG --- stderr ---
2024-04-12 20:58:50,503 DEBUG
2024-04-12 20:58:50,504 INFO
2024-04-12 20:58:50,504 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-2 --output jsonpath={.status.startTime}
2024-04-12 20:58:50,504 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:50,879 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:50,879 DEBUG --- stdout ---
2024-04-12 20:58:50,879 DEBUG 2024-04-12T16:45:58Z
2024-04-12 20:58:50,879 DEBUG --- stderr ---
2024-04-12 20:58:50,879 DEBUG
2024-04-12 20:58:50,880 INFO
2024-04-12 20:58:50,880 INFO ----------- Check pod ds-idrepo-2 filesystem is accessible -----------
2024-04-12 20:58:51,763 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:51,763 INFO
2024-04-12 20:58:51,763 INFO ----------------- Check pod ds-idrepo-2 restart count -----------------
2024-04-12 20:58:51,764 INFO
2024-04-12 20:58:51,764 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:58:51,764 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:52,141 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:52,141 DEBUG --- stdout ---
2024-04-12 20:58:52,141 DEBUG 0
2024-04-12 20:58:52,141 DEBUG --- stderr ---
2024-04-12 20:58:52,141 DEBUG
2024-04-12 20:58:52,141 INFO Pod ds-idrepo-2 has been restarted 0 times.
2024-04-12 20:58:52,141 INFO
2024-04-12 20:58:52,141 INFO ****************************** Initializing component pods for AM ******************************
2024-04-12 20:58:52,141 INFO
2024-04-12 20:58:52,141 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:58:52,142 INFO
2024-04-12 20:58:52,142 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:58:52,142 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:52,527 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:52,527 DEBUG --- stdout ---
2024-04-12 20:58:52,527 DEBUG 3
2024-04-12 20:58:52,527 DEBUG --- stderr ---
2024-04-12 20:58:52,527 DEBUG
2024-04-12 20:58:52,527 INFO
2024-04-12 20:58:52,527 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:58:52,527 INFO
2024-04-12 20:58:52,528 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-12 20:58:52,528 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:58:52,964 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-12 20:58:52,964 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:52,964 DEBUG --- stdout ---
2024-04-12 20:58:52,964 DEBUG am-6d96779b5d-pfljh am-6d96779b5d-qbg7l am-6d96779b5d-zn69c
2024-04-12 20:58:52,964 DEBUG --- stderr ---
2024-04-12 20:58:52,964 DEBUG
2024-04-12 20:58:52,964 INFO
2024-04-12 20:58:52,964 INFO -------------- Check pod am-6d96779b5d-pfljh is running --------------
2024-04-12 20:58:52,964 INFO
2024-04-12 20:58:52,964 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-6d96779b5d-pfljh -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:58:52,965 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:53,352 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:53,352 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:53,352 DEBUG --- stdout ---
2024-04-12 20:58:53,352 DEBUG Running
2024-04-12 20:58:53,352 DEBUG --- stderr ---
2024-04-12 20:58:53,352 DEBUG
2024-04-12 20:58:53,353 INFO
2024-04-12 20:58:53,353 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-6d96779b5d-pfljh -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:58:53,353 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:53,720 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:53,720 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:53,720 DEBUG --- stdout ---
2024-04-12 20:58:53,720 DEBUG true
2024-04-12 20:58:53,721 DEBUG --- stderr ---
2024-04-12 20:58:53,721 DEBUG
2024-04-12 20:58:53,721 INFO
2024-04-12 20:58:53,721 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-6d96779b5d-pfljh --output jsonpath={.status.startTime}
2024-04-12 20:58:53,721 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:54,093 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:54,093 DEBUG --- stdout ---
2024-04-12 20:58:54,094 DEBUG 2024-04-12T16:46:39Z
2024-04-12 20:58:54,094 DEBUG --- stderr ---
2024-04-12 20:58:54,094 DEBUG
2024-04-12 20:58:54,094 INFO
2024-04-12 20:58:54,094 INFO ------- Check pod am-6d96779b5d-pfljh filesystem is accessible -------
2024-04-12 20:58:54,918 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:54,919 INFO
2024-04-12 20:58:54,919 INFO ------------- Check pod am-6d96779b5d-pfljh restart count -------------
2024-04-12 20:58:54,919 INFO
2024-04-12 20:58:54,920 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-6d96779b5d-pfljh --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:58:54,920 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:55,275 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:55,275 DEBUG --- stdout ---
2024-04-12 20:58:55,276 DEBUG 0
2024-04-12 20:58:55,276 DEBUG --- stderr ---
2024-04-12 20:58:55,276 DEBUG
2024-04-12 20:58:55,276 INFO Pod am-6d96779b5d-pfljh has been restarted 0 times.
2024-04-12 20:58:55,276 INFO
2024-04-12 20:58:55,276 INFO -------------- Check pod am-6d96779b5d-qbg7l is running --------------
2024-04-12 20:58:55,276 INFO
2024-04-12 20:58:55,276 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-6d96779b5d-qbg7l -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:58:55,276 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:55,631 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:55,631 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:55,631 DEBUG --- stdout ---
2024-04-12 20:58:55,631 DEBUG Running
2024-04-12 20:58:55,631 DEBUG --- stderr ---
2024-04-12 20:58:55,631 DEBUG
2024-04-12 20:58:55,632 INFO
2024-04-12 20:58:55,632 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-6d96779b5d-qbg7l -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:58:55,632 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:55,984 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:55,984 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:55,984 DEBUG --- stdout ---
2024-04-12 20:58:55,984 DEBUG true
2024-04-12 20:58:55,984 DEBUG --- stderr ---
2024-04-12 20:58:55,984 DEBUG
2024-04-12 20:58:55,984 INFO
2024-04-12 20:58:55,984 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-6d96779b5d-qbg7l --output jsonpath={.status.startTime}
2024-04-12 20:58:55,984 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:56,347 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:56,347 DEBUG --- stdout ---
2024-04-12 20:58:56,347 DEBUG 2024-04-12T16:46:38Z
2024-04-12 20:58:56,347 DEBUG --- stderr ---
2024-04-12 20:58:56,347 DEBUG
2024-04-12 20:58:56,348 INFO
2024-04-12 20:58:56,348 INFO ------- Check pod am-6d96779b5d-qbg7l filesystem is accessible -------
2024-04-12 20:58:57,177 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:57,178 INFO
2024-04-12 20:58:57,178 INFO ------------- Check pod am-6d96779b5d-qbg7l restart count -------------
2024-04-12 20:58:57,178 INFO
2024-04-12 20:58:57,178 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-6d96779b5d-qbg7l --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:58:57,178 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:57,551 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:57,551 DEBUG --- stdout ---
2024-04-12 20:58:57,551 DEBUG 0
2024-04-12 20:58:57,552 DEBUG --- stderr ---
2024-04-12 20:58:57,552 DEBUG
2024-04-12 20:58:57,552 INFO Pod am-6d96779b5d-qbg7l has been restarted 0 times.
2024-04-12 20:58:57,552 INFO
2024-04-12 20:58:57,552 INFO -------------- Check pod am-6d96779b5d-zn69c is running --------------
2024-04-12 20:58:57,552 INFO
2024-04-12 20:58:57,552 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-6d96779b5d-zn69c -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:58:57,552 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:57,930 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:57,930 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:57,930 DEBUG --- stdout ---
2024-04-12 20:58:57,930 DEBUG Running
2024-04-12 20:58:57,930 DEBUG --- stderr ---
2024-04-12 20:58:57,930 DEBUG
2024-04-12 20:58:57,931 INFO
2024-04-12 20:58:57,931 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-6d96779b5d-zn69c -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:58:57,931 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:58:58,287 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:58,287 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:58,287 DEBUG --- stdout ---
2024-04-12 20:58:58,287 DEBUG true
2024-04-12 20:58:58,287 DEBUG --- stderr ---
2024-04-12 20:58:58,287 DEBUG
2024-04-12 20:58:58,287 INFO
2024-04-12 20:58:58,287 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-6d96779b5d-zn69c --output jsonpath={.status.startTime}
2024-04-12 20:58:58,287 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:58,646 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:58,647 DEBUG --- stdout ---
2024-04-12 20:58:58,647 DEBUG 2024-04-12T16:46:38Z
2024-04-12 20:58:58,647 DEBUG --- stderr ---
2024-04-12 20:58:58,647 DEBUG
2024-04-12 20:58:58,647 INFO
2024-04-12 20:58:58,647 INFO ------- Check pod am-6d96779b5d-zn69c filesystem is accessible -------
2024-04-12 20:58:59,500 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:58:59,500 INFO
2024-04-12 20:58:59,500 INFO ------------- Check pod am-6d96779b5d-zn69c restart count -------------
2024-04-12 20:58:59,500 INFO
2024-04-12 20:58:59,501 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-6d96779b5d-zn69c --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:58:59,501 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:58:59,881 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:58:59,881 DEBUG --- stdout ---
2024-04-12 20:58:59,881 DEBUG 0
2024-04-12 20:58:59,881 DEBUG --- stderr ---
2024-04-12 20:58:59,881 DEBUG
2024-04-12 20:58:59,881 INFO Pod am-6d96779b5d-zn69c has been restarted 0 times.
2024-04-12 20:58:59,881 INFO
2024-04-12 20:58:59,881 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-12 20:58:59,881 INFO
2024-04-12 20:58:59,881 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:58:59,882 INFO
2024-04-12 20:58:59,882 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm --field-selector status.phase!=Failed get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-12 20:58:59,882 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:00,261 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:00,262 DEBUG --- stdout ---
2024-04-12 20:59:00,262 DEBUG amster-twfcc
2024-04-12 20:59:00,262 DEBUG --- stderr ---
2024-04-12 20:59:00,262 DEBUG
2024-04-12 20:59:00,262 INFO
2024-04-12 20:59:00,262 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-12 20:59:00,262 INFO
2024-04-12 20:59:00,262 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:00,262 INFO
2024-04-12 20:59:00,262 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:00,262 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:00,644 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:00,644 DEBUG --- stdout ---
2024-04-12 20:59:00,644 DEBUG 2
2024-04-12 20:59:00,644 DEBUG --- stderr ---
2024-04-12 20:59:00,644 DEBUG
2024-04-12 20:59:00,644 INFO
2024-04-12 20:59:00,644 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:00,644 INFO
2024-04-12 20:59:00,644 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=idm --output jsonpath={.items[*].metadata.name}` | grep 2
2024-04-12 20:59:00,645 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:01,025 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-12 20:59:01,026 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:01,026 DEBUG --- stdout ---
2024-04-12 20:59:01,026 DEBUG idm-b85f9d978-b559t idm-b85f9d978-dz57p
2024-04-12 20:59:01,026 DEBUG --- stderr ---
2024-04-12 20:59:01,026 DEBUG
2024-04-12 20:59:01,026 INFO
2024-04-12 20:59:01,026 INFO -------------- Check pod idm-b85f9d978-b559t is running --------------
2024-04-12 20:59:01,026 INFO
2024-04-12 20:59:01,026 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-b559t -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:59:01,026 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:01,384 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:01,385 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:01,385 DEBUG --- stdout ---
2024-04-12 20:59:01,385 DEBUG Running
2024-04-12 20:59:01,385 DEBUG --- stderr ---
2024-04-12 20:59:01,385 DEBUG
2024-04-12 20:59:01,385 INFO
2024-04-12 20:59:01,385 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-b559t -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:59:01,385 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:01,737 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:01,737 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:01,737 DEBUG --- stdout ---
2024-04-12 20:59:01,737 DEBUG true
2024-04-12 20:59:01,737 DEBUG --- stderr ---
2024-04-12 20:59:01,737 DEBUG
2024-04-12 20:59:01,737 INFO
2024-04-12 20:59:01,737 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-b559t --output jsonpath={.status.startTime}
2024-04-12 20:59:01,737 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:02,092 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:02,092 DEBUG --- stdout ---
2024-04-12 20:59:02,092 DEBUG 2024-04-12T16:46:39Z
2024-04-12 20:59:02,092 DEBUG --- stderr ---
2024-04-12 20:59:02,092 DEBUG
2024-04-12 20:59:02,093 INFO
2024-04-12 20:59:02,093 INFO ------- Check pod idm-b85f9d978-b559t filesystem is accessible -------
2024-04-12 20:59:02,909 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:02,910 INFO
2024-04-12 20:59:02,910 INFO ------------- Check pod idm-b85f9d978-b559t restart count -------------
2024-04-12 20:59:02,910 INFO
2024-04-12 20:59:02,910 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-b559t --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:59:02,910 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:03,262 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:03,263 DEBUG --- stdout ---
2024-04-12 20:59:03,263 DEBUG 0
2024-04-12 20:59:03,263 DEBUG --- stderr ---
2024-04-12 20:59:03,263 DEBUG
2024-04-12 20:59:03,263 INFO Pod idm-b85f9d978-b559t has been restarted 0 times.
2024-04-12 20:59:03,263 INFO
2024-04-12 20:59:03,263 INFO -------------- Check pod idm-b85f9d978-dz57p is running --------------
2024-04-12 20:59:03,263 INFO
2024-04-12 20:59:03,263 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-dz57p -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:59:03,263 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:03,623 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:03,623 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:03,623 DEBUG --- stdout ---
2024-04-12 20:59:03,623 DEBUG Running
2024-04-12 20:59:03,623 DEBUG --- stderr ---
2024-04-12 20:59:03,623 DEBUG
2024-04-12 20:59:03,623 INFO
2024-04-12 20:59:03,623 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-dz57p -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:59:03,623 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:03,992 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:03,993 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:03,993 DEBUG --- stdout ---
2024-04-12 20:59:03,993 DEBUG true
2024-04-12 20:59:03,993 DEBUG --- stderr ---
2024-04-12 20:59:03,993 DEBUG
2024-04-12 20:59:03,993 INFO
2024-04-12 20:59:03,993 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-dz57p --output jsonpath={.status.startTime}
2024-04-12 20:59:03,993 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:04,367 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:04,367 DEBUG --- stdout ---
2024-04-12 20:59:04,367 DEBUG 2024-04-12T16:46:39Z
2024-04-12 20:59:04,367 DEBUG --- stderr ---
2024-04-12 20:59:04,367 DEBUG
2024-04-12 20:59:04,368 INFO
2024-04-12 20:59:04,368 INFO ------- Check pod idm-b85f9d978-dz57p filesystem is accessible -------
2024-04-12 20:59:05,215 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:05,218 INFO
2024-04-12 20:59:05,218 INFO ------------- Check pod idm-b85f9d978-dz57p restart count -------------
2024-04-12 20:59:05,219 INFO
2024-04-12 20:59:05,219 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-dz57p --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:59:05,219 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:05,580 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:05,581 DEBUG --- stdout ---
2024-04-12 20:59:05,581 DEBUG 0
2024-04-12 20:59:05,581 DEBUG --- stderr ---
2024-04-12 20:59:05,581 DEBUG
2024-04-12 20:59:05,581 INFO Pod idm-b85f9d978-dz57p has been restarted 0 times.
2024-04-12 20:59:05,581 INFO
2024-04-12 20:59:05,581 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-12 20:59:05,581 INFO
2024-04-12 20:59:05,581 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:05,581 INFO
2024-04-12 20:59:05,581 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:05,581 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:05,883 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:05,883 DEBUG --- stdout ---
2024-04-12 20:59:05,883 DEBUG 1
2024-04-12 20:59:05,884 DEBUG --- stderr ---
2024-04-12 20:59:05,884 DEBUG
2024-04-12 20:59:05,884 INFO
2024-04-12 20:59:05,884 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:05,884 INFO
2024-04-12 20:59:05,884 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-12 20:59:05,884 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:06,188 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-12 20:59:06,188 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:06,188 DEBUG --- stdout ---
2024-04-12 20:59:06,188 DEBUG end-user-ui-58878d7668-5gqgr
2024-04-12 20:59:06,188 DEBUG --- stderr ---
2024-04-12 20:59:06,188 DEBUG
2024-04-12 20:59:06,188 INFO
2024-04-12 20:59:06,188 INFO ---------- Check pod end-user-ui-58878d7668-5gqgr is running ----------
2024-04-12 20:59:06,188 INFO
2024-04-12 20:59:06,188 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods end-user-ui-58878d7668-5gqgr -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:59:06,188 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:06,500 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:06,500 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:06,500 DEBUG --- stdout ---
2024-04-12 20:59:06,500 DEBUG Running
2024-04-12 20:59:06,501 DEBUG --- stderr ---
2024-04-12 20:59:06,501 DEBUG
2024-04-12 20:59:06,501 INFO
2024-04-12 20:59:06,501 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods end-user-ui-58878d7668-5gqgr -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:59:06,501 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:06,819 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:06,820 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:06,820 DEBUG --- stdout ---
2024-04-12 20:59:06,820 DEBUG true
2024-04-12 20:59:06,820 DEBUG --- stderr ---
2024-04-12 20:59:06,820 DEBUG
2024-04-12 20:59:06,820 INFO
2024-04-12 20:59:06,821 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod end-user-ui-58878d7668-5gqgr --output jsonpath={.status.startTime}
2024-04-12 20:59:06,821 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:07,138 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:07,138 DEBUG --- stdout ---
2024-04-12 20:59:07,138 DEBUG 2024-04-12T16:47:46Z
2024-04-12 20:59:07,139 DEBUG --- stderr ---
2024-04-12 20:59:07,139 DEBUG
2024-04-12 20:59:07,139 INFO
2024-04-12 20:59:07,139 INFO --- Check pod end-user-ui-58878d7668-5gqgr filesystem is accessible ---
2024-04-12 20:59:07,905 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:07,906 INFO
2024-04-12 20:59:07,906 INFO -------- Check pod end-user-ui-58878d7668-5gqgr restart count --------
2024-04-12 20:59:07,906 INFO
2024-04-12 20:59:07,906 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod end-user-ui-58878d7668-5gqgr --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:59:07,906 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:08,201 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:08,201 DEBUG --- stdout ---
2024-04-12 20:59:08,201 DEBUG 0
2024-04-12 20:59:08,201 DEBUG --- stderr ---
2024-04-12 20:59:08,201 DEBUG
2024-04-12 20:59:08,201 INFO Pod end-user-ui-58878d7668-5gqgr has been restarted 0 times.
2024-04-12 20:59:08,202 INFO
2024-04-12 20:59:08,202 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-12 20:59:08,202 INFO
2024-04-12 20:59:08,202 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:08,202 INFO
2024-04-12 20:59:08,202 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:08,202 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:08,499 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:08,499 DEBUG --- stdout ---
2024-04-12 20:59:08,499 DEBUG 1
2024-04-12 20:59:08,499 DEBUG --- stderr ---
2024-04-12 20:59:08,499 DEBUG
2024-04-12 20:59:08,499 INFO
2024-04-12 20:59:08,499 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:08,500 INFO
2024-04-12 20:59:08,500 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-12 20:59:08,500 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:08,810 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-12 20:59:08,810 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:08,810 DEBUG --- stdout ---
2024-04-12 20:59:08,810 DEBUG login-ui-5fdf49fc66-svvhv
2024-04-12 20:59:08,810 DEBUG --- stderr ---
2024-04-12 20:59:08,810 DEBUG
2024-04-12 20:59:08,810 INFO
2024-04-12 20:59:08,810 INFO ----------- Check pod login-ui-5fdf49fc66-svvhv is running -----------
2024-04-12 20:59:08,810 INFO
2024-04-12 20:59:08,810 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods login-ui-5fdf49fc66-svvhv -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:59:08,810 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:09,128 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:09,128 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:09,128 DEBUG --- stdout ---
2024-04-12 20:59:09,129 DEBUG Running
2024-04-12 20:59:09,129 DEBUG --- stderr ---
2024-04-12 20:59:09,129 DEBUG
2024-04-12 20:59:09,129 INFO
2024-04-12 20:59:09,129 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods login-ui-5fdf49fc66-svvhv -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:59:09,129 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:09,428 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:09,428 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:09,428 DEBUG --- stdout ---
2024-04-12 20:59:09,428 DEBUG true
2024-04-12 20:59:09,429 DEBUG --- stderr ---
2024-04-12 20:59:09,429 DEBUG
2024-04-12 20:59:09,429 INFO
2024-04-12 20:59:09,429 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod login-ui-5fdf49fc66-svvhv --output jsonpath={.status.startTime}
2024-04-12 20:59:09,429 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:09,724 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:09,725 DEBUG --- stdout ---
2024-04-12 20:59:09,725 DEBUG 2024-04-12T16:47:47Z
2024-04-12 20:59:09,725 DEBUG --- stderr ---
2024-04-12 20:59:09,725 DEBUG
2024-04-12 20:59:09,725 INFO
2024-04-12 20:59:09,726 INFO ---- Check pod login-ui-5fdf49fc66-svvhv filesystem is accessible ----
2024-04-12 20:59:10,502 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:10,502 INFO
2024-04-12 20:59:10,507 INFO ---------- Check pod login-ui-5fdf49fc66-svvhv restart count ----------
2024-04-12 20:59:10,508 INFO
2024-04-12 20:59:10,508 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod login-ui-5fdf49fc66-svvhv --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:59:10,508 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:10,817 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:10,817 DEBUG --- stdout ---
2024-04-12 20:59:10,817 DEBUG 0
2024-04-12 20:59:10,817 DEBUG --- stderr ---
2024-04-12 20:59:10,817 DEBUG
2024-04-12 20:59:10,817 INFO Pod login-ui-5fdf49fc66-svvhv has been restarted 0 times.
2024-04-12 20:59:10,818 INFO
2024-04-12 20:59:10,818 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-12 20:59:10,818 INFO
2024-04-12 20:59:10,818 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:10,818 INFO
2024-04-12 20:59:10,818 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:10,818 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:11,121 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:11,121 DEBUG --- stdout ---
2024-04-12 20:59:11,121 DEBUG 1
2024-04-12 20:59:11,121 DEBUG --- stderr ---
2024-04-12 20:59:11,121 DEBUG
2024-04-12 20:59:11,121 INFO
2024-04-12 20:59:11,121 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:11,121 INFO
2024-04-12 20:59:11,121 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-12 20:59:11,122 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:11,425 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-12 20:59:11,425 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:11,425 DEBUG --- stdout ---
2024-04-12 20:59:11,425 DEBUG admin-ui-c4c7bf584-vlxw7
2024-04-12 20:59:11,425 DEBUG --- stderr ---
2024-04-12 20:59:11,425 DEBUG
2024-04-12 20:59:11,425 INFO
2024-04-12 20:59:11,425 INFO ------------ Check pod admin-ui-c4c7bf584-vlxw7 is running ------------
2024-04-12 20:59:11,425 INFO
2024-04-12 20:59:11,425 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods admin-ui-c4c7bf584-vlxw7 -o=jsonpath={.status.phase} | grep "Running"
2024-04-12 20:59:11,425 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:11,743 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:11,744 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:11,744 DEBUG --- stdout ---
2024-04-12 20:59:11,744 DEBUG Running
2024-04-12 20:59:11,744 DEBUG --- stderr ---
2024-04-12 20:59:11,744 DEBUG
2024-04-12 20:59:11,744 INFO
2024-04-12 20:59:11,744 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods admin-ui-c4c7bf584-vlxw7 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-12 20:59:11,744 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-12 20:59:12,046 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:12,046 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:12,046 DEBUG --- stdout ---
2024-04-12 20:59:12,046 DEBUG true
2024-04-12 20:59:12,046 DEBUG --- stderr ---
2024-04-12 20:59:12,047 DEBUG
2024-04-12 20:59:12,047 INFO
2024-04-12 20:59:12,047 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod admin-ui-c4c7bf584-vlxw7 --output jsonpath={.status.startTime}
2024-04-12 20:59:12,047 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:12,343 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:12,343 DEBUG --- stdout ---
2024-04-12 20:59:12,343 DEBUG 2024-04-12T16:47:46Z
2024-04-12 20:59:12,344 DEBUG --- stderr ---
2024-04-12 20:59:12,344 DEBUG
2024-04-12 20:59:12,344 INFO
2024-04-12 20:59:12,344 INFO ----- Check pod admin-ui-c4c7bf584-vlxw7 filesystem is accessible -----
2024-04-12 20:59:13,103 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:13,104 INFO
2024-04-12 20:59:13,104 INFO ---------- Check pod admin-ui-c4c7bf584-vlxw7 restart count ----------
2024-04-12 20:59:13,104 INFO
2024-04-12 20:59:13,104 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod admin-ui-c4c7bf584-vlxw7 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-12 20:59:13,104 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:13,420 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:13,421 DEBUG --- stdout ---
2024-04-12 20:59:13,421 DEBUG 0
2024-04-12 20:59:13,421 DEBUG --- stderr ---
2024-04-12 20:59:13,421 DEBUG
2024-04-12 20:59:13,421 INFO Pod admin-ui-c4c7bf584-vlxw7 has been restarted 0 times.
2024-04-12 20:59:13,421 INFO
2024-04-12 20:59:13,421 INFO ***************************** Checking DS-CTS component is running *****************************
2024-04-12 20:59:13,421 INFO
2024-04-12 20:59:13,421 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:13,421 INFO
2024-04-12 20:59:13,421 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:13,422 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:13,805 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:13,805 DEBUG --- stdout ---
2024-04-12 20:59:13,805 DEBUG 3
2024-04-12 20:59:13,805 DEBUG --- stderr ---
2024-04-12 20:59:13,805 DEBUG
2024-04-12 20:59:13,805 INFO
2024-04-12 20:59:13,805 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-12 20:59:13,806 INFO
2024-04-12 20:59:13,806 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-cts --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3"
2024-04-12 20:59:13,806 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-12 20:59:14,188 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:14,189 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:14,189 DEBUG --- stdout ---
2024-04-12 20:59:14,189 DEBUG current:3 ready:3 replicas:3
2024-04-12 20:59:14,189 DEBUG --- stderr ---
2024-04-12 20:59:14,189 DEBUG
2024-04-12 20:59:14,189 INFO
2024-04-12 20:59:14,189 INFO *************************** Checking DS-IDREPO component is running ***************************
2024-04-12 20:59:14,189 INFO
2024-04-12 20:59:14,189 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:14,189 INFO
2024-04-12 20:59:14,189 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:14,189 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:14,555 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:14,555 DEBUG --- stdout ---
2024-04-12 20:59:14,555 DEBUG 3
2024-04-12 20:59:14,555 DEBUG --- stderr ---
2024-04-12 20:59:14,555 DEBUG
2024-04-12 20:59:14,556 INFO
2024-04-12 20:59:14,556 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-12 20:59:14,556 INFO
2024-04-12 20:59:14,556 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-idrepo --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3"
2024-04-12 20:59:14,556 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-12 20:59:14,974 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:14,974 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:14,974 DEBUG --- stdout ---
2024-04-12 20:59:14,975 DEBUG current:3 ready:3 replicas:3
2024-04-12 20:59:14,975 DEBUG --- stderr ---
2024-04-12 20:59:14,975 DEBUG
2024-04-12 20:59:14,975 INFO
2024-04-12 20:59:14,975 INFO ******************************* Checking AM component is running *******************************
2024-04-12 20:59:14,975 INFO
2024-04-12 20:59:14,975 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:14,975 INFO
2024-04-12 20:59:14,975 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:14,975 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:15,363 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:15,363 DEBUG --- stdout ---
2024-04-12 20:59:15,364 DEBUG 3
2024-04-12 20:59:15,364 DEBUG --- stderr ---
2024-04-12 20:59:15,364 DEBUG
2024-04-12 20:59:15,364 INFO
2024-04-12 20:59:15,364 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-12 20:59:15,364 INFO
2024-04-12 20:59:15,364 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments am --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:3 replicas:3"
2024-04-12 20:59:15,364 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-12 20:59:15,766 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:15,767 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:15,768 DEBUG --- stdout ---
2024-04-12 20:59:15,768 DEBUG ready:3 replicas:3
2024-04-12 20:59:15,768 DEBUG --- stderr ---
2024-04-12 20:59:15,768 DEBUG
2024-04-12 20:59:15,768 INFO
2024-04-12 20:59:15,768 INFO ***************************** Checking AMSTER component is running *****************************
2024-04-12 20:59:15,768 INFO
2024-04-12 20:59:15,768 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:15,768 INFO
2024-04-12 20:59:15,768 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-12 20:59:15,769 INFO
2024-04-12 20:59:15,769 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get jobs amster --output jsonpath="{.status.succeeded}" | grep "1"
2024-04-12 20:59:15,769 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-12 20:59:16,144 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:16,144 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:16,144 DEBUG --- stdout ---
2024-04-12 20:59:16,144 DEBUG 1
2024-04-12 20:59:16,144 DEBUG --- stderr ---
2024-04-12 20:59:16,144 DEBUG
2024-04-12 20:59:16,144 INFO
2024-04-12 20:59:16,144 INFO ****************************** Checking IDM component is running ******************************
2024-04-12 20:59:16,144 INFO
2024-04-12 20:59:16,144 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:16,145 INFO
2024-04-12 20:59:16,145 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:16,145 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:16,523 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:16,523 DEBUG --- stdout ---
2024-04-12 20:59:16,523 DEBUG 2
2024-04-12 20:59:16,523 DEBUG --- stderr ---
2024-04-12 20:59:16,523 DEBUG
2024-04-12 20:59:16,524 INFO
2024-04-12 20:59:16,524 INFO -------------- Waiting for 2 expected pod(s) to be ready --------------
2024-04-12 20:59:16,524 INFO
2024-04-12 20:59:16,524 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment idm --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:2 replicas:2"
2024-04-12 20:59:16,524 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-12 20:59:16,898 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:16,898 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:16,898 DEBUG --- stdout ---
2024-04-12 20:59:16,898 DEBUG ready:2 replicas:2
2024-04-12 20:59:16,898 DEBUG --- stderr ---
2024-04-12 20:59:16,898 DEBUG
2024-04-12 20:59:16,899 INFO
2024-04-12 20:59:16,899 INFO ************************** Checking END-USER-UI component is running **************************
2024-04-12 20:59:16,899 INFO
2024-04-12 20:59:16,899 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:16,899 INFO
2024-04-12 20:59:16,899 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:16,899 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:17,218 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:17,218 DEBUG --- stdout ---
2024-04-12 20:59:17,218 DEBUG 1
2024-04-12 20:59:17,219 DEBUG --- stderr ---
2024-04-12 20:59:17,219 DEBUG
2024-04-12 20:59:17,219 INFO
2024-04-12 20:59:17,219 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-12 20:59:17,219 INFO
2024-04-12 20:59:17,219 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments end-user-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-12 20:59:17,219 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-12 20:59:17,552 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:17,553 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:17,553 DEBUG --- stdout ---
2024-04-12 20:59:17,553 DEBUG ready:1 replicas:1
2024-04-12 20:59:17,553 DEBUG --- stderr ---
2024-04-12 20:59:17,553 DEBUG
2024-04-12 20:59:17,553 INFO
2024-04-12 20:59:17,553 INFO **************************** Checking LOGIN-UI component is running ****************************
2024-04-12 20:59:17,553 INFO
2024-04-12 20:59:17,553 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:17,553 INFO
2024-04-12 20:59:17,553 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:17,553 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:17,870 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:17,870 DEBUG --- stdout ---
2024-04-12 20:59:17,870 DEBUG 1
2024-04-12 20:59:17,870 DEBUG --- stderr ---
2024-04-12 20:59:17,870 DEBUG
2024-04-12 20:59:17,870 INFO
2024-04-12 20:59:17,870 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-12 20:59:17,871 INFO
2024-04-12 20:59:17,871 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments login-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-12 20:59:17,871 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-12 20:59:18,234 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:18,235 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:18,235 DEBUG --- stdout ---
2024-04-12 20:59:18,235 DEBUG ready:1 replicas:1
2024-04-12 20:59:18,235 DEBUG --- stderr ---
2024-04-12 20:59:18,235 DEBUG
2024-04-12 20:59:18,235 INFO
2024-04-12 20:59:18,235 INFO **************************** Checking ADMIN-UI component is running ****************************
2024-04-12 20:59:18,235 INFO
2024-04-12 20:59:18,235 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:18,235 INFO
2024-04-12 20:59:18,235 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:18,235 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:18,548 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:18,549 DEBUG --- stdout ---
2024-04-12 20:59:18,549 DEBUG 1
2024-04-12 20:59:18,549 DEBUG --- stderr ---
2024-04-12 20:59:18,549 DEBUG
2024-04-12 20:59:18,549 INFO
2024-04-12 20:59:18,549 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-12 20:59:18,549 INFO
2024-04-12 20:59:18,549 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments admin-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-12 20:59:18,549 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-12 20:59:18,898 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-12 20:59:18,899 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:18,899 DEBUG --- stdout ---
2024-04-12 20:59:18,899 DEBUG ready:1 replicas:1
2024-04-12 20:59:18,899 DEBUG --- stderr ---
2024-04-12 20:59:18,899 DEBUG
2024-04-12 20:59:18,899 INFO
2024-04-12 20:59:18,899 INFO ****************************** Livecheck stage: After deployment ******************************
2024-04-12 20:59:18,899 INFO
2024-04-12 20:59:18,899 INFO ---------------------- Running DS-CTS livecheck ----------------------
2024-04-12 20:59:18,899 INFO Livecheck to ds-cts-0
2024-04-12 20:59:18,900 INFO
2024-04-12 20:59:18,900 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.dirmanager\.pw}"
2024-04-12 20:59:18,900 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-12 20:59:19,234 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:19,234 DEBUG --- stdout ---
2024-04-12 20:59:19,234 DEBUG aTFZMWdHVUhYNGtueWxwanVpT3Z4MXZCazZRNnNKUjc=
2024-04-12 20:59:19,235 DEBUG --- stderr ---
2024-04-12 20:59:19,235 DEBUG
2024-04-12 20:59:19,235 INFO
2024-04-12 20:59:19,235 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-12 20:59:21,099 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:59:21,101 DEBUG --- stdout ---
2024-04-12 20:59:21,101 DEBUG dn:
alive: true
2024-04-12 20:59:21,101 DEBUG --- stderr ---
2024-04-12 20:59:21,101 DEBUG
2024-04-12 20:59:21,101 INFO
2024-04-12 20:59:21,102 INFO Livecheck to ds-cts-1
2024-04-12 20:59:21,102 INFO
2024-04-12 20:59:21,102 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-12 20:59:23,045 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:59:23,045 DEBUG --- stdout ---
2024-04-12 20:59:23,046 DEBUG dn:
alive: true
2024-04-12 20:59:23,046 DEBUG --- stderr ---
2024-04-12 20:59:23,046 DEBUG
2024-04-12 20:59:23,046 INFO
2024-04-12 20:59:23,046 INFO Livecheck to ds-cts-2
2024-04-12 20:59:23,046 INFO
2024-04-12 20:59:23,046 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-12 20:59:24,867 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:59:24,867 DEBUG --- stdout ---
2024-04-12 20:59:24,867 DEBUG dn:
alive: true
2024-04-12 20:59:24,867 DEBUG --- stderr ---
2024-04-12 20:59:24,867 DEBUG
2024-04-12 20:59:24,867 INFO
2024-04-12 20:59:24,867 INFO
2024-04-12 20:59:24,867 INFO --------------------- Running DS-IDREPO livecheck ---------------------
2024-04-12 20:59:24,867 INFO Livecheck to ds-idrepo-0
2024-04-12 20:59:24,868 INFO
2024-04-12 20:59:24,868 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.dirmanager\.pw}"
2024-04-12 20:59:24,868 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-12 20:59:25,206 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:25,206 DEBUG --- stdout ---
2024-04-12 20:59:25,206 DEBUG aTFZMWdHVUhYNGtueWxwanVpT3Z4MXZCazZRNnNKUjc=
2024-04-12 20:59:25,206 DEBUG --- stderr ---
2024-04-12 20:59:25,206 DEBUG
2024-04-12 20:59:25,206 INFO
2024-04-12 20:59:25,207 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-12 20:59:27,012 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:59:27,012 DEBUG --- stdout ---
2024-04-12 20:59:27,012 DEBUG dn:
alive: true
2024-04-12 20:59:27,012 DEBUG --- stderr ---
2024-04-12 20:59:27,012 DEBUG
2024-04-12 20:59:27,012 INFO
2024-04-12 20:59:27,012 INFO Livecheck to ds-idrepo-1
2024-04-12 20:59:27,012 INFO
2024-04-12 20:59:27,013 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-12 20:59:28,824 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:59:28,824 DEBUG --- stdout ---
2024-04-12 20:59:28,824 DEBUG dn:
alive: true
2024-04-12 20:59:28,825 DEBUG --- stderr ---
2024-04-12 20:59:28,825 DEBUG
2024-04-12 20:59:28,825 INFO
2024-04-12 20:59:28,825 INFO Livecheck to ds-idrepo-2
2024-04-12 20:59:28,825 INFO
2024-04-12 20:59:28,826 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-12 20:59:30,720 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-12 20:59:30,720 DEBUG --- stdout ---
2024-04-12 20:59:30,720 DEBUG dn:
alive: true
2024-04-12 20:59:30,720 DEBUG --- stderr ---
2024-04-12 20:59:30,720 DEBUG
2024-04-12 20:59:30,721 INFO
2024-04-12 20:59:30,721 INFO
2024-04-12 20:59:30,721 INFO ------------------------ Running AM livecheck ------------------------
2024-04-12 20:59:30,721 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready
2024-04-12 20:59:30,722 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready"
2024-04-12 20:59:31,006 INFO [http_cmd]: http status code OK
2024-04-12 20:59:31,006 DEBUG --- status code ---
2024-04-12 20:59:31,006 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:31,006 DEBUG --- http response ---
2024-04-12 20:59:31,006 DEBUG
2024-04-12 20:59:31,006 INFO
2024-04-12 20:59:31,007 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.AM_PASSWORDS_AMADMIN_CLEAR}"
2024-04-12 20:59:31,007 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-12 20:59:31,370 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:31,370 DEBUG --- stdout ---
2024-04-12 20:59:31,370 DEBUG UjFiV0xNcHpYN2VCeFpveFJYanQwQ1l6
2024-04-12 20:59:31,370 DEBUG --- stderr ---
2024-04-12 20:59:31,370 DEBUG
2024-04-12 20:59:31,370 INFO Authenticate user amadmin via REST
2024-04-12 20:59:31,371 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/"
2024-04-12 20:59:31,593 INFO [http_cmd]: http status code OK
2024-04-12 20:59:31,593 DEBUG --- status code ---
2024-04-12 20:59:31,593 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:31,593 DEBUG --- http response ---
2024-04-12 20:59:31,594 DEBUG {
"tokenId": "Y0p99_Sr1dXPmZbokeYYfUZTNQw.*AAJTSQACMDIAAlNLABx5U0taZW5yUloxTHp2YlJCR1ByckpheXR5RWs9AAR0eXBlAANDVFMAAlMxAAIwMQ..*",
"successUrl": "/am/console",
"realm": "/"
}
2024-04-12 20:59:31,596 INFO
2024-04-12 20:59:31,597 INFO ---------------------- Running AMSTER livecheck ----------------------
2024-04-12 20:59:31,597 INFO
2024-04-12 20:59:31,597 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:31,597 INFO
2024-04-12 20:59:31,597 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm --field-selector status.phase!=Failed get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-12 20:59:31,597 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:31,954 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:31,955 DEBUG --- stdout ---
2024-04-12 20:59:31,955 DEBUG amster-twfcc
2024-04-12 20:59:31,955 DEBUG --- stderr ---
2024-04-12 20:59:31,955 DEBUG
2024-04-12 20:59:32,574 INFO Amster livecheck is passed
2024-04-12 20:59:32,575 INFO
2024-04-12 20:59:32,575 INFO ------------------------ Running IDM livecheck ------------------------
2024-04-12 20:59:32,576 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/ping
2024-04-12 20:59:32,576 INFO
2024-04-12 20:59:32,576 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.OPENIDM_ADMIN_PASSWORD}"
2024-04-12 20:59:32,576 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-12 20:59:32,911 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:32,912 DEBUG --- stdout ---
2024-04-12 20:59:32,912 DEBUG c3MyN000Wld5ckFDYmJ0YThnODlLZ3BN
2024-04-12 20:59:32,912 DEBUG --- stderr ---
2024-04-12 20:59:32,912 DEBUG
2024-04-12 20:59:32,912 INFO Set admin password: ss27M4ZWyrACbbta8g89KgpM
2024-04-12 20:59:32,913 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/ping"
2024-04-12 20:59:33,129 INFO [http_cmd]: http status code OK
2024-04-12 20:59:33,129 DEBUG --- status code ---
2024-04-12 20:59:33,130 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:33,130 DEBUG --- http response ---
2024-04-12 20:59:33,130 DEBUG {
"_id": "",
"_rev": "",
"shortDesc": "OpenIDM ready",
"state": "ACTIVE_READY"
}
2024-04-12 20:59:33,130 INFO Create new LoginSession for user "amadmin"
2024-04-12 20:59:33,130 INFO [LoginSession] Get "access_token" token for user "amadmin"
2024-04-12 20:59:33,130 INFO [LoginSession] Obtaining new "access_token" token from server because it was not obtained yet for the user
2024-04-12 20:59:33,130 INFO [LoginSession] Get "token_id" token for user "amadmin"
2024-04-12 20:59:33,130 INFO [LoginSession] Obtaining new "token_id" token from server because it was not obtained yet for the user
2024-04-12 20:59:33,130 INFO Authenticate user amadmin via REST
2024-04-12 20:59:33,130 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/"
2024-04-12 20:59:33,349 INFO [http_cmd]: http status code OK
2024-04-12 20:59:33,349 DEBUG --- status code ---
2024-04-12 20:59:33,349 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:33,349 DEBUG --- http response ---
2024-04-12 20:59:33,349 DEBUG {
"tokenId": "90BftDERsSy96T4-2IXcCvIyEbg.*AAJTSQACMDIAAlNLABw3RXpyME9oN2hhVGxtUUxDYU5sYXpNV3BHVGs9AAR0eXBlAANDVFMAAlMxAAIwMQ..*",
"successUrl": "/am/console",
"realm": "/"
}
2024-04-12 20:59:33,349 INFO [LoginSession] Obtaining session info
2024-04-12 20:59:33,350 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/realms/root/sessions?_action=getSessionInfo"
2024-04-12 20:59:33,573 INFO [http_cmd]: http status code OK
2024-04-12 20:59:33,573 DEBUG --- status code ---
2024-04-12 20:59:33,573 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:33,573 DEBUG --- http response ---
2024-04-12 20:59:33,573 DEBUG {
"username": "amadmin",
"universalId": "id=amadmin,ou=user,ou=am-config",
"realm": "/",
"latestAccessTime": "2024-04-12T20:59:33Z",
"maxIdleExpirationTime": "2024-04-12T21:02:33Z",
"maxSessionExpirationTime": "2024-04-12T21:02:32Z",
"properties": {
"AMCtxId": "c16f296b-b694-4b11-b446-c2d931141f16-142119139"
}
}
2024-04-12 20:59:33,577 INFO Obtain Oauth2 authz code via REST
2024-04-12 20:59:33,577 DEBUG Oauth2 authorize for access token
2024-04-12 20:59:33,577 INFO [http_cmd]: curl --request POST --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=90BftDERsSy96T4-2IXcCvIyEbg.*AAJTSQACMDIAAlNLABw3RXpyME9oN2hhVGxtUUxDYU5sYXpNV3BHVGs9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" --cookie "route=1712955574.307.38272.642183|f60edb382037eb2df1e800d563ad78a7" --data '{"decision": "Allow", "csrf": "90BftDERsSy96T4-2IXcCvIyEbg.*AAJTSQACMDIAAlNLABw3RXpyME9oN2hhVGxtUUxDYU5sYXpNV3BHVGs9AAR0eXBlAANDVFMAAlMxAAIwMQ..*"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/am/oauth2/authorize?client_id=smokeclient&scope=fr:idm:*&redirect_uri=https://fake.com&response_type=code&realm=/"
2024-04-12 20:59:33,837 INFO [http_cmd]: http status code OK
2024-04-12 20:59:33,837 DEBUG --- status code ---
2024-04-12 20:59:33,837 DEBUG http status code is 302 (expected 302)
2024-04-12 20:59:33,837 DEBUG --- http response ---
2024-04-12 20:59:33,837 DEBUG {'Date': 'Fri, 12 Apr 2024 20:59:33 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store', 'Location': 'https://fake.com?code=Omg-0LKfQYmwOSK-8ifBLSRqut0&iss=https%3A%2F%2Fxlou.iam.xlou-cdm.engineeringpit.com%2Fam%2Foauth2&client_id=smokeclient', 'Pragma': 'no-cache', 'Set-Cookie': 'OAUTH_REQUEST_ATTRIBUTES=DELETED; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Path=/; Secure; HttpOnly; SameSite=none', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains'}
2024-04-12 20:59:33,837 DEBUG Oauth2 exchange authorize code for access token
2024-04-12 20:59:33,837 INFO [http_cmd]: curl -L --request POST --data '{"grant_type": "authorization_code", "redirect_uri": "https://fake.com", "client_id": "smokeclient", "code": "Omg-0LKfQYmwOSK-8ifBLSRqut0"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/am/oauth2/access_token?realm=/"
2024-04-12 20:59:33,923 INFO [http_cmd]: http status code OK
2024-04-12 20:59:33,923 DEBUG --- status code ---
2024-04-12 20:59:33,923 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:33,923 DEBUG --- http response ---
2024-04-12 20:59:33,923 DEBUG {
"access_token": "uizMinAoqCGUW4l4BNhaJz3ajh8",
"scope": "fr:idm:*",
"token_type": "Bearer",
"expires_in": 3599
}
2024-04-12 20:59:33,925 INFO Additional check to sync/queue to ensure sync with DS is ok (see LODESTAR-1245)
2024-04-12 20:59:33,925 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/sync/queue?_queryFilter=true&_pageSize=1"
2024-04-12 20:59:34,150 INFO [http_cmd]: http status code OK
2024-04-12 20:59:34,150 DEBUG --- status code ---
2024-04-12 20:59:34,150 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:34,150 DEBUG --- http response ---
2024-04-12 20:59:34,151 DEBUG {
"result": [],
"resultCount": 0,
"pagedResultsCookie": null,
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}
2024-04-12 20:59:34,152 INFO Try to create, query and delete a user
2024-04-12 20:59:34,153 INFO Create user idm_nt5x9turjf in IDM
2024-04-12 20:59:34,153 INFO [http_cmd]: curl --header "Authorization: Bearer uizMinAoqCGUW4l4BNhaJz3ajh8" --header "Content-Type: application/json" --insecure -L --request POST --data '{"userName": "idm_nt5x9turjf", "givenName": "Peter", "sn": "Parker", "password": "T35tr0ck123", "telephoneNumber": "6669876987", "description": "IDM Test User", "roles": [], "mail": "idm_nt5x9turjf@forgerock.com"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user?_action=create"
2024-04-12 20:59:34,421 INFO [http_cmd]: http status code OK
2024-04-12 20:59:34,421 DEBUG --- status code ---
2024-04-12 20:59:34,421 DEBUG http status code is 201 (expected [200, 201])
2024-04-12 20:59:34,421 DEBUG --- http response ---
2024-04-12 20:59:34,422 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-12 20:59:34,422 DEBUG ----- output -----
2024-04-12 20:59:34,422 DEBUG {
2024-04-12 20:59:34,422 DEBUG "_id": "c8fde50f-eb5c-44e1-a2d5-04e76b405506",
2024-04-12 20:59:34,422 DEBUG "_rev": "c98c90e0-633d-4fe0-9250-a9e8b604eec0-1006100",
2024-04-12 20:59:34,422 DEBUG "country": null,
2024-04-12 20:59:34,422 DEBUG "mail": "idm_nt5x9turjf@forgerock.com",
2024-04-12 20:59:34,422 DEBUG "memberOfOrgIDs": [],
2024-04-12 20:59:34,422 DEBUG "postalCode": null,
2024-04-12 20:59:34,422 DEBUG "profileImage": null,
2024-04-12 20:59:34,422 DEBUG "expireAccount": null,
2024-04-12 20:59:34,422 DEBUG "activeDate": null,
2024-04-12 20:59:34,422 DEBUG
2024-04-12 20:59:34,422 DEBUG
2024-04-12 20:59:34,423 DEBUG
2024-04-12 20:59:34,423 DEBUG [...]
2024-04-12 20:59:34,423 DEBUG
2024-04-12 20:59:34,423 DEBUG
2024-04-12 20:59:34,423 DEBUG
2024-04-12 20:59:34,423 DEBUG "aliasList": [],
2024-04-12 20:59:34,423 DEBUG "kbaInfo": [],
2024-04-12 20:59:34,423 DEBUG "inactiveDate": null,
2024-04-12 20:59:34,423 DEBUG "sn": "Parker",
2024-04-12 20:59:34,423 DEBUG "preferences": null,
2024-04-12 20:59:34,423 DEBUG "userName": "idm_nt5x9turjf",
2024-04-12 20:59:34,423 DEBUG "passwordLastChangedTime": "2024-04-12T20:59:34.231Z",
2024-04-12 20:59:34,423 DEBUG "effectiveRoles": [],
2024-04-12 20:59:34,424 DEBUG "activateAccount": null
2024-04-12 20:59:34,424 DEBUG }
2024-04-12 20:59:34,424 INFO Query IDM to verify UserID c8fde50f-eb5c-44e1-a2d5-04e76b405506 is visible
2024-04-12 20:59:34,424 INFO [http_cmd]: curl --insecure -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user/c8fde50f-eb5c-44e1-a2d5-04e76b405506"
2024-04-12 20:59:34,638 INFO [http_cmd]: http status code OK
2024-04-12 20:59:34,638 DEBUG --- status code ---
2024-04-12 20:59:34,638 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:34,638 DEBUG --- http response ---
2024-04-12 20:59:34,638 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-12 20:59:34,638 DEBUG ----- output -----
2024-04-12 20:59:34,638 DEBUG {
2024-04-12 20:59:34,638 DEBUG "_id": "c8fde50f-eb5c-44e1-a2d5-04e76b405506",
2024-04-12 20:59:34,638 DEBUG "_rev": "c98c90e0-633d-4fe0-9250-a9e8b604eec0-1006100",
2024-04-12 20:59:34,638 DEBUG "country": null,
2024-04-12 20:59:34,638 DEBUG "mail": "idm_nt5x9turjf@forgerock.com",
2024-04-12 20:59:34,638 DEBUG "memberOfOrgIDs": [],
2024-04-12 20:59:34,638 DEBUG "postalCode": null,
2024-04-12 20:59:34,638 DEBUG "profileImage": null,
2024-04-12 20:59:34,638 DEBUG "expireAccount": null,
2024-04-12 20:59:34,638 DEBUG "activeDate": null,
2024-04-12 20:59:34,638 DEBUG
2024-04-12 20:59:34,638 DEBUG
2024-04-12 20:59:34,638 DEBUG
2024-04-12 20:59:34,639 DEBUG [...]
2024-04-12 20:59:34,639 DEBUG
2024-04-12 20:59:34,639 DEBUG
2024-04-12 20:59:34,639 DEBUG
2024-04-12 20:59:34,639 DEBUG "aliasList": [],
2024-04-12 20:59:34,639 DEBUG "kbaInfo": [],
2024-04-12 20:59:34,639 DEBUG "inactiveDate": null,
2024-04-12 20:59:34,639 DEBUG "sn": "Parker",
2024-04-12 20:59:34,639 DEBUG "preferences": null,
2024-04-12 20:59:34,639 DEBUG "userName": "idm_nt5x9turjf",
2024-04-12 20:59:34,639 DEBUG "passwordLastChangedTime": "2024-04-12T20:59:34.231Z",
2024-04-12 20:59:34,639 DEBUG "effectiveRoles": [],
2024-04-12 20:59:34,639 DEBUG "activateAccount": null
2024-04-12 20:59:34,639 DEBUG }
2024-04-12 20:59:34,639 INFO Authenticate as user idm_nt5x9turjf
2024-04-12 20:59:34,640 INFO [http_cmd]: curl --insecure -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/authentication?_action=login"
2024-04-12 20:59:34,848 INFO [http_cmd]: http status code OK
2024-04-12 20:59:34,848 DEBUG --- status code ---
2024-04-12 20:59:34,848 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:34,848 DEBUG --- http response ---
2024-04-12 20:59:34,849 DEBUG {
"_id": "login",
"authorization": {
"id": "anonymous",
"roles": [
"internal/role/openidm-reg"
],
"component": "internal/user"
},
"authenticationId": "anonymous"
}
2024-04-12 20:59:34,851 INFO [http_cmd]: curl --insecure -L --request DELETE "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user/c8fde50f-eb5c-44e1-a2d5-04e76b405506"
2024-04-12 20:59:35,081 INFO [http_cmd]: http status code OK
2024-04-12 20:59:35,081 DEBUG --- status code ---
2024-04-12 20:59:35,081 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:35,081 DEBUG --- http response ---
2024-04-12 20:59:35,082 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-12 20:59:35,082 DEBUG ----- output -----
2024-04-12 20:59:35,082 DEBUG {
2024-04-12 20:59:35,082 DEBUG "_id": "c8fde50f-eb5c-44e1-a2d5-04e76b405506",
2024-04-12 20:59:35,082 DEBUG "_rev": "c98c90e0-633d-4fe0-9250-a9e8b604eec0-1006100",
2024-04-12 20:59:35,082 DEBUG "country": null,
2024-04-12 20:59:35,082 DEBUG "mail": "idm_nt5x9turjf@forgerock.com",
2024-04-12 20:59:35,082 DEBUG "memberOfOrgIDs": [],
2024-04-12 20:59:35,082 DEBUG "postalCode": null,
2024-04-12 20:59:35,082 DEBUG "profileImage": null,
2024-04-12 20:59:35,082 DEBUG "expireAccount": null,
2024-04-12 20:59:35,082 DEBUG "activeDate": null,
2024-04-12 20:59:35,082 DEBUG
2024-04-12 20:59:35,082 DEBUG
2024-04-12 20:59:35,082 DEBUG
2024-04-12 20:59:35,082 DEBUG [...]
2024-04-12 20:59:35,082 DEBUG
2024-04-12 20:59:35,082 DEBUG
2024-04-12 20:59:35,082 DEBUG
2024-04-12 20:59:35,082 DEBUG "aliasList": [],
2024-04-12 20:59:35,082 DEBUG "kbaInfo": [],
2024-04-12 20:59:35,082 DEBUG "inactiveDate": null,
2024-04-12 20:59:35,082 DEBUG "sn": "Parker",
2024-04-12 20:59:35,082 DEBUG "preferences": null,
2024-04-12 20:59:35,082 DEBUG "userName": "idm_nt5x9turjf",
2024-04-12 20:59:35,082 DEBUG "passwordLastChangedTime": "2024-04-12T20:59:34.231Z",
2024-04-12 20:59:35,082 DEBUG "effectiveRoles": [],
2024-04-12 20:59:35,082 DEBUG "activateAccount": null
2024-04-12 20:59:35,082 DEBUG }
2024-04-12 20:59:35,085 INFO
2024-04-12 20:59:35,085 INFO -------------------- Running END-USER-UI livecheck --------------------
2024-04-12 20:59:35,085 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/enduser
2024-04-12 20:59:35,085 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/enduser"
2024-04-12 20:59:35,377 INFO [http_cmd]: http status code OK
2024-04-12 20:59:35,377 DEBUG --- status code ---
2024-04-12 20:59:35,377 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:35,377 DEBUG --- http response ---
2024-04-12 20:59:35,377 DEBUG Identity Management
[]
2024-04-12 20:59:35,379 INFO
2024-04-12 20:59:35,379 INFO --------------------- Running LOGIN-UI livecheck ---------------------
2024-04-12 20:59:35,379 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/XUI
2024-04-12 20:59:35,379 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/XUI"
2024-04-12 20:59:35,661 INFO [http_cmd]: http status code OK
2024-04-12 20:59:35,661 DEBUG --- status code ---
2024-04-12 20:59:35,661 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:35,661 DEBUG --- http response ---
2024-04-12 20:59:35,661 DEBUG Login
[]
2024-04-12 20:59:35,663 INFO
2024-04-12 20:59:35,664 INFO --------------------- Running ADMIN-UI livecheck ---------------------
2024-04-12 20:59:35,664 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/platform
2024-04-12 20:59:35,664 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/platform"
2024-04-12 20:59:35,943 INFO [http_cmd]: http status code OK
2024-04-12 20:59:35,943 DEBUG --- status code ---
2024-04-12 20:59:35,943 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:35,943 DEBUG --- http response ---
2024-04-12 20:59:35,944 DEBUG Platform Admin
[]
2024-04-12 20:59:35,946 INFO LIVECHECK SUCCEEDED
2024-04-12 20:59:35,947 INFO
2024-04-12 20:59:35,947 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-12 20:59:35,948 INFO
2024-04-12 20:59:35,948 INFO --------------------- Get DS-CTS software version ---------------------
2024-04-12 20:59:35,948 INFO
2024-04-12 20:59:35,948 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- find /opt/opendj -name opendj-core.jar
2024-04-12 20:59:35,948 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-12 20:59:36,790 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:36,790 DEBUG --- stdout ---
2024-04-12 20:59:36,790 DEBUG /opt/opendj/lib/opendj-core.jar
2024-04-12 20:59:36,790 DEBUG --- stderr ---
2024-04-12 20:59:36,790 DEBUG
2024-04-12 20:59:36,790 INFO
2024-04-12 20:59:36,790 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-cts-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-cts_info/opendj-core.jar --container ds
2024-04-12 20:59:36,790 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:38,024 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:38,024 DEBUG --- stdout ---
2024-04-12 20:59:38,025 DEBUG tar: Removing leading `/' from member names
2024-04-12 20:59:38,025 DEBUG --- stderr ---
2024-04-12 20:59:38,025 DEBUG
2024-04-12 20:59:38,032 INFO
2024-04-12 20:59:38,032 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-12 20:59:38,032 INFO
2024-04-12 20:59:38,033 INFO ------------------- Get DS-IDREPO software version -------------------
2024-04-12 20:59:38,033 INFO
2024-04-12 20:59:38,033 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- find /opt/opendj -name opendj-core.jar
2024-04-12 20:59:38,033 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-12 20:59:38,844 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:38,845 DEBUG --- stdout ---
2024-04-12 20:59:38,845 DEBUG /opt/opendj/lib/opendj-core.jar
2024-04-12 20:59:38,845 DEBUG --- stderr ---
2024-04-12 20:59:38,845 DEBUG
2024-04-12 20:59:38,845 INFO
2024-04-12 20:59:38,845 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-idrepo-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-idrepo_info/opendj-core.jar --container ds
2024-04-12 20:59:38,845 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:40,085 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:40,085 DEBUG --- stdout ---
2024-04-12 20:59:40,085 DEBUG tar: Removing leading `/' from member names
2024-04-12 20:59:40,085 DEBUG --- stderr ---
2024-04-12 20:59:40,085 DEBUG
2024-04-12 20:59:40,094 INFO
2024-04-12 20:59:40,094 INFO ****************************** Initializing component pods for AM ******************************
2024-04-12 20:59:40,094 INFO
2024-04-12 20:59:40,094 INFO ----------------------- Get AM software version -----------------------
2024-04-12 20:59:40,094 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version
2024-04-12 20:59:40,094 INFO [LoginSession] Get "token_id" token for user "amadmin"
2024-04-12 20:59:40,095 DEBUG [LoginSession] re-using "token_id" token that remains valid for 2m 42s
2024-04-12 20:59:40,095 INFO [http_cmd]: curl -L --request GET --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=90BftDERsSy96T4-2IXcCvIyEbg.*AAJTSQACMDIAAlNLABw3RXpyME9oN2hhVGxtUUxDYU5sYXpNV3BHVGs9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" --cookie "route=1712955574.307.38272.642183|f60edb382037eb2df1e800d563ad78a7" "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version"
2024-04-12 20:59:40,313 INFO [http_cmd]: http status code OK
2024-04-12 20:59:40,313 DEBUG --- status code ---
2024-04-12 20:59:40,314 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:40,314 DEBUG --- http response ---
2024-04-12 20:59:40,314 DEBUG {
"_id": "version",
"_rev": "-2070515436",
"version": "7.6.0-SNAPSHOT",
"fullVersion": "ForgeRock Access Management 7.6.0-SNAPSHOT Build 0449af26230b3dbde39b69d27b76a420998ce4a5 (2024-April-11 15:09)",
"revision": "0449af26230b3dbde39b69d27b76a420998ce4a5",
"date": "2024-April-11 15:09"
}
2024-04-12 20:59:40,317 INFO
2024-04-12 20:59:40,318 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-12 20:59:40,318 INFO
2024-04-12 20:59:40,318 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-12 20:59:40,318 INFO
2024-04-12 20:59:40,318 INFO ---------------------- Get IDM software version ----------------------
2024-04-12 20:59:40,318 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/version
2024-04-12 20:59:40,318 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/version"
2024-04-12 20:59:40,533 INFO [http_cmd]: http status code OK
2024-04-12 20:59:40,533 DEBUG --- status code ---
2024-04-12 20:59:40,533 DEBUG http status code is 200 (expected 200)
2024-04-12 20:59:40,533 DEBUG --- http response ---
2024-04-12 20:59:40,533 DEBUG {
"_id": "version",
"productVersion": "7.6.0-SNAPSHOT",
"productBuildDate": "20240411202127",
"productRevision": "8554e0d1c5"
}
2024-04-12 20:59:40,536 INFO
2024-04-12 20:59:40,536 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-12 20:59:40,536 INFO
2024-04-12 20:59:40,536 INFO ------------------ Get END-USER-UI software version ------------------
2024-04-12 20:59:40,537 INFO
2024-04-12 20:59:40,537 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec end-user-ui-58878d7668-5gqgr --container end-user-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-12 20:59:40,537 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-12 20:59:41,279 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:41,279 DEBUG --- stdout ---
2024-04-12 20:59:41,279 DEBUG /usr/share/nginx/html/js/chunk-vendors.0380a414.js
2024-04-12 20:59:41,279 DEBUG --- stderr ---
2024-04-12 20:59:41,279 DEBUG
2024-04-12 20:59:41,279 INFO
2024-04-12 20:59:41,280 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp end-user-ui-58878d7668-5gqgr:/usr/share/nginx/html/js/chunk-vendors.0380a414.js /tmp/end-user-ui_info/chunk-vendors.0380a414.js --container end-user-ui
2024-04-12 20:59:41,280 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:42,394 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:42,394 DEBUG --- stdout ---
2024-04-12 20:59:42,395 DEBUG tar: Removing leading `/' from member names
2024-04-12 20:59:42,395 DEBUG --- stderr ---
2024-04-12 20:59:42,395 DEBUG
2024-04-12 20:59:42,422 INFO
2024-04-12 20:59:42,422 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-12 20:59:42,422 INFO
2024-04-12 20:59:42,422 INFO -------------------- Get LOGIN-UI software version --------------------
2024-04-12 20:59:42,422 INFO
2024-04-12 20:59:42,422 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec login-ui-5fdf49fc66-svvhv --container login-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-12 20:59:42,422 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-12 20:59:43,153 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:43,154 DEBUG --- stdout ---
2024-04-12 20:59:43,155 DEBUG /usr/share/nginx/html/js/chunk-vendors.f2fa5365.js
2024-04-12 20:59:43,155 DEBUG --- stderr ---
2024-04-12 20:59:43,155 DEBUG
2024-04-12 20:59:43,155 INFO
2024-04-12 20:59:43,155 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp login-ui-5fdf49fc66-svvhv:/usr/share/nginx/html/js/chunk-vendors.f2fa5365.js /tmp/login-ui_info/chunk-vendors.f2fa5365.js --container login-ui
2024-04-12 20:59:43,155 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:44,282 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:44,282 DEBUG --- stdout ---
2024-04-12 20:59:44,282 DEBUG tar: Removing leading `/' from member names
2024-04-12 20:59:44,282 DEBUG --- stderr ---
2024-04-12 20:59:44,282 DEBUG
2024-04-12 20:59:44,300 INFO
2024-04-12 20:59:44,300 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-12 20:59:44,300 INFO
2024-04-12 20:59:44,300 INFO -------------------- Get ADMIN-UI software version --------------------
2024-04-12 20:59:44,300 INFO
2024-04-12 20:59:44,300 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec admin-ui-c4c7bf584-vlxw7 --container admin-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-12 20:59:44,300 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-12 20:59:45,037 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:45,038 DEBUG --- stdout ---
2024-04-12 20:59:45,038 DEBUG /usr/share/nginx/html/js/chunk-vendors.b3d879ac.js
2024-04-12 20:59:45,038 DEBUG --- stderr ---
2024-04-12 20:59:45,038 DEBUG
2024-04-12 20:59:45,038 INFO
2024-04-12 20:59:45,038 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp admin-ui-c4c7bf584-vlxw7:/usr/share/nginx/html/js/chunk-vendors.b3d879ac.js /tmp/admin-ui_info/chunk-vendors.b3d879ac.js --container admin-ui
2024-04-12 20:59:45,038 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:46,219 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:46,220 DEBUG --- stdout ---
2024-04-12 20:59:46,220 DEBUG tar: Removing leading `/' from member names
2024-04-12 20:59:46,220 DEBUG --- stderr ---
2024-04-12 20:59:46,220 DEBUG
2024-04-12 20:59:46,273 INFO
2024-04-12 20:59:46,274 INFO
2024-04-12 20:59:46,274 INFO ====================================================================================================
2024-04-12 20:59:46,274 INFO ================ Admin password for DS-CTS is: i1Y1gGUHX4knylpjuiOvx1vBk6Q6sJR7 ================
2024-04-12 20:59:46,274 INFO ====================================================================================================
2024-04-12 20:59:46,274 INFO
2024-04-12 20:59:46,274 INFO ====================================================================================================
2024-04-12 20:59:46,274 INFO ============== Admin password for DS-IDREPO is: i1Y1gGUHX4knylpjuiOvx1vBk6Q6sJR7 ==============
2024-04-12 20:59:46,274 INFO ====================================================================================================
2024-04-12 20:59:46,274 INFO
2024-04-12 20:59:46,274 INFO ====================================================================================================
2024-04-12 20:59:46,274 INFO ====================== Admin password for AM is: R1bWLMpzX7eBxZoxRXjt0CYz ======================
2024-04-12 20:59:46,274 INFO ====================================================================================================
2024-04-12 20:59:46,274 INFO
2024-04-12 20:59:46,274 INFO ====================================================================================================
2024-04-12 20:59:46,274 INFO ===================== Admin password for IDM is: ss27M4ZWyrACbbta8g89KgpM =====================
2024-04-12 20:59:46,274 INFO ====================================================================================================
2024-04-12 20:59:46,274 INFO
2024-04-12 20:59:46,274 INFO *************************************** Dumping pod list ***************************************
2024-04-12 20:59:46,275 INFO Dumping pod list to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/_pod-list.txt
2024-04-12 20:59:46,795 INFO Dumping platform-config configmap to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/_platform-config_configmap.txt
2024-04-12 20:59:47,166 INFO
2024-04-12 20:59:47,166 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-12 20:59:47,167 INFO
2024-04-12 20:59:47,167 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:47,167 INFO
2024-04-12 20:59:47,167 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:47,167 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:47,590 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:47,590 DEBUG --- stdout ---
2024-04-12 20:59:47,590 DEBUG 3
2024-04-12 20:59:47,591 DEBUG --- stderr ---
2024-04-12 20:59:47,591 DEBUG
2024-04-12 20:59:47,591 INFO
2024-04-12 20:59:47,591 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:47,591 INFO
2024-04-12 20:59:47,591 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-cts --output jsonpath={.items[*].metadata.name}
2024-04-12 20:59:47,591 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:48,052 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:48,053 DEBUG --- stdout ---
2024-04-12 20:59:48,053 DEBUG ds-cts-0 ds-cts-1 ds-cts-2
2024-04-12 20:59:48,053 DEBUG --- stderr ---
2024-04-12 20:59:48,053 DEBUG
2024-04-12 20:59:48,053 INFO
2024-04-12 20:59:48,053 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-12 20:59:48,053 INFO
2024-04-12 20:59:48,053 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:48,053 INFO
2024-04-12 20:59:48,053 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:48,053 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:48,436 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:48,438 DEBUG --- stdout ---
2024-04-12 20:59:48,438 DEBUG 3
2024-04-12 20:59:48,438 DEBUG --- stderr ---
2024-04-12 20:59:48,438 DEBUG
2024-04-12 20:59:48,438 INFO
2024-04-12 20:59:48,438 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:48,439 INFO
2024-04-12 20:59:48,439 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-idrepo --output jsonpath={.items[*].metadata.name}
2024-04-12 20:59:48,439 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:48,884 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:48,885 DEBUG --- stdout ---
2024-04-12 20:59:48,885 DEBUG ds-idrepo-0 ds-idrepo-1 ds-idrepo-2
2024-04-12 20:59:48,885 DEBUG --- stderr ---
2024-04-12 20:59:48,885 DEBUG
2024-04-12 20:59:48,885 INFO
2024-04-12 20:59:48,885 INFO ****************************** Initializing component pods for AM ******************************
2024-04-12 20:59:48,885 INFO
2024-04-12 20:59:48,885 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:48,885 INFO
2024-04-12 20:59:48,885 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:48,885 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:49,247 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:49,247 DEBUG --- stdout ---
2024-04-12 20:59:49,248 DEBUG 3
2024-04-12 20:59:49,248 DEBUG --- stderr ---
2024-04-12 20:59:49,248 DEBUG
2024-04-12 20:59:49,248 INFO
2024-04-12 20:59:49,248 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:49,248 INFO
2024-04-12 20:59:49,248 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}
2024-04-12 20:59:49,248 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:49,684 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:49,684 DEBUG --- stdout ---
2024-04-12 20:59:49,684 DEBUG am-6d96779b5d-pfljh am-6d96779b5d-qbg7l am-6d96779b5d-zn69c
2024-04-12 20:59:49,684 DEBUG --- stderr ---
2024-04-12 20:59:49,684 DEBUG
2024-04-12 20:59:49,685 INFO
2024-04-12 20:59:49,685 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-12 20:59:49,685 INFO
2024-04-12 20:59:49,685 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:49,685 INFO
2024-04-12 20:59:49,685 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-12 20:59:49,685 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:50,069 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:50,069 DEBUG --- stdout ---
2024-04-12 20:59:50,069 DEBUG amster-twfcc
2024-04-12 20:59:50,069 DEBUG --- stderr ---
2024-04-12 20:59:50,069 DEBUG
2024-04-12 20:59:50,069 INFO
2024-04-12 20:59:50,069 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-12 20:59:50,069 INFO
2024-04-12 20:59:50,069 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:50,070 INFO
2024-04-12 20:59:50,070 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:50,070 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:50,454 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:50,454 DEBUG --- stdout ---
2024-04-12 20:59:50,454 DEBUG 2
2024-04-12 20:59:50,454 DEBUG --- stderr ---
2024-04-12 20:59:50,454 DEBUG
2024-04-12 20:59:50,454 INFO
2024-04-12 20:59:50,454 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:50,455 INFO
2024-04-12 20:59:50,455 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=idm --output jsonpath={.items[*].metadata.name}
2024-04-12 20:59:50,455 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:50,845 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:50,845 DEBUG --- stdout ---
2024-04-12 20:59:50,845 DEBUG idm-b85f9d978-b559t idm-b85f9d978-dz57p
2024-04-12 20:59:50,845 DEBUG --- stderr ---
2024-04-12 20:59:50,845 DEBUG
2024-04-12 20:59:50,845 INFO
2024-04-12 20:59:50,845 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-12 20:59:50,845 INFO
2024-04-12 20:59:50,845 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:50,845 INFO
2024-04-12 20:59:50,846 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:50,846 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:51,209 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:51,209 DEBUG --- stdout ---
2024-04-12 20:59:51,209 DEBUG 1
2024-04-12 20:59:51,209 DEBUG --- stderr ---
2024-04-12 20:59:51,209 DEBUG
2024-04-12 20:59:51,209 INFO
2024-04-12 20:59:51,209 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:51,210 INFO
2024-04-12 20:59:51,210 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].metadata.name}
2024-04-12 20:59:51,210 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:51,542 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:51,542 DEBUG --- stdout ---
2024-04-12 20:59:51,542 DEBUG end-user-ui-58878d7668-5gqgr
2024-04-12 20:59:51,542 DEBUG --- stderr ---
2024-04-12 20:59:51,542 DEBUG
2024-04-12 20:59:51,542 INFO
2024-04-12 20:59:51,542 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-12 20:59:51,542 INFO
2024-04-12 20:59:51,542 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:51,542 INFO
2024-04-12 20:59:51,543 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:51,543 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:51,849 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:51,849 DEBUG --- stdout ---
2024-04-12 20:59:51,849 DEBUG 1
2024-04-12 20:59:51,849 DEBUG --- stderr ---
2024-04-12 20:59:51,849 DEBUG
2024-04-12 20:59:51,850 INFO
2024-04-12 20:59:51,850 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:51,850 INFO
2024-04-12 20:59:51,850 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].metadata.name}
2024-04-12 20:59:51,850 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:52,161 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:52,162 DEBUG --- stdout ---
2024-04-12 20:59:52,162 DEBUG login-ui-5fdf49fc66-svvhv
2024-04-12 20:59:52,162 DEBUG --- stderr ---
2024-04-12 20:59:52,162 DEBUG
2024-04-12 20:59:52,162 INFO
2024-04-12 20:59:52,162 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-12 20:59:52,162 INFO
2024-04-12 20:59:52,162 INFO --------------------- Get expected number of pods ---------------------
2024-04-12 20:59:52,162 INFO
2024-04-12 20:59:52,162 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-12 20:59:52,162 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-12 20:59:52,464 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:52,464 DEBUG --- stdout ---
2024-04-12 20:59:52,464 DEBUG 1
2024-04-12 20:59:52,464 DEBUG --- stderr ---
2024-04-12 20:59:52,464 DEBUG
2024-04-12 20:59:52,464 INFO
2024-04-12 20:59:52,464 INFO ---------------------------- Get pod list ----------------------------
2024-04-12 20:59:52,465 INFO
2024-04-12 20:59:52,465 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].metadata.name}
2024-04-12 20:59:52,465 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-12 20:59:52,797 INFO [loop_until]: OK (rc = 0)
2024-04-12 20:59:52,797 DEBUG --- stdout ---
2024-04-12 20:59:52,797 DEBUG admin-ui-c4c7bf584-vlxw7
2024-04-12 20:59:52,797 DEBUG --- stderr ---
2024-04-12 20:59:52,797 DEBUG
2024-04-12 20:59:52,797 INFO
2024-04-12 20:59:52,798 INFO *********************************** Dumping components logs ***********************************
2024-04-12 20:59:52,798 INFO
2024-04-12 20:59:52,798 INFO ----------------------- Dumping logs for DS-CTS -----------------------
2024-04-12 20:59:54,162 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-cts-0.txt
2024-04-12 20:59:54,163 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-cts-0.txt
2024-04-12 20:59:54,164 INFO Check pod logs for errors
2024-04-12 20:59:55,485 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-cts-1.txt
2024-04-12 20:59:55,486 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-cts-1.txt
2024-04-12 20:59:55,487 INFO Check pod logs for errors
2024-04-12 20:59:56,824 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-cts-2.txt
2024-04-12 20:59:56,825 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-cts-2.txt
2024-04-12 20:59:56,828 INFO Check pod logs for errors
2024-04-12 20:59:56,831 INFO
2024-04-12 20:59:56,831 INFO --------------------- Dumping logs for DS-IDREPO ---------------------
2024-04-12 20:59:58,201 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-idrepo-0.txt
2024-04-12 20:59:58,201 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-idrepo-0.txt
2024-04-12 20:59:58,203 INFO Check pod logs for errors
2024-04-12 20:59:59,523 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-idrepo-1.txt
2024-04-12 20:59:59,525 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-idrepo-1.txt
2024-04-12 20:59:59,526 INFO Check pod logs for errors
2024-04-12 21:00:00,876 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-idrepo-2.txt
2024-04-12 21:00:00,877 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/ds-idrepo-2.txt
2024-04-12 21:00:00,879 INFO Check pod logs for errors
2024-04-12 21:00:00,882 INFO
2024-04-12 21:00:00,882 INFO ------------------------- Dumping logs for AM -------------------------
2024-04-12 21:00:02,247 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/am-6d96779b5d-pfljh.txt
2024-04-12 21:00:02,248 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/am-6d96779b5d-pfljh.txt
2024-04-12 21:00:02,249 INFO Check pod logs for errors
2024-04-12 21:00:03,455 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/am-6d96779b5d-qbg7l.txt
2024-04-12 21:00:03,456 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/am-6d96779b5d-qbg7l.txt
2024-04-12 21:00:03,456 INFO Check pod logs for errors
2024-04-12 21:00:04,888 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/am-6d96779b5d-zn69c.txt
2024-04-12 21:00:04,891 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/am-6d96779b5d-zn69c.txt
2024-04-12 21:00:04,894 INFO Check pod logs for errors
2024-04-12 21:00:04,896 INFO
2024-04-12 21:00:04,896 INFO ----------------------- Dumping logs for AMSTER -----------------------
2024-04-12 21:00:05,978 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/amster-twfcc.txt
2024-04-12 21:00:05,979 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/amster-twfcc.txt
2024-04-12 21:00:05,979 INFO Check pod logs for errors
2024-04-12 21:00:05,979 INFO
2024-04-12 21:00:05,979 INFO ------------------------ Dumping logs for IDM ------------------------
2024-04-12 21:00:07,416 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/idm-b85f9d978-b559t.txt
2024-04-12 21:00:07,416 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/idm-b85f9d978-b559t.txt
2024-04-12 21:00:07,419 INFO Check pod logs for errors
2024-04-12 21:00:08,846 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/idm-b85f9d978-dz57p.txt
2024-04-12 21:00:08,846 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/idm-b85f9d978-dz57p.txt
2024-04-12 21:00:08,849 INFO Check pod logs for errors
2024-04-12 21:00:08,851 INFO
2024-04-12 21:00:08,851 INFO -------------------- Dumping logs for END-USER-UI --------------------
2024-04-12 21:00:09,935 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/end-user-ui-58878d7668-5gqgr.txt
2024-04-12 21:00:09,935 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/end-user-ui-58878d7668-5gqgr.txt
2024-04-12 21:00:09,936 INFO Check pod logs for errors
2024-04-12 21:00:09,937 INFO
2024-04-12 21:00:09,937 INFO ---------------------- Dumping logs for LOGIN-UI ----------------------
2024-04-12 21:00:10,980 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/login-ui-5fdf49fc66-svvhv.txt
2024-04-12 21:00:10,981 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/login-ui-5fdf49fc66-svvhv.txt
2024-04-12 21:00:10,982 INFO Check pod logs for errors
2024-04-12 21:00:10,983 INFO
2024-04-12 21:00:10,983 INFO ---------------------- Dumping logs for ADMIN-UI ----------------------
2024-04-12 21:00:12,057 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/admin-ui-c4c7bf584-vlxw7.txt
2024-04-12 21:00:12,058 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240412_205946-after-deployment/admin-ui-c4c7bf584-vlxw7.txt
2024-04-12 21:00:12,059 INFO Check pod logs for errors
[2024-04-12 21:00:12] - INFO: Deployment ready
________________________________________________________________________________
[2024-04-12 21:00:12] Discover_all_components post : Post method
________________________________________________________________________________
Setting result to PASS
Task has been successfully stopped