--Task--
name: Discover_all_components
enabled: True
class_name: DiscoverComponentsTask
source_name: controller
source_namespace: >default<
target_name: controller
target_namespace: >default<
start: 0.0
stop: None
timeout: no timeout
loop: False
interval: None
dependencies: []
wait_for: []
preceding_task: None
options: {}
group_name: None
Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock
________________________________________________________________________________
[2024-04-15 20:55:07] Discover_all_components pre : Initialising task parameters
________________________________________________________________________________
task will be executed on controller (localhost)
________________________________________________________________________________
[2024-04-15 20:55:07] Discover_all_components step1 : Discover components
________________________________________________________________________________
2024-04-15 20:55:07,380 INFO Deployment was skipped.
2024-04-15 20:55:07,726 INFO
2024-04-15 20:55:07,726 INFO ************************************* Creating deployment *************************************
2024-04-15 20:55:07,726 INFO Creating normal (forgeops) type deployment for deployment: stack
2024-04-15 20:55:07,744 INFO
2024-04-15 20:55:07,744 INFO ------- Custom component configuration present. Loading values -------
2024-04-15 20:55:07,744 INFO
2024-04-15 20:55:07,745 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker ds cts
2024-04-15 20:55:07,830 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:07,830 DEBUG --- stdout ---
2024-04-15 20:55:07,830 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/cts
2024-04-15 20:55:07,831 DEBUG --- stderr ---
2024-04-15 20:55:07,831 DEBUG
2024-04-15 20:55:07,831 INFO
2024-04-15 20:55:07,831 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/cts/Dockerfile: gcr.io/forgerock-io/ds/pit1:7.6.0-d49ad6a5879fab3530729d8773297783d042f903
2024-04-15 20:55:07,831 INFO
2024-04-15 20:55:07,831 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker ds idrepo
2024-04-15 20:55:07,911 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:07,911 DEBUG --- stdout ---
2024-04-15 20:55:07,911 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/idrepo
2024-04-15 20:55:07,911 DEBUG --- stderr ---
2024-04-15 20:55:07,911 DEBUG
2024-04-15 20:55:07,911 INFO
2024-04-15 20:55:07,911 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/idrepo/Dockerfile: gcr.io/forgerock-io/ds/pit1:7.6.0-d49ad6a5879fab3530729d8773297783d042f903
2024-04-15 20:55:07,911 INFO
2024-04-15 20:55:07,912 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker am
2024-04-15 20:55:07,993 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:07,993 DEBUG --- stdout ---
2024-04-15 20:55:07,993 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/am
2024-04-15 20:55:07,993 DEBUG --- stderr ---
2024-04-15 20:55:07,993 DEBUG
2024-04-15 20:55:07,993 INFO
2024-04-15 20:55:07,994 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/am/Dockerfile: gcr.io/forgerock-io/am-cdk/pit1:7.6.0-bbf6791c961b39ef101ebad36686ee2c3cde94bb
2024-04-15 20:55:07,994 INFO
2024-04-15 20:55:07,994 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker amster
2024-04-15 20:55:08,077 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:08,077 DEBUG --- stdout ---
2024-04-15 20:55:08,077 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/amster
2024-04-15 20:55:08,078 DEBUG --- stderr ---
2024-04-15 20:55:08,078 DEBUG
2024-04-15 20:55:08,078 INFO
2024-04-15 20:55:08,078 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/amster/Dockerfile: gcr.io/forgerock-io/amster/pit1:7.6.0-bbf6791c961b39ef101ebad36686ee2c3cde94bb
2024-04-15 20:55:08,078 INFO
2024-04-15 20:55:08,078 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker idm
2024-04-15 20:55:08,159 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:08,159 DEBUG --- stdout ---
2024-04-15 20:55:08,159 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/idm
2024-04-15 20:55:08,159 DEBUG --- stderr ---
2024-04-15 20:55:08,160 DEBUG
2024-04-15 20:55:08,160 INFO
2024-04-15 20:55:08,160 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/idm/Dockerfile: gcr.io/forgerock-io/idm-cdk/pit1:7.6.0-8554e0d1c5044f4f6c6504cd366196848874e695
2024-04-15 20:55:08,160 INFO
2024-04-15 20:55:08,160 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base end-user-ui
2024-04-15 20:55:08,240 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:08,240 DEBUG --- stdout ---
2024-04-15 20:55:08,240 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/end-user-ui
2024-04-15 20:55:08,240 DEBUG --- stderr ---
2024-04-15 20:55:08,240 DEBUG
2024-04-15 20:55:08,240 INFO
2024-04-15 20:55:08,243 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/end-user-ui/kustomization.yaml: gcr.io/forgerock-io/platform-enduser-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-15 20:55:08,244 INFO
2024-04-15 20:55:08,244 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base login-ui
2024-04-15 20:55:08,323 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:08,324 DEBUG --- stdout ---
2024-04-15 20:55:08,324 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/login-ui
2024-04-15 20:55:08,324 DEBUG --- stderr ---
2024-04-15 20:55:08,324 DEBUG
2024-04-15 20:55:08,324 INFO
2024-04-15 20:55:08,327 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/login-ui/kustomization.yaml: gcr.io/forgerock-io/platform-login-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-15 20:55:08,327 INFO
2024-04-15 20:55:08,327 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base admin-ui
2024-04-15 20:55:08,412 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:08,413 DEBUG --- stdout ---
2024-04-15 20:55:08,413 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/admin-ui
2024-04-15 20:55:08,413 DEBUG --- stderr ---
2024-04-15 20:55:08,413 DEBUG
2024-04-15 20:55:08,413 INFO
2024-04-15 20:55:08,416 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/admin-ui/kustomization.yaml: gcr.io/forgerock-io/platform-admin-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-15 20:55:08,416 INFO Loading external components from config.yaml
2024-04-15 20:55:08,535 INFO No external components defined.
2024-04-15 20:55:08,658 INFO Setting context for components to: gke_engineeringpit_us-east1-d_xlou-cdm
2024-04-15 20:55:08,658 INFO
2024-04-15 20:55:08,658 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-15 20:55:08,658 INFO
2024-04-15 20:55:08,658 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:08,658 INFO
2024-04-15 20:55:08,658 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:08,659 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:09,054 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:09,055 DEBUG --- stdout ---
2024-04-15 20:55:09,055 DEBUG 3
2024-04-15 20:55:09,055 DEBUG --- stderr ---
2024-04-15 20:55:09,055 DEBUG
2024-04-15 20:55:09,055 INFO
2024-04-15 20:55:09,055 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:55:09,055 INFO
2024-04-15 20:55:09,055 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-cts --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-15 20:55:09,055 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:55:09,498 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 20:55:09,498 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:09,498 DEBUG --- stdout ---
2024-04-15 20:55:09,498 DEBUG ds-cts-0 ds-cts-1 ds-cts-2
2024-04-15 20:55:09,498 DEBUG --- stderr ---
2024-04-15 20:55:09,498 DEBUG
2024-04-15 20:55:09,498 INFO
2024-04-15 20:55:09,498 INFO -------------------- Check pod ds-cts-0 is running --------------------
2024-04-15 20:55:09,499 INFO
2024-04-15 20:55:09,499 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-0 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:09,499 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:09,897 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:09,897 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:09,897 DEBUG --- stdout ---
2024-04-15 20:55:09,897 DEBUG Running
2024-04-15 20:55:09,897 DEBUG --- stderr ---
2024-04-15 20:55:09,897 DEBUG
2024-04-15 20:55:09,897 INFO
2024-04-15 20:55:09,897 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-0 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:09,898 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:10,268 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:10,269 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:10,269 DEBUG --- stdout ---
2024-04-15 20:55:10,269 DEBUG true
2024-04-15 20:55:10,269 DEBUG --- stderr ---
2024-04-15 20:55:10,269 DEBUG
2024-04-15 20:55:10,269 INFO
2024-04-15 20:55:10,269 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-0 --output jsonpath={.status.startTime}
2024-04-15 20:55:10,269 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:10,635 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:10,635 DEBUG --- stdout ---
2024-04-15 20:55:10,635 DEBUG 2024-04-13T22:38:33Z
2024-04-15 20:55:10,635 DEBUG --- stderr ---
2024-04-15 20:55:10,635 DEBUG
2024-04-15 20:55:10,636 INFO
2024-04-15 20:55:10,636 INFO ------------- Check pod ds-cts-0 filesystem is accessible -------------
2024-04-15 20:55:11,469 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:11,469 INFO
2024-04-15 20:55:11,469 INFO ------------------ Check pod ds-cts-0 restart count ------------------
2024-04-15 20:55:11,470 INFO
2024-04-15 20:55:11,470 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-0 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:11,470 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:11,868 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:11,868 DEBUG --- stdout ---
2024-04-15 20:55:11,868 DEBUG 0
2024-04-15 20:55:11,868 DEBUG --- stderr ---
2024-04-15 20:55:11,868 DEBUG
2024-04-15 20:55:11,868 INFO Pod ds-cts-0 has been restarted 0 times.
2024-04-15 20:55:11,868 INFO
2024-04-15 20:55:11,869 INFO -------------------- Check pod ds-cts-1 is running --------------------
2024-04-15 20:55:11,869 INFO
2024-04-15 20:55:11,869 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-1 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:11,869 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:12,251 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:12,251 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:12,251 DEBUG --- stdout ---
2024-04-15 20:55:12,251 DEBUG Running
2024-04-15 20:55:12,251 DEBUG --- stderr ---
2024-04-15 20:55:12,251 DEBUG
2024-04-15 20:55:12,251 INFO
2024-04-15 20:55:12,251 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-1 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:12,252 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:12,613 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:12,613 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:12,613 DEBUG --- stdout ---
2024-04-15 20:55:12,613 DEBUG true
2024-04-15 20:55:12,613 DEBUG --- stderr ---
2024-04-15 20:55:12,613 DEBUG
2024-04-15 20:55:12,613 INFO
2024-04-15 20:55:12,613 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-1 --output jsonpath={.status.startTime}
2024-04-15 20:55:12,613 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:12,982 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:12,983 DEBUG --- stdout ---
2024-04-15 20:55:12,983 DEBUG 2024-04-15T16:51:45Z
2024-04-15 20:55:12,983 DEBUG --- stderr ---
2024-04-15 20:55:12,983 DEBUG
2024-04-15 20:55:12,983 INFO
2024-04-15 20:55:12,983 INFO ------------- Check pod ds-cts-1 filesystem is accessible -------------
2024-04-15 20:55:13,890 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:13,891 INFO
2024-04-15 20:55:13,891 INFO ------------------ Check pod ds-cts-1 restart count ------------------
2024-04-15 20:55:13,891 INFO
2024-04-15 20:55:13,891 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-1 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:13,891 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:14,287 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:14,287 DEBUG --- stdout ---
2024-04-15 20:55:14,287 DEBUG 0
2024-04-15 20:55:14,287 DEBUG --- stderr ---
2024-04-15 20:55:14,287 DEBUG
2024-04-15 20:55:14,288 INFO Pod ds-cts-1 has been restarted 0 times.
2024-04-15 20:55:14,288 INFO
2024-04-15 20:55:14,288 INFO -------------------- Check pod ds-cts-2 is running --------------------
2024-04-15 20:55:14,288 INFO
2024-04-15 20:55:14,288 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:14,288 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:14,666 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:14,667 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:14,668 DEBUG --- stdout ---
2024-04-15 20:55:14,668 DEBUG Running
2024-04-15 20:55:14,668 DEBUG --- stderr ---
2024-04-15 20:55:14,668 DEBUG
2024-04-15 20:55:14,668 INFO
2024-04-15 20:55:14,668 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:14,669 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:15,032 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:15,032 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:15,032 DEBUG --- stdout ---
2024-04-15 20:55:15,032 DEBUG true
2024-04-15 20:55:15,032 DEBUG --- stderr ---
2024-04-15 20:55:15,032 DEBUG
2024-04-15 20:55:15,032 INFO
2024-04-15 20:55:15,032 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-2 --output jsonpath={.status.startTime}
2024-04-15 20:55:15,032 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:15,402 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:15,402 DEBUG --- stdout ---
2024-04-15 20:55:15,402 DEBUG 2024-04-15T16:52:10Z
2024-04-15 20:55:15,403 DEBUG --- stderr ---
2024-04-15 20:55:15,403 DEBUG
2024-04-15 20:55:15,404 INFO
2024-04-15 20:55:15,404 INFO ------------- Check pod ds-cts-2 filesystem is accessible -------------
2024-04-15 20:55:16,253 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:16,253 INFO
2024-04-15 20:55:16,254 INFO ------------------ Check pod ds-cts-2 restart count ------------------
2024-04-15 20:55:16,254 INFO
2024-04-15 20:55:16,254 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:16,254 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:16,642 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:16,642 DEBUG --- stdout ---
2024-04-15 20:55:16,642 DEBUG 0
2024-04-15 20:55:16,642 DEBUG --- stderr ---
2024-04-15 20:55:16,642 DEBUG
2024-04-15 20:55:16,642 INFO Pod ds-cts-2 has been restarted 0 times.
2024-04-15 20:55:16,642 INFO
2024-04-15 20:55:16,642 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-15 20:55:16,642 INFO
2024-04-15 20:55:16,642 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:16,643 INFO
2024-04-15 20:55:16,643 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:16,643 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:17,044 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:17,044 DEBUG --- stdout ---
2024-04-15 20:55:17,044 DEBUG 3
2024-04-15 20:55:17,044 DEBUG --- stderr ---
2024-04-15 20:55:17,044 DEBUG
2024-04-15 20:55:17,045 INFO
2024-04-15 20:55:17,045 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:55:17,045 INFO
2024-04-15 20:55:17,045 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-idrepo --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-15 20:55:17,045 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:55:17,522 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 20:55:17,522 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:17,522 DEBUG --- stdout ---
2024-04-15 20:55:17,522 DEBUG ds-idrepo-0 ds-idrepo-1 ds-idrepo-2
2024-04-15 20:55:17,523 DEBUG --- stderr ---
2024-04-15 20:55:17,523 DEBUG
2024-04-15 20:55:17,523 INFO
2024-04-15 20:55:17,523 INFO ------------------ Check pod ds-idrepo-0 is running ------------------
2024-04-15 20:55:17,523 INFO
2024-04-15 20:55:17,523 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-0 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:17,523 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:17,911 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:17,911 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:17,911 DEBUG --- stdout ---
2024-04-15 20:55:17,911 DEBUG Running
2024-04-15 20:55:17,911 DEBUG --- stderr ---
2024-04-15 20:55:17,911 DEBUG
2024-04-15 20:55:17,912 INFO
2024-04-15 20:55:17,912 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-0 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:17,912 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:18,289 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:18,289 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:18,289 DEBUG --- stdout ---
2024-04-15 20:55:18,289 DEBUG true
2024-04-15 20:55:18,289 DEBUG --- stderr ---
2024-04-15 20:55:18,289 DEBUG
2024-04-15 20:55:18,289 INFO
2024-04-15 20:55:18,289 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-0 --output jsonpath={.status.startTime}
2024-04-15 20:55:18,289 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:18,655 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:18,655 DEBUG --- stdout ---
2024-04-15 20:55:18,655 DEBUG 2024-04-13T22:38:34Z
2024-04-15 20:55:18,655 DEBUG --- stderr ---
2024-04-15 20:55:18,655 DEBUG
2024-04-15 20:55:18,656 INFO
2024-04-15 20:55:18,656 INFO ----------- Check pod ds-idrepo-0 filesystem is accessible -----------
2024-04-15 20:55:19,481 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:19,481 INFO
2024-04-15 20:55:19,481 INFO ----------------- Check pod ds-idrepo-0 restart count -----------------
2024-04-15 20:55:19,482 INFO
2024-04-15 20:55:19,482 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-0 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:19,482 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:19,856 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:19,856 DEBUG --- stdout ---
2024-04-15 20:55:19,858 DEBUG 0
2024-04-15 20:55:19,858 DEBUG --- stderr ---
2024-04-15 20:55:19,858 DEBUG
2024-04-15 20:55:19,858 INFO Pod ds-idrepo-0 has been restarted 0 times.
2024-04-15 20:55:19,859 INFO
2024-04-15 20:55:19,859 INFO ------------------ Check pod ds-idrepo-1 is running ------------------
2024-04-15 20:55:19,859 INFO
2024-04-15 20:55:19,859 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-1 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:19,859 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:20,222 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:20,222 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:20,223 DEBUG --- stdout ---
2024-04-15 20:55:20,223 DEBUG Running
2024-04-15 20:55:20,223 DEBUG --- stderr ---
2024-04-15 20:55:20,223 DEBUG
2024-04-15 20:55:20,223 INFO
2024-04-15 20:55:20,223 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-1 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:20,223 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:20,588 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:20,588 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:20,588 DEBUG --- stdout ---
2024-04-15 20:55:20,588 DEBUG true
2024-04-15 20:55:20,588 DEBUG --- stderr ---
2024-04-15 20:55:20,588 DEBUG
2024-04-15 20:55:20,588 INFO
2024-04-15 20:55:20,588 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-1 --output jsonpath={.status.startTime}
2024-04-15 20:55:20,588 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:20,960 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:20,960 DEBUG --- stdout ---
2024-04-15 20:55:20,960 DEBUG 2024-04-15T20:48:06Z
2024-04-15 20:55:20,960 DEBUG --- stderr ---
2024-04-15 20:55:20,960 DEBUG
2024-04-15 20:55:20,961 INFO
2024-04-15 20:55:20,961 INFO ----------- Check pod ds-idrepo-1 filesystem is accessible -----------
2024-04-15 20:55:21,810 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:21,810 INFO
2024-04-15 20:55:21,810 INFO ----------------- Check pod ds-idrepo-1 restart count -----------------
2024-04-15 20:55:21,810 INFO
2024-04-15 20:55:21,810 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-1 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:21,810 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:22,210 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:22,211 DEBUG --- stdout ---
2024-04-15 20:55:22,211 DEBUG 0
2024-04-15 20:55:22,211 DEBUG --- stderr ---
2024-04-15 20:55:22,211 DEBUG
2024-04-15 20:55:22,211 INFO Pod ds-idrepo-1 has been restarted 0 times.
2024-04-15 20:55:22,211 INFO
2024-04-15 20:55:22,211 INFO ------------------ Check pod ds-idrepo-2 is running ------------------
2024-04-15 20:55:22,212 INFO
2024-04-15 20:55:22,212 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:22,212 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:22,622 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:22,626 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:22,626 DEBUG --- stdout ---
2024-04-15 20:55:22,626 DEBUG Running
2024-04-15 20:55:22,626 DEBUG --- stderr ---
2024-04-15 20:55:22,626 DEBUG
2024-04-15 20:55:22,627 INFO
2024-04-15 20:55:22,627 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:22,627 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:23,038 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:23,038 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:23,038 DEBUG --- stdout ---
2024-04-15 20:55:23,038 DEBUG true
2024-04-15 20:55:23,038 DEBUG --- stderr ---
2024-04-15 20:55:23,038 DEBUG
2024-04-15 20:55:23,038 INFO
2024-04-15 20:55:23,038 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-2 --output jsonpath={.status.startTime}
2024-04-15 20:55:23,039 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:23,437 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:23,437 DEBUG --- stdout ---
2024-04-15 20:55:23,437 DEBUG 2024-04-15T20:48:42Z
2024-04-15 20:55:23,437 DEBUG --- stderr ---
2024-04-15 20:55:23,437 DEBUG
2024-04-15 20:55:23,438 INFO
2024-04-15 20:55:23,438 INFO ----------- Check pod ds-idrepo-2 filesystem is accessible -----------
2024-04-15 20:55:24,289 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:24,289 INFO
2024-04-15 20:55:24,289 INFO ----------------- Check pod ds-idrepo-2 restart count -----------------
2024-04-15 20:55:24,289 INFO
2024-04-15 20:55:24,289 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:24,289 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:24,676 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:24,676 DEBUG --- stdout ---
2024-04-15 20:55:24,676 DEBUG 0
2024-04-15 20:55:24,677 DEBUG --- stderr ---
2024-04-15 20:55:24,677 DEBUG
2024-04-15 20:55:24,677 INFO Pod ds-idrepo-2 has been restarted 0 times.
2024-04-15 20:55:24,677 INFO
2024-04-15 20:55:24,677 INFO ****************************** Initializing component pods for AM ******************************
2024-04-15 20:55:24,677 INFO
2024-04-15 20:55:24,677 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:24,677 INFO
2024-04-15 20:55:24,677 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:24,677 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:25,099 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:25,103 DEBUG --- stdout ---
2024-04-15 20:55:25,103 DEBUG 3
2024-04-15 20:55:25,103 DEBUG --- stderr ---
2024-04-15 20:55:25,104 DEBUG
2024-04-15 20:55:25,104 INFO
2024-04-15 20:55:25,104 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:55:25,105 INFO
2024-04-15 20:55:25,105 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-15 20:55:25,105 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:55:25,567 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 20:55:25,567 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:25,568 DEBUG --- stdout ---
2024-04-15 20:55:25,568 DEBUG am-7d597cf7d6-8ctft am-7d597cf7d6-8vr8f am-7d597cf7d6-9wz9h
2024-04-15 20:55:25,568 DEBUG --- stderr ---
2024-04-15 20:55:25,568 DEBUG
2024-04-15 20:55:25,568 INFO
2024-04-15 20:55:25,568 INFO -------------- Check pod am-7d597cf7d6-8ctft is running --------------
2024-04-15 20:55:25,568 INFO
2024-04-15 20:55:25,568 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-8ctft -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:25,568 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:25,984 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:25,985 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:25,985 DEBUG --- stdout ---
2024-04-15 20:55:25,985 DEBUG Running
2024-04-15 20:55:25,985 DEBUG --- stderr ---
2024-04-15 20:55:25,985 DEBUG
2024-04-15 20:55:25,985 INFO
2024-04-15 20:55:25,985 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-8ctft -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:25,985 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:26,375 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:26,376 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:26,376 DEBUG --- stdout ---
2024-04-15 20:55:26,376 DEBUG true
2024-04-15 20:55:26,376 DEBUG --- stderr ---
2024-04-15 20:55:26,376 DEBUG
2024-04-15 20:55:26,376 INFO
2024-04-15 20:55:26,376 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8ctft --output jsonpath={.status.startTime}
2024-04-15 20:55:26,376 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:26,775 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:26,776 DEBUG --- stdout ---
2024-04-15 20:55:26,776 DEBUG 2024-04-13T22:40:34Z
2024-04-15 20:55:26,776 DEBUG --- stderr ---
2024-04-15 20:55:26,776 DEBUG
2024-04-15 20:55:26,777 INFO
2024-04-15 20:55:26,777 INFO ------- Check pod am-7d597cf7d6-8ctft filesystem is accessible -------
2024-04-15 20:55:27,634 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:27,635 INFO
2024-04-15 20:55:27,635 INFO ------------- Check pod am-7d597cf7d6-8ctft restart count -------------
2024-04-15 20:55:27,635 INFO
2024-04-15 20:55:27,635 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8ctft --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:27,635 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:28,035 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:28,036 DEBUG --- stdout ---
2024-04-15 20:55:28,036 DEBUG 0
2024-04-15 20:55:28,036 DEBUG --- stderr ---
2024-04-15 20:55:28,036 DEBUG
2024-04-15 20:55:28,036 INFO Pod am-7d597cf7d6-8ctft has been restarted 0 times.
2024-04-15 20:55:28,036 INFO
2024-04-15 20:55:28,036 INFO -------------- Check pod am-7d597cf7d6-8vr8f is running --------------
2024-04-15 20:55:28,036 INFO
2024-04-15 20:55:28,036 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-8vr8f -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:28,037 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:28,425 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:28,425 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:28,425 DEBUG --- stdout ---
2024-04-15 20:55:28,425 DEBUG Running
2024-04-15 20:55:28,425 DEBUG --- stderr ---
2024-04-15 20:55:28,425 DEBUG
2024-04-15 20:55:28,426 INFO
2024-04-15 20:55:28,426 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-8vr8f -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:28,426 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:28,847 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:28,847 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:28,847 DEBUG --- stdout ---
2024-04-15 20:55:28,847 DEBUG true
2024-04-15 20:55:28,847 DEBUG --- stderr ---
2024-04-15 20:55:28,847 DEBUG
2024-04-15 20:55:28,847 INFO
2024-04-15 20:55:28,847 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8vr8f --output jsonpath={.status.startTime}
2024-04-15 20:55:28,847 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:29,231 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:29,231 DEBUG --- stdout ---
2024-04-15 20:55:29,232 DEBUG 2024-04-13T22:40:34Z
2024-04-15 20:55:29,232 DEBUG --- stderr ---
2024-04-15 20:55:29,232 DEBUG
2024-04-15 20:55:29,232 INFO
2024-04-15 20:55:29,232 INFO ------- Check pod am-7d597cf7d6-8vr8f filesystem is accessible -------
2024-04-15 20:55:30,076 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:30,076 INFO
2024-04-15 20:55:30,076 INFO ------------- Check pod am-7d597cf7d6-8vr8f restart count -------------
2024-04-15 20:55:30,076 INFO
2024-04-15 20:55:30,076 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8vr8f --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:30,076 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:30,443 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:30,445 DEBUG --- stdout ---
2024-04-15 20:55:30,445 DEBUG 0
2024-04-15 20:55:30,446 DEBUG --- stderr ---
2024-04-15 20:55:30,446 DEBUG
2024-04-15 20:55:30,446 INFO Pod am-7d597cf7d6-8vr8f has been restarted 0 times.
2024-04-15 20:55:30,446 INFO
2024-04-15 20:55:30,446 INFO -------------- Check pod am-7d597cf7d6-9wz9h is running --------------
2024-04-15 20:55:30,446 INFO
2024-04-15 20:55:30,446 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-9wz9h -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:30,446 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:30,808 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:30,808 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:30,808 DEBUG --- stdout ---
2024-04-15 20:55:30,808 DEBUG Running
2024-04-15 20:55:30,808 DEBUG --- stderr ---
2024-04-15 20:55:30,808 DEBUG
2024-04-15 20:55:30,809 INFO
2024-04-15 20:55:30,809 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-9wz9h -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:30,809 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:31,196 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:31,197 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:31,197 DEBUG --- stdout ---
2024-04-15 20:55:31,197 DEBUG true
2024-04-15 20:55:31,197 DEBUG --- stderr ---
2024-04-15 20:55:31,197 DEBUG
2024-04-15 20:55:31,197 INFO
2024-04-15 20:55:31,197 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-9wz9h --output jsonpath={.status.startTime}
2024-04-15 20:55:31,197 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:31,576 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:31,576 DEBUG --- stdout ---
2024-04-15 20:55:31,576 DEBUG 2024-04-13T22:40:34Z
2024-04-15 20:55:31,576 DEBUG --- stderr ---
2024-04-15 20:55:31,576 DEBUG
2024-04-15 20:55:31,577 INFO
2024-04-15 20:55:31,577 INFO ------- Check pod am-7d597cf7d6-9wz9h filesystem is accessible -------
2024-04-15 20:55:32,422 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:32,423 INFO
2024-04-15 20:55:32,423 INFO ------------- Check pod am-7d597cf7d6-9wz9h restart count -------------
2024-04-15 20:55:32,423 INFO
2024-04-15 20:55:32,423 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-9wz9h --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:32,423 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:32,805 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:32,806 DEBUG --- stdout ---
2024-04-15 20:55:32,806 DEBUG 0
2024-04-15 20:55:32,806 DEBUG --- stderr ---
2024-04-15 20:55:32,806 DEBUG
2024-04-15 20:55:32,806 INFO Pod am-7d597cf7d6-9wz9h has been restarted 0 times.
2024-04-15 20:55:32,806 INFO
2024-04-15 20:55:32,806 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-15 20:55:32,806 INFO
2024-04-15 20:55:32,806 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:55:32,807 INFO
2024-04-15 20:55:32,807 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm --field-selector status.phase!=Failed get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-15 20:55:32,807 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:55:33,174 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:33,175 DEBUG --- stdout ---
2024-04-15 20:55:33,175 DEBUG amster-qv96p
2024-04-15 20:55:33,175 DEBUG --- stderr ---
2024-04-15 20:55:33,175 DEBUG
2024-04-15 20:55:33,175 INFO
2024-04-15 20:55:33,175 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-15 20:55:33,175 INFO
2024-04-15 20:55:33,175 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:33,176 INFO
2024-04-15 20:55:33,176 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:33,176 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:33,544 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:33,544 DEBUG --- stdout ---
2024-04-15 20:55:33,544 DEBUG 2
2024-04-15 20:55:33,544 DEBUG --- stderr ---
2024-04-15 20:55:33,544 DEBUG
2024-04-15 20:55:33,544 INFO
2024-04-15 20:55:33,544 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:55:33,545 INFO
2024-04-15 20:55:33,545 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=idm --output jsonpath={.items[*].metadata.name}` | grep 2
2024-04-15 20:55:33,545 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:55:33,932 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 20:55:33,933 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:33,933 DEBUG --- stdout ---
2024-04-15 20:55:33,933 DEBUG idm-b85f9d978-j5pbm idm-b85f9d978-rl76f
2024-04-15 20:55:33,933 DEBUG --- stderr ---
2024-04-15 20:55:33,933 DEBUG
2024-04-15 20:55:33,933 INFO
2024-04-15 20:55:33,933 INFO -------------- Check pod idm-b85f9d978-j5pbm is running --------------
2024-04-15 20:55:33,933 INFO
2024-04-15 20:55:33,933 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-j5pbm -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:33,933 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:34,296 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:34,296 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:34,296 DEBUG --- stdout ---
2024-04-15 20:55:34,296 DEBUG Running
2024-04-15 20:55:34,296 DEBUG --- stderr ---
2024-04-15 20:55:34,296 DEBUG
2024-04-15 20:55:34,297 INFO
2024-04-15 20:55:34,297 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-j5pbm -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:34,297 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:34,659 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:34,659 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:34,659 DEBUG --- stdout ---
2024-04-15 20:55:34,659 DEBUG true
2024-04-15 20:55:34,659 DEBUG --- stderr ---
2024-04-15 20:55:34,660 DEBUG
2024-04-15 20:55:34,660 INFO
2024-04-15 20:55:34,660 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-j5pbm --output jsonpath={.status.startTime}
2024-04-15 20:55:34,660 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:35,026 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:35,026 DEBUG --- stdout ---
2024-04-15 20:55:35,026 DEBUG 2024-04-13T22:40:34Z
2024-04-15 20:55:35,026 DEBUG --- stderr ---
2024-04-15 20:55:35,026 DEBUG
2024-04-15 20:55:35,027 INFO
2024-04-15 20:55:35,027 INFO ------- Check pod idm-b85f9d978-j5pbm filesystem is accessible -------
2024-04-15 20:55:35,854 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:35,854 INFO
2024-04-15 20:55:35,854 INFO ------------- Check pod idm-b85f9d978-j5pbm restart count -------------
2024-04-15 20:55:35,855 INFO
2024-04-15 20:55:35,855 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-j5pbm --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:35,855 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:36,246 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:36,246 DEBUG --- stdout ---
2024-04-15 20:55:36,247 DEBUG 0
2024-04-15 20:55:36,247 DEBUG --- stderr ---
2024-04-15 20:55:36,247 DEBUG
2024-04-15 20:55:36,247 INFO Pod idm-b85f9d978-j5pbm has been restarted 0 times.
2024-04-15 20:55:36,247 INFO
2024-04-15 20:55:36,247 INFO -------------- Check pod idm-b85f9d978-rl76f is running --------------
2024-04-15 20:55:36,247 INFO
2024-04-15 20:55:36,247 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-rl76f -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:36,247 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:36,636 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:36,636 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:36,636 DEBUG --- stdout ---
2024-04-15 20:55:36,637 DEBUG Running
2024-04-15 20:55:36,637 DEBUG --- stderr ---
2024-04-15 20:55:36,637 DEBUG
2024-04-15 20:55:36,637 INFO
2024-04-15 20:55:36,637 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-rl76f -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:36,637 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:37,025 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:37,026 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:37,026 DEBUG --- stdout ---
2024-04-15 20:55:37,026 DEBUG true
2024-04-15 20:55:37,026 DEBUG --- stderr ---
2024-04-15 20:55:37,026 DEBUG
2024-04-15 20:55:37,026 INFO
2024-04-15 20:55:37,026 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-rl76f --output jsonpath={.status.startTime}
2024-04-15 20:55:37,026 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:37,396 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:37,396 DEBUG --- stdout ---
2024-04-15 20:55:37,396 DEBUG 2024-04-13T22:40:34Z
2024-04-15 20:55:37,396 DEBUG --- stderr ---
2024-04-15 20:55:37,396 DEBUG
2024-04-15 20:55:37,397 INFO
2024-04-15 20:55:37,397 INFO ------- Check pod idm-b85f9d978-rl76f filesystem is accessible -------
2024-04-15 20:55:38,225 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:38,225 INFO
2024-04-15 20:55:38,226 INFO ------------- Check pod idm-b85f9d978-rl76f restart count -------------
2024-04-15 20:55:38,226 INFO
2024-04-15 20:55:38,226 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-rl76f --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:38,226 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:38,598 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:38,598 DEBUG --- stdout ---
2024-04-15 20:55:38,599 DEBUG 0
2024-04-15 20:55:38,599 DEBUG --- stderr ---
2024-04-15 20:55:38,599 DEBUG
2024-04-15 20:55:38,599 INFO Pod idm-b85f9d978-rl76f has been restarted 0 times.
2024-04-15 20:55:38,599 INFO
2024-04-15 20:55:38,599 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-15 20:55:38,599 INFO
2024-04-15 20:55:38,599 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:38,599 INFO
2024-04-15 20:55:38,599 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:38,599 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:38,918 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:38,918 DEBUG --- stdout ---
2024-04-15 20:55:38,918 DEBUG 1
2024-04-15 20:55:38,918 DEBUG --- stderr ---
2024-04-15 20:55:38,918 DEBUG
2024-04-15 20:55:38,918 INFO
2024-04-15 20:55:38,918 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:55:38,919 INFO
2024-04-15 20:55:38,919 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-15 20:55:38,919 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:55:39,263 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 20:55:39,263 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:39,263 DEBUG --- stdout ---
2024-04-15 20:55:39,264 DEBUG end-user-ui-58878d7668-zg4h2
2024-04-15 20:55:39,264 DEBUG --- stderr ---
2024-04-15 20:55:39,264 DEBUG
2024-04-15 20:55:39,264 INFO
2024-04-15 20:55:39,264 INFO ---------- Check pod end-user-ui-58878d7668-zg4h2 is running ----------
2024-04-15 20:55:39,264 INFO
2024-04-15 20:55:39,264 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods end-user-ui-58878d7668-zg4h2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:39,264 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:39,627 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:39,627 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:39,627 DEBUG --- stdout ---
2024-04-15 20:55:39,627 DEBUG Running
2024-04-15 20:55:39,627 DEBUG --- stderr ---
2024-04-15 20:55:39,627 DEBUG
2024-04-15 20:55:39,628 INFO
2024-04-15 20:55:39,628 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods end-user-ui-58878d7668-zg4h2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:39,628 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:39,956 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:39,956 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:39,956 DEBUG --- stdout ---
2024-04-15 20:55:39,957 DEBUG true
2024-04-15 20:55:39,957 DEBUG --- stderr ---
2024-04-15 20:55:39,957 DEBUG
2024-04-15 20:55:39,957 INFO
2024-04-15 20:55:39,957 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod end-user-ui-58878d7668-zg4h2 --output jsonpath={.status.startTime}
2024-04-15 20:55:39,957 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:40,260 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:40,260 DEBUG --- stdout ---
2024-04-15 20:55:40,260 DEBUG 2024-04-13T22:41:43Z
2024-04-15 20:55:40,260 DEBUG --- stderr ---
2024-04-15 20:55:40,260 DEBUG
2024-04-15 20:55:40,261 INFO
2024-04-15 20:55:40,261 INFO --- Check pod end-user-ui-58878d7668-zg4h2 filesystem is accessible ---
2024-04-15 20:55:41,021 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:41,021 INFO
2024-04-15 20:55:41,021 INFO -------- Check pod end-user-ui-58878d7668-zg4h2 restart count --------
2024-04-15 20:55:41,021 INFO
2024-04-15 20:55:41,021 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod end-user-ui-58878d7668-zg4h2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:41,021 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:41,351 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:41,352 DEBUG --- stdout ---
2024-04-15 20:55:41,352 DEBUG 0
2024-04-15 20:55:41,352 DEBUG --- stderr ---
2024-04-15 20:55:41,352 DEBUG
2024-04-15 20:55:41,352 INFO Pod end-user-ui-58878d7668-zg4h2 has been restarted 0 times.
2024-04-15 20:55:41,352 INFO
2024-04-15 20:55:41,353 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-15 20:55:41,353 INFO
2024-04-15 20:55:41,353 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:41,353 INFO
2024-04-15 20:55:41,353 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:41,353 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:41,699 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:41,700 DEBUG --- stdout ---
2024-04-15 20:55:41,700 DEBUG 1
2024-04-15 20:55:41,700 DEBUG --- stderr ---
2024-04-15 20:55:41,700 DEBUG
2024-04-15 20:55:41,700 INFO
2024-04-15 20:55:41,700 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:55:41,700 INFO
2024-04-15 20:55:41,700 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-15 20:55:41,700 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:55:42,034 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 20:55:42,034 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:42,034 DEBUG --- stdout ---
2024-04-15 20:55:42,034 DEBUG login-ui-5fdf49fc66-2wnx7
2024-04-15 20:55:42,035 DEBUG --- stderr ---
2024-04-15 20:55:42,035 DEBUG
2024-04-15 20:55:42,035 INFO
2024-04-15 20:55:42,035 INFO ----------- Check pod login-ui-5fdf49fc66-2wnx7 is running -----------
2024-04-15 20:55:42,035 INFO
2024-04-15 20:55:42,035 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods login-ui-5fdf49fc66-2wnx7 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:42,035 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:42,363 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:42,363 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:42,363 DEBUG --- stdout ---
2024-04-15 20:55:42,363 DEBUG Running
2024-04-15 20:55:42,363 DEBUG --- stderr ---
2024-04-15 20:55:42,363 DEBUG
2024-04-15 20:55:42,363 INFO
2024-04-15 20:55:42,363 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods login-ui-5fdf49fc66-2wnx7 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:42,363 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:42,667 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:42,667 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:42,667 DEBUG --- stdout ---
2024-04-15 20:55:42,667 DEBUG true
2024-04-15 20:55:42,667 DEBUG --- stderr ---
2024-04-15 20:55:42,667 DEBUG
2024-04-15 20:55:42,668 INFO
2024-04-15 20:55:42,668 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod login-ui-5fdf49fc66-2wnx7 --output jsonpath={.status.startTime}
2024-04-15 20:55:42,668 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:42,972 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:42,973 DEBUG --- stdout ---
2024-04-15 20:55:42,973 DEBUG 2024-04-13T22:41:43Z
2024-04-15 20:55:42,973 DEBUG --- stderr ---
2024-04-15 20:55:42,973 DEBUG
2024-04-15 20:55:42,974 INFO
2024-04-15 20:55:42,974 INFO ---- Check pod login-ui-5fdf49fc66-2wnx7 filesystem is accessible ----
2024-04-15 20:55:43,784 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:43,784 INFO
2024-04-15 20:55:43,784 INFO ---------- Check pod login-ui-5fdf49fc66-2wnx7 restart count ----------
2024-04-15 20:55:43,785 INFO
2024-04-15 20:55:43,785 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod login-ui-5fdf49fc66-2wnx7 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:43,785 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:44,124 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:44,124 DEBUG --- stdout ---
2024-04-15 20:55:44,124 DEBUG 0
2024-04-15 20:55:44,124 DEBUG --- stderr ---
2024-04-15 20:55:44,124 DEBUG
2024-04-15 20:55:44,124 INFO Pod login-ui-5fdf49fc66-2wnx7 has been restarted 0 times.
2024-04-15 20:55:44,124 INFO
2024-04-15 20:55:44,124 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-15 20:55:44,124 INFO
2024-04-15 20:55:44,124 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:44,125 INFO
2024-04-15 20:55:44,125 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:44,125 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:44,436 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:44,436 DEBUG --- stdout ---
2024-04-15 20:55:44,436 DEBUG 1
2024-04-15 20:55:44,436 DEBUG --- stderr ---
2024-04-15 20:55:44,436 DEBUG
2024-04-15 20:55:44,437 INFO
2024-04-15 20:55:44,437 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:55:44,437 INFO
2024-04-15 20:55:44,437 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-15 20:55:44,437 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:55:44,760 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 20:55:44,760 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:44,760 DEBUG --- stdout ---
2024-04-15 20:55:44,760 DEBUG admin-ui-c4c7bf584-mbzlq
2024-04-15 20:55:44,760 DEBUG --- stderr ---
2024-04-15 20:55:44,761 DEBUG
2024-04-15 20:55:44,761 INFO
2024-04-15 20:55:44,761 INFO ------------ Check pod admin-ui-c4c7bf584-mbzlq is running ------------
2024-04-15 20:55:44,761 INFO
2024-04-15 20:55:44,761 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods admin-ui-c4c7bf584-mbzlq -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 20:55:44,761 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:45,088 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:45,088 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:45,088 DEBUG --- stdout ---
2024-04-15 20:55:45,088 DEBUG Running
2024-04-15 20:55:45,088 DEBUG --- stderr ---
2024-04-15 20:55:45,088 DEBUG
2024-04-15 20:55:45,089 INFO
2024-04-15 20:55:45,089 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods admin-ui-c4c7bf584-mbzlq -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 20:55:45,089 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 20:55:45,416 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:45,417 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:45,417 DEBUG --- stdout ---
2024-04-15 20:55:45,417 DEBUG true
2024-04-15 20:55:45,417 DEBUG --- stderr ---
2024-04-15 20:55:45,417 DEBUG
2024-04-15 20:55:45,417 INFO
2024-04-15 20:55:45,417 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod admin-ui-c4c7bf584-mbzlq --output jsonpath={.status.startTime}
2024-04-15 20:55:45,417 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:45,759 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:45,759 DEBUG --- stdout ---
2024-04-15 20:55:45,760 DEBUG 2024-04-13T22:41:42Z
2024-04-15 20:55:45,760 DEBUG --- stderr ---
2024-04-15 20:55:45,760 DEBUG
2024-04-15 20:55:45,761 INFO
2024-04-15 20:55:45,761 INFO ----- Check pod admin-ui-c4c7bf584-mbzlq filesystem is accessible -----
2024-04-15 20:55:46,550 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:46,550 INFO
2024-04-15 20:55:46,551 INFO ---------- Check pod admin-ui-c4c7bf584-mbzlq restart count ----------
2024-04-15 20:55:46,551 INFO
2024-04-15 20:55:46,551 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod admin-ui-c4c7bf584-mbzlq --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 20:55:46,551 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:46,881 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:46,881 DEBUG --- stdout ---
2024-04-15 20:55:46,881 DEBUG 0
2024-04-15 20:55:46,881 DEBUG --- stderr ---
2024-04-15 20:55:46,881 DEBUG
2024-04-15 20:55:46,882 INFO Pod admin-ui-c4c7bf584-mbzlq has been restarted 0 times.
2024-04-15 20:55:46,882 INFO
2024-04-15 20:55:46,882 INFO ***************************** Checking DS-CTS component is running *****************************
2024-04-15 20:55:46,882 INFO
2024-04-15 20:55:46,882 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:46,882 INFO
2024-04-15 20:55:46,882 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:46,883 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:47,281 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:47,281 DEBUG --- stdout ---
2024-04-15 20:55:47,281 DEBUG 3
2024-04-15 20:55:47,281 DEBUG --- stderr ---
2024-04-15 20:55:47,281 DEBUG
2024-04-15 20:55:47,281 INFO
2024-04-15 20:55:47,282 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-15 20:55:47,282 INFO
2024-04-15 20:55:47,282 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-cts --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3"
2024-04-15 20:55:47,282 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 20:55:47,693 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:47,694 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:47,694 DEBUG --- stdout ---
2024-04-15 20:55:47,694 DEBUG current:3 ready:3 replicas:3
2024-04-15 20:55:47,694 DEBUG --- stderr ---
2024-04-15 20:55:47,694 DEBUG
2024-04-15 20:55:47,694 INFO
2024-04-15 20:55:47,694 INFO *************************** Checking DS-IDREPO component is running ***************************
2024-04-15 20:55:47,694 INFO
2024-04-15 20:55:47,695 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:47,695 INFO
2024-04-15 20:55:47,695 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:47,695 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:48,095 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:48,096 DEBUG --- stdout ---
2024-04-15 20:55:48,096 DEBUG 3
2024-04-15 20:55:48,096 DEBUG --- stderr ---
2024-04-15 20:55:48,096 DEBUG
2024-04-15 20:55:48,096 INFO
2024-04-15 20:55:48,096 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-15 20:55:48,096 INFO
2024-04-15 20:55:48,097 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-idrepo --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3"
2024-04-15 20:55:48,097 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 20:55:48,515 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:48,515 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:48,515 DEBUG --- stdout ---
2024-04-15 20:55:48,515 DEBUG current:3 ready:3 replicas:3
2024-04-15 20:55:48,515 DEBUG --- stderr ---
2024-04-15 20:55:48,515 DEBUG
2024-04-15 20:55:48,515 INFO
2024-04-15 20:55:48,515 INFO ******************************* Checking AM component is running *******************************
2024-04-15 20:55:48,515 INFO
2024-04-15 20:55:48,515 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:48,516 INFO
2024-04-15 20:55:48,516 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:48,516 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:48,916 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:48,917 DEBUG --- stdout ---
2024-04-15 20:55:48,917 DEBUG 3
2024-04-15 20:55:48,917 DEBUG --- stderr ---
2024-04-15 20:55:48,917 DEBUG
2024-04-15 20:55:48,917 INFO
2024-04-15 20:55:48,917 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-15 20:55:48,917 INFO
2024-04-15 20:55:48,917 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments am --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:3 replicas:3"
2024-04-15 20:55:48,918 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 20:55:49,332 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:49,333 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:49,333 DEBUG --- stdout ---
2024-04-15 20:55:49,333 DEBUG ready:3 replicas:3
2024-04-15 20:55:49,333 DEBUG --- stderr ---
2024-04-15 20:55:49,333 DEBUG
2024-04-15 20:55:49,333 INFO
2024-04-15 20:55:49,333 INFO ***************************** Checking AMSTER component is running *****************************
2024-04-15 20:55:49,333 INFO
2024-04-15 20:55:49,333 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:49,333 INFO
2024-04-15 20:55:49,333 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-15 20:55:49,333 INFO
2024-04-15 20:55:49,333 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get jobs amster --output jsonpath="{.status.succeeded}" | grep "1"
2024-04-15 20:55:49,333 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 20:55:49,705 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:49,705 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:49,705 DEBUG --- stdout ---
2024-04-15 20:55:49,705 DEBUG 1
2024-04-15 20:55:49,706 DEBUG --- stderr ---
2024-04-15 20:55:49,706 DEBUG
2024-04-15 20:55:49,706 INFO
2024-04-15 20:55:49,706 INFO ****************************** Checking IDM component is running ******************************
2024-04-15 20:55:49,706 INFO
2024-04-15 20:55:49,706 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:49,706 INFO
2024-04-15 20:55:49,706 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:49,706 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:50,112 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:50,112 DEBUG --- stdout ---
2024-04-15 20:55:50,112 DEBUG 2
2024-04-15 20:55:50,112 DEBUG --- stderr ---
2024-04-15 20:55:50,112 DEBUG
2024-04-15 20:55:50,112 INFO
2024-04-15 20:55:50,112 INFO -------------- Waiting for 2 expected pod(s) to be ready --------------
2024-04-15 20:55:50,113 INFO
2024-04-15 20:55:50,114 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment idm --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:2 replicas:2"
2024-04-15 20:55:50,114 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 20:55:50,543 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:50,543 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:50,543 DEBUG --- stdout ---
2024-04-15 20:55:50,543 DEBUG ready:2 replicas:2
2024-04-15 20:55:50,543 DEBUG --- stderr ---
2024-04-15 20:55:50,543 DEBUG
2024-04-15 20:55:50,543 INFO
2024-04-15 20:55:50,543 INFO ************************** Checking END-USER-UI component is running **************************
2024-04-15 20:55:50,543 INFO
2024-04-15 20:55:50,543 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:50,544 INFO
2024-04-15 20:55:50,544 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:50,544 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:50,879 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:50,879 DEBUG --- stdout ---
2024-04-15 20:55:50,879 DEBUG 1
2024-04-15 20:55:50,879 DEBUG --- stderr ---
2024-04-15 20:55:50,879 DEBUG
2024-04-15 20:55:50,879 INFO
2024-04-15 20:55:50,879 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-15 20:55:50,879 INFO
2024-04-15 20:55:50,879 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments end-user-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-15 20:55:50,880 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 20:55:51,235 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:51,236 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:51,236 DEBUG --- stdout ---
2024-04-15 20:55:51,236 DEBUG ready:1 replicas:1
2024-04-15 20:55:51,236 DEBUG --- stderr ---
2024-04-15 20:55:51,236 DEBUG
2024-04-15 20:55:51,236 INFO
2024-04-15 20:55:51,236 INFO **************************** Checking LOGIN-UI component is running ****************************
2024-04-15 20:55:51,236 INFO
2024-04-15 20:55:51,236 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:51,236 INFO
2024-04-15 20:55:51,236 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:51,236 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:51,569 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:51,569 DEBUG --- stdout ---
2024-04-15 20:55:51,571 DEBUG 1
2024-04-15 20:55:51,571 DEBUG --- stderr ---
2024-04-15 20:55:51,571 DEBUG
2024-04-15 20:55:51,571 INFO
2024-04-15 20:55:51,571 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-15 20:55:51,572 INFO
2024-04-15 20:55:51,572 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments login-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-15 20:55:51,572 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 20:55:51,929 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:51,929 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:51,930 DEBUG --- stdout ---
2024-04-15 20:55:51,930 DEBUG ready:1 replicas:1
2024-04-15 20:55:51,930 DEBUG --- stderr ---
2024-04-15 20:55:51,930 DEBUG
2024-04-15 20:55:51,930 INFO
2024-04-15 20:55:51,930 INFO **************************** Checking ADMIN-UI component is running ****************************
2024-04-15 20:55:51,930 INFO
2024-04-15 20:55:51,930 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:55:51,930 INFO
2024-04-15 20:55:51,930 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:55:51,930 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:55:52,287 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:52,287 DEBUG --- stdout ---
2024-04-15 20:55:52,287 DEBUG 1
2024-04-15 20:55:52,287 DEBUG --- stderr ---
2024-04-15 20:55:52,287 DEBUG
2024-04-15 20:55:52,288 INFO
2024-04-15 20:55:52,288 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-15 20:55:52,288 INFO
2024-04-15 20:55:52,288 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments admin-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-15 20:55:52,288 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 20:55:52,613 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 20:55:52,613 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:52,613 DEBUG --- stdout ---
2024-04-15 20:55:52,613 DEBUG ready:1 replicas:1
2024-04-15 20:55:52,614 DEBUG --- stderr ---
2024-04-15 20:55:52,614 DEBUG
2024-04-15 20:55:52,614 INFO
2024-04-15 20:55:52,614 INFO ****************************** Livecheck stage: After deployment ******************************
2024-04-15 20:55:52,614 INFO
2024-04-15 20:55:52,614 INFO ---------------------- Running DS-CTS livecheck ----------------------
2024-04-15 20:55:52,614 INFO Livecheck to ds-cts-0
2024-04-15 20:55:52,614 INFO
2024-04-15 20:55:52,614 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.dirmanager\.pw}"
2024-04-15 20:55:52,614 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-15 20:55:52,944 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:52,944 DEBUG --- stdout ---
2024-04-15 20:55:52,944 DEBUG YjJFdDlYdUlSY0J3bnRZVWk3SjdmN09BU1M5WWJRcXg=
2024-04-15 20:55:52,944 DEBUG --- stderr ---
2024-04-15 20:55:52,944 DEBUG
2024-04-15 20:55:52,945 INFO
2024-04-15 20:55:52,945 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 20:55:54,727 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:54,727 DEBUG --- stdout ---
2024-04-15 20:55:54,727 DEBUG dn:
alive: true
2024-04-15 20:55:54,727 DEBUG --- stderr ---
2024-04-15 20:55:54,728 DEBUG
2024-04-15 20:55:54,728 INFO
2024-04-15 20:55:54,728 INFO Livecheck to ds-cts-1
2024-04-15 20:55:54,728 INFO
2024-04-15 20:55:54,728 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 20:55:56,490 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:56,490 DEBUG --- stdout ---
2024-04-15 20:55:56,490 DEBUG dn:
alive: true
2024-04-15 20:55:56,490 DEBUG --- stderr ---
2024-04-15 20:55:56,490 DEBUG
2024-04-15 20:55:56,490 INFO
2024-04-15 20:55:56,490 INFO Livecheck to ds-cts-2
2024-04-15 20:55:56,491 INFO
2024-04-15 20:55:56,491 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 20:55:58,233 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:55:58,234 DEBUG --- stdout ---
2024-04-15 20:55:58,234 DEBUG dn:
alive: true
2024-04-15 20:55:58,234 DEBUG --- stderr ---
2024-04-15 20:55:58,234 DEBUG
2024-04-15 20:55:58,234 INFO
2024-04-15 20:55:58,234 INFO
2024-04-15 20:55:58,234 INFO --------------------- Running DS-IDREPO livecheck ---------------------
2024-04-15 20:55:58,234 INFO Livecheck to ds-idrepo-0
2024-04-15 20:55:58,234 INFO
2024-04-15 20:55:58,234 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.dirmanager\.pw}"
2024-04-15 20:55:58,234 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-15 20:55:58,533 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:55:58,533 DEBUG --- stdout ---
2024-04-15 20:55:58,533 DEBUG YjJFdDlYdUlSY0J3bnRZVWk3SjdmN09BU1M5WWJRcXg=
2024-04-15 20:55:58,533 DEBUG --- stderr ---
2024-04-15 20:55:58,533 DEBUG
2024-04-15 20:55:58,533 INFO
2024-04-15 20:55:58,534 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 20:56:00,273 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:56:00,273 DEBUG --- stdout ---
2024-04-15 20:56:00,273 DEBUG dn:
alive: true
2024-04-15 20:56:00,273 DEBUG --- stderr ---
2024-04-15 20:56:00,273 DEBUG
2024-04-15 20:56:00,273 INFO
2024-04-15 20:56:00,273 INFO Livecheck to ds-idrepo-1
2024-04-15 20:56:00,273 INFO
2024-04-15 20:56:00,274 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 20:56:02,023 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:56:02,025 DEBUG --- stdout ---
2024-04-15 20:56:02,025 DEBUG dn:
alive: true
2024-04-15 20:56:02,025 DEBUG --- stderr ---
2024-04-15 20:56:02,025 DEBUG
2024-04-15 20:56:02,025 INFO
2024-04-15 20:56:02,026 INFO Livecheck to ds-idrepo-2
2024-04-15 20:56:02,026 INFO
2024-04-15 20:56:02,027 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 20:56:03,803 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 20:56:03,803 DEBUG --- stdout ---
2024-04-15 20:56:03,803 DEBUG dn:
alive: true
2024-04-15 20:56:03,803 DEBUG --- stderr ---
2024-04-15 20:56:03,803 DEBUG
2024-04-15 20:56:03,803 INFO
2024-04-15 20:56:03,803 INFO
2024-04-15 20:56:03,803 INFO ------------------------ Running AM livecheck ------------------------
2024-04-15 20:56:03,803 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready
2024-04-15 20:56:03,804 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready"
2024-04-15 20:56:04,102 INFO [http_cmd]: http status code OK
2024-04-15 20:56:04,102 DEBUG --- status code ---
2024-04-15 20:56:04,102 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:04,102 DEBUG --- http response ---
2024-04-15 20:56:04,102 DEBUG
2024-04-15 20:56:04,103 INFO
2024-04-15 20:56:04,103 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.AM_PASSWORDS_AMADMIN_CLEAR}"
2024-04-15 20:56:04,103 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-15 20:56:04,447 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:04,447 DEBUG --- stdout ---
2024-04-15 20:56:04,447 DEBUG WU53YjE0T1M1SDNjODNsV0txOWpibG5U
2024-04-15 20:56:04,447 DEBUG --- stderr ---
2024-04-15 20:56:04,447 DEBUG
2024-04-15 20:56:04,448 INFO Authenticate user amadmin via REST
2024-04-15 20:56:04,448 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/"
2024-04-15 20:56:04,725 INFO [http_cmd]: http status code OK
2024-04-15 20:56:04,725 DEBUG --- status code ---
2024-04-15 20:56:04,725 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:04,725 DEBUG --- http response ---
2024-04-15 20:56:04,725 DEBUG {
"tokenId": "RUGpUmL5-8CqWP-ynrPyigTdLEw.*AAJTSQACMDIAAlNLABxrajk0MngvMHhwWS9zaThpdXRGQitSaEYwRzA9AAR0eXBlAANDVFMAAlMxAAIwMQ..*",
"successUrl": "/am/console",
"realm": "/"
}
2024-04-15 20:56:04,730 INFO
2024-04-15 20:56:04,731 INFO ---------------------- Running AMSTER livecheck ----------------------
2024-04-15 20:56:04,731 INFO
2024-04-15 20:56:04,731 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:56:04,732 INFO
2024-04-15 20:56:04,732 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm --field-selector status.phase!=Failed get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-15 20:56:04,732 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:56:05,118 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:05,119 DEBUG --- stdout ---
2024-04-15 20:56:05,119 DEBUG amster-qv96p
2024-04-15 20:56:05,119 DEBUG --- stderr ---
2024-04-15 20:56:05,119 DEBUG
2024-04-15 20:56:05,817 INFO Amster livecheck is passed
2024-04-15 20:56:05,818 INFO
2024-04-15 20:56:05,818 INFO ------------------------ Running IDM livecheck ------------------------
2024-04-15 20:56:05,818 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/ping
2024-04-15 20:56:05,818 INFO
2024-04-15 20:56:05,818 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.OPENIDM_ADMIN_PASSWORD}"
2024-04-15 20:56:05,819 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-15 20:56:06,142 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:06,142 DEBUG --- stdout ---
2024-04-15 20:56:06,142 DEBUG MVVYcW1nZmVQQjROSGdoYkdjdWFzUlhD
2024-04-15 20:56:06,142 DEBUG --- stderr ---
2024-04-15 20:56:06,142 DEBUG
2024-04-15 20:56:06,143 INFO Set admin password: 1UXqmgfePB4NHghbGcuasRXC
2024-04-15 20:56:06,143 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/ping"
2024-04-15 20:56:06,360 INFO [http_cmd]: http status code OK
2024-04-15 20:56:06,360 DEBUG --- status code ---
2024-04-15 20:56:06,360 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:06,360 DEBUG --- http response ---
2024-04-15 20:56:06,360 DEBUG {
"_id": "",
"_rev": "",
"shortDesc": "OpenIDM ready",
"state": "ACTIVE_READY"
}
2024-04-15 20:56:06,360 INFO Create new LoginSession for user "amadmin"
2024-04-15 20:56:06,360 INFO [LoginSession] Get "access_token" token for user "amadmin"
2024-04-15 20:56:06,360 INFO [LoginSession] Obtaining new "access_token" token from server because it was not obtained yet for the user
2024-04-15 20:56:06,360 INFO [LoginSession] Get "token_id" token for user "amadmin"
2024-04-15 20:56:06,360 INFO [LoginSession] Obtaining new "token_id" token from server because it was not obtained yet for the user
2024-04-15 20:56:06,361 INFO Authenticate user amadmin via REST
2024-04-15 20:56:06,361 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/"
2024-04-15 20:56:06,636 INFO [http_cmd]: http status code OK
2024-04-15 20:56:06,636 DEBUG --- status code ---
2024-04-15 20:56:06,636 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:06,636 DEBUG --- http response ---
2024-04-15 20:56:06,636 DEBUG {
"tokenId": "WiaYtJRX04gYYFqrE6Ghr7H8p6g.*AAJTSQACMDIAAlNLABxOTkNYRThUQkh1WUxaODNiOWN5QmxkSXFOSUU9AAR0eXBlAANDVFMAAlMxAAIwMQ..*",
"successUrl": "/am/console",
"realm": "/"
}
2024-04-15 20:56:06,636 INFO [LoginSession] Obtaining session info
2024-04-15 20:56:06,637 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/realms/root/sessions?_action=getSessionInfo"
2024-04-15 20:56:06,859 INFO [http_cmd]: http status code OK
2024-04-15 20:56:06,859 DEBUG --- status code ---
2024-04-15 20:56:06,860 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:06,860 DEBUG --- http response ---
2024-04-15 20:56:06,860 DEBUG {
"username": "amadmin",
"universalId": "id=amadmin,ou=user,ou=am-config",
"realm": "/",
"latestAccessTime": "2024-04-15T20:56:06Z",
"maxIdleExpirationTime": "2024-04-15T20:59:06Z",
"maxSessionExpirationTime": "2024-04-15T20:59:05Z",
"properties": {
"AMCtxId": "b5eef5cc-b013-4174-8d27-302dd8ef0203-1146795227"
}
}
2024-04-15 20:56:06,866 INFO Obtain Oauth2 authz code via REST
2024-04-15 20:56:06,866 DEBUG Oauth2 authorize for access token
2024-04-15 20:56:06,867 INFO [http_cmd]: curl --request POST --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=WiaYtJRX04gYYFqrE6Ghr7H8p6g.*AAJTSQACMDIAAlNLABxOTkNYRThUQkh1WUxaODNiOWN5QmxkSXFOSUU9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" --cookie "route=1713214567.544.49085.350152|f60edb382037eb2df1e800d563ad78a7" --data '{"decision": "Allow", "csrf": "WiaYtJRX04gYYFqrE6Ghr7H8p6g.*AAJTSQACMDIAAlNLABxOTkNYRThUQkh1WUxaODNiOWN5QmxkSXFOSUU9AAR0eXBlAANDVFMAAlMxAAIwMQ..*"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/am/oauth2/authorize?client_id=smokeclient&scope=fr:idm:*&redirect_uri=https://fake.com&response_type=code&realm=/"
2024-04-15 20:56:07,105 INFO [http_cmd]: http status code OK
2024-04-15 20:56:07,106 DEBUG --- status code ---
2024-04-15 20:56:07,106 DEBUG http status code is 302 (expected 302)
2024-04-15 20:56:07,106 DEBUG --- http response ---
2024-04-15 20:56:07,107 DEBUG {'Date': 'Mon, 15 Apr 2024 20:56:07 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store', 'Location': 'https://fake.com?code=9wD0ca0j3ZyURV11k0wny450i5M&iss=https%3A%2F%2Fxlou.iam.xlou-cdm.engineeringpit.com%2Fam%2Foauth2&client_id=smokeclient', 'Pragma': 'no-cache', 'Set-Cookie': 'OAUTH_REQUEST_ATTRIBUTES=DELETED; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Path=/; Secure; HttpOnly; SameSite=none', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains'}
2024-04-15 20:56:07,107 DEBUG Oauth2 exchange authorize code for access token
2024-04-15 20:56:07,108 INFO [http_cmd]: curl -L --request POST --data '{"grant_type": "authorization_code", "redirect_uri": "https://fake.com", "client_id": "smokeclient", "code": "9wD0ca0j3ZyURV11k0wny450i5M"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/am/oauth2/access_token?realm=/"
2024-04-15 20:56:07,191 INFO [http_cmd]: http status code OK
2024-04-15 20:56:07,191 DEBUG --- status code ---
2024-04-15 20:56:07,191 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:07,191 DEBUG --- http response ---
2024-04-15 20:56:07,191 DEBUG {
"access_token": "1udGVJBcBFJ1aq3Zg-MvgvsZ3bo",
"scope": "fr:idm:*",
"token_type": "Bearer",
"expires_in": 3599
}
2024-04-15 20:56:07,193 INFO Additional check to sync/queue to ensure sync with DS is ok (see LODESTAR-1245)
2024-04-15 20:56:07,194 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/sync/queue?_queryFilter=true&_pageSize=1"
2024-04-15 20:56:07,426 INFO [http_cmd]: http status code OK
2024-04-15 20:56:07,427 DEBUG --- status code ---
2024-04-15 20:56:07,427 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:07,427 DEBUG --- http response ---
2024-04-15 20:56:07,427 DEBUG {
"result": [],
"resultCount": 0,
"pagedResultsCookie": null,
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}
2024-04-15 20:56:07,430 INFO Try to create, query and delete a user
2024-04-15 20:56:07,431 INFO Create user idm_unmbc8xoik in IDM
2024-04-15 20:56:07,431 INFO [http_cmd]: curl --header "Authorization: Bearer 1udGVJBcBFJ1aq3Zg-MvgvsZ3bo" --header "Content-Type: application/json" --insecure -L --request POST --data '{"userName": "idm_unmbc8xoik", "givenName": "Peter", "sn": "Parker", "password": "T35tr0ck123", "telephoneNumber": "6669876987", "description": "IDM Test User", "roles": [], "mail": "idm_unmbc8xoik@forgerock.com"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user?_action=create"
2024-04-15 20:56:07,676 INFO [http_cmd]: http status code OK
2024-04-15 20:56:07,676 DEBUG --- status code ---
2024-04-15 20:56:07,676 DEBUG http status code is 201 (expected [200, 201])
2024-04-15 20:56:07,676 DEBUG --- http response ---
2024-04-15 20:56:07,677 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-15 20:56:07,677 DEBUG ----- output -----
2024-04-15 20:56:07,677 DEBUG {
2024-04-15 20:56:07,677 DEBUG "_id": "a7436859-9248-43b8-9bc5-e487f5ec08ac",
2024-04-15 20:56:07,677 DEBUG "_rev": "d2dd67f6-c775-4646-8c43-6112461729ba-1066747",
2024-04-15 20:56:07,677 DEBUG "country": null,
2024-04-15 20:56:07,677 DEBUG "mail": "idm_unmbc8xoik@forgerock.com",
2024-04-15 20:56:07,677 DEBUG "memberOfOrgIDs": [],
2024-04-15 20:56:07,677 DEBUG "postalCode": null,
2024-04-15 20:56:07,677 DEBUG "profileImage": null,
2024-04-15 20:56:07,677 DEBUG "expireAccount": null,
2024-04-15 20:56:07,677 DEBUG "activeDate": null,
2024-04-15 20:56:07,677 DEBUG
2024-04-15 20:56:07,677 DEBUG
2024-04-15 20:56:07,677 DEBUG
2024-04-15 20:56:07,677 DEBUG [...]
2024-04-15 20:56:07,677 DEBUG
2024-04-15 20:56:07,677 DEBUG
2024-04-15 20:56:07,677 DEBUG
2024-04-15 20:56:07,677 DEBUG "aliasList": [],
2024-04-15 20:56:07,677 DEBUG "kbaInfo": [],
2024-04-15 20:56:07,677 DEBUG "inactiveDate": null,
2024-04-15 20:56:07,677 DEBUG "sn": "Parker",
2024-04-15 20:56:07,677 DEBUG "preferences": null,
2024-04-15 20:56:07,677 DEBUG "userName": "idm_unmbc8xoik",
2024-04-15 20:56:07,677 DEBUG "passwordLastChangedTime": "2024-04-15T20:56:07.465Z",
2024-04-15 20:56:07,677 DEBUG "effectiveRoles": [],
2024-04-15 20:56:07,677 DEBUG "activateAccount": null
2024-04-15 20:56:07,677 DEBUG }
2024-04-15 20:56:07,678 INFO Query IDM to verify UserID a7436859-9248-43b8-9bc5-e487f5ec08ac is visible
2024-04-15 20:56:07,678 INFO [http_cmd]: curl --insecure -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user/a7436859-9248-43b8-9bc5-e487f5ec08ac"
2024-04-15 20:56:07,885 INFO [http_cmd]: http status code OK
2024-04-15 20:56:07,886 DEBUG --- status code ---
2024-04-15 20:56:07,886 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:07,886 DEBUG --- http response ---
2024-04-15 20:56:07,886 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-15 20:56:07,886 DEBUG ----- output -----
2024-04-15 20:56:07,886 DEBUG {
2024-04-15 20:56:07,886 DEBUG "_id": "a7436859-9248-43b8-9bc5-e487f5ec08ac",
2024-04-15 20:56:07,886 DEBUG "_rev": "d2dd67f6-c775-4646-8c43-6112461729ba-1066747",
2024-04-15 20:56:07,886 DEBUG "country": null,
2024-04-15 20:56:07,886 DEBUG "mail": "idm_unmbc8xoik@forgerock.com",
2024-04-15 20:56:07,886 DEBUG "memberOfOrgIDs": [],
2024-04-15 20:56:07,886 DEBUG "postalCode": null,
2024-04-15 20:56:07,886 DEBUG "profileImage": null,
2024-04-15 20:56:07,886 DEBUG "expireAccount": null,
2024-04-15 20:56:07,886 DEBUG "activeDate": null,
2024-04-15 20:56:07,886 DEBUG
2024-04-15 20:56:07,886 DEBUG
2024-04-15 20:56:07,886 DEBUG
2024-04-15 20:56:07,886 DEBUG [...]
2024-04-15 20:56:07,886 DEBUG
2024-04-15 20:56:07,886 DEBUG
2024-04-15 20:56:07,886 DEBUG
2024-04-15 20:56:07,886 DEBUG "aliasList": [],
2024-04-15 20:56:07,886 DEBUG "kbaInfo": [],
2024-04-15 20:56:07,886 DEBUG "inactiveDate": null,
2024-04-15 20:56:07,887 DEBUG "sn": "Parker",
2024-04-15 20:56:07,887 DEBUG "preferences": null,
2024-04-15 20:56:07,887 DEBUG "userName": "idm_unmbc8xoik",
2024-04-15 20:56:07,887 DEBUG "passwordLastChangedTime": "2024-04-15T20:56:07.465Z",
2024-04-15 20:56:07,887 DEBUG "effectiveRoles": [],
2024-04-15 20:56:07,887 DEBUG "activateAccount": null
2024-04-15 20:56:07,887 DEBUG }
2024-04-15 20:56:07,887 INFO Authenticate as user idm_unmbc8xoik
2024-04-15 20:56:07,887 INFO [http_cmd]: curl --insecure -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/authentication?_action=login"
2024-04-15 20:56:08,092 INFO [http_cmd]: http status code OK
2024-04-15 20:56:08,092 DEBUG --- status code ---
2024-04-15 20:56:08,092 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:08,092 DEBUG --- http response ---
2024-04-15 20:56:08,092 DEBUG {
"_id": "login",
"authorization": {
"id": "anonymous",
"roles": [
"internal/role/openidm-reg"
],
"component": "internal/user"
},
"authenticationId": "anonymous"
}
2024-04-15 20:56:08,093 INFO [http_cmd]: curl --insecure -L --request DELETE "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user/a7436859-9248-43b8-9bc5-e487f5ec08ac"
2024-04-15 20:56:08,335 INFO [http_cmd]: http status code OK
2024-04-15 20:56:08,335 DEBUG --- status code ---
2024-04-15 20:56:08,335 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:08,335 DEBUG --- http response ---
2024-04-15 20:56:08,335 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-15 20:56:08,335 DEBUG ----- output -----
2024-04-15 20:56:08,335 DEBUG {
2024-04-15 20:56:08,335 DEBUG "_id": "a7436859-9248-43b8-9bc5-e487f5ec08ac",
2024-04-15 20:56:08,335 DEBUG "_rev": "d2dd67f6-c775-4646-8c43-6112461729ba-1066747",
2024-04-15 20:56:08,335 DEBUG "country": null,
2024-04-15 20:56:08,335 DEBUG "mail": "idm_unmbc8xoik@forgerock.com",
2024-04-15 20:56:08,335 DEBUG "memberOfOrgIDs": [],
2024-04-15 20:56:08,335 DEBUG "postalCode": null,
2024-04-15 20:56:08,335 DEBUG "profileImage": null,
2024-04-15 20:56:08,335 DEBUG "expireAccount": null,
2024-04-15 20:56:08,335 DEBUG "activeDate": null,
2024-04-15 20:56:08,335 DEBUG
2024-04-15 20:56:08,336 DEBUG
2024-04-15 20:56:08,336 DEBUG
2024-04-15 20:56:08,336 DEBUG [...]
2024-04-15 20:56:08,336 DEBUG
2024-04-15 20:56:08,336 DEBUG
2024-04-15 20:56:08,336 DEBUG
2024-04-15 20:56:08,336 DEBUG "aliasList": [],
2024-04-15 20:56:08,336 DEBUG "kbaInfo": [],
2024-04-15 20:56:08,336 DEBUG "inactiveDate": null,
2024-04-15 20:56:08,336 DEBUG "sn": "Parker",
2024-04-15 20:56:08,336 DEBUG "preferences": null,
2024-04-15 20:56:08,336 DEBUG "userName": "idm_unmbc8xoik",
2024-04-15 20:56:08,336 DEBUG "passwordLastChangedTime": "2024-04-15T20:56:07.465Z",
2024-04-15 20:56:08,336 DEBUG "effectiveRoles": [],
2024-04-15 20:56:08,336 DEBUG "activateAccount": null
2024-04-15 20:56:08,336 DEBUG }
2024-04-15 20:56:08,338 INFO
2024-04-15 20:56:08,338 INFO -------------------- Running END-USER-UI livecheck --------------------
2024-04-15 20:56:08,338 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/enduser
2024-04-15 20:56:08,339 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/enduser"
2024-04-15 20:56:08,616 INFO [http_cmd]: http status code OK
2024-04-15 20:56:08,616 DEBUG --- status code ---
2024-04-15 20:56:08,616 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:08,616 DEBUG --- http response ---
2024-04-15 20:56:08,616 DEBUG Identity Management
[]
2024-04-15 20:56:08,618 INFO
2024-04-15 20:56:08,619 INFO --------------------- Running LOGIN-UI livecheck ---------------------
2024-04-15 20:56:08,619 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/XUI
2024-04-15 20:56:08,619 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/XUI"
2024-04-15 20:56:08,897 INFO [http_cmd]: http status code OK
2024-04-15 20:56:08,898 DEBUG --- status code ---
2024-04-15 20:56:08,898 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:08,898 DEBUG --- http response ---
2024-04-15 20:56:08,898 DEBUG Login
[]
2024-04-15 20:56:08,900 INFO
2024-04-15 20:56:08,900 INFO --------------------- Running ADMIN-UI livecheck ---------------------
2024-04-15 20:56:08,900 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/platform
2024-04-15 20:56:08,900 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/platform"
2024-04-15 20:56:09,177 INFO [http_cmd]: http status code OK
2024-04-15 20:56:09,177 DEBUG --- status code ---
2024-04-15 20:56:09,177 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:09,177 DEBUG --- http response ---
2024-04-15 20:56:09,178 DEBUG Platform Admin
[]
2024-04-15 20:56:09,179 INFO LIVECHECK SUCCEEDED
2024-04-15 20:56:09,180 INFO
2024-04-15 20:56:09,180 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-15 20:56:09,180 INFO
2024-04-15 20:56:09,180 INFO --------------------- Get DS-CTS software version ---------------------
2024-04-15 20:56:09,180 INFO
2024-04-15 20:56:09,180 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- find /opt/opendj -name opendj-core.jar
2024-04-15 20:56:09,180 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 20:56:09,991 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:09,991 DEBUG --- stdout ---
2024-04-15 20:56:09,991 DEBUG /opt/opendj/lib/opendj-core.jar
2024-04-15 20:56:09,991 DEBUG --- stderr ---
2024-04-15 20:56:09,991 DEBUG
2024-04-15 20:56:09,992 INFO
2024-04-15 20:56:09,992 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-cts-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-cts_info/opendj-core.jar --container ds
2024-04-15 20:56:09,992 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:11,222 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:11,223 DEBUG --- stdout ---
2024-04-15 20:56:11,223 DEBUG tar: Removing leading `/' from member names
2024-04-15 20:56:11,223 DEBUG --- stderr ---
2024-04-15 20:56:11,223 DEBUG
2024-04-15 20:56:11,232 INFO
2024-04-15 20:56:11,232 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-15 20:56:11,232 INFO
2024-04-15 20:56:11,232 INFO ------------------- Get DS-IDREPO software version -------------------
2024-04-15 20:56:11,232 INFO
2024-04-15 20:56:11,232 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- find /opt/opendj -name opendj-core.jar
2024-04-15 20:56:11,232 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 20:56:12,043 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:12,043 DEBUG --- stdout ---
2024-04-15 20:56:12,044 DEBUG /opt/opendj/lib/opendj-core.jar
2024-04-15 20:56:12,044 DEBUG --- stderr ---
2024-04-15 20:56:12,044 DEBUG
2024-04-15 20:56:12,044 INFO
2024-04-15 20:56:12,044 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-idrepo-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-idrepo_info/opendj-core.jar --container ds
2024-04-15 20:56:12,044 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:13,303 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:13,303 DEBUG --- stdout ---
2024-04-15 20:56:13,303 DEBUG tar: Removing leading `/' from member names
2024-04-15 20:56:13,304 DEBUG --- stderr ---
2024-04-15 20:56:13,304 DEBUG
2024-04-15 20:56:13,312 INFO
2024-04-15 20:56:13,312 INFO ****************************** Initializing component pods for AM ******************************
2024-04-15 20:56:13,312 INFO
2024-04-15 20:56:13,312 INFO ----------------------- Get AM software version -----------------------
2024-04-15 20:56:13,312 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version
2024-04-15 20:56:13,312 INFO [LoginSession] Get "token_id" token for user "amadmin"
2024-04-15 20:56:13,312 DEBUG [LoginSession] re-using "token_id" token that remains valid for 2m 42s
2024-04-15 20:56:13,313 INFO [http_cmd]: curl -L --request GET --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=WiaYtJRX04gYYFqrE6Ghr7H8p6g.*AAJTSQACMDIAAlNLABxOTkNYRThUQkh1WUxaODNiOWN5QmxkSXFOSUU9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" --cookie "route=1713214567.544.49085.350152|f60edb382037eb2df1e800d563ad78a7" "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version"
2024-04-15 20:56:13,532 INFO [http_cmd]: http status code OK
2024-04-15 20:56:13,532 DEBUG --- status code ---
2024-04-15 20:56:13,532 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:13,532 DEBUG --- http response ---
2024-04-15 20:56:13,532 DEBUG {
"_id": "version",
"_rev": "509589602",
"version": "7.6.0-SNAPSHOT",
"fullVersion": "ForgeRock Access Management 7.6.0-SNAPSHOT Build bbf6791c961b39ef101ebad36686ee2c3cde94bb (2024-April-12 18:28)",
"revision": "bbf6791c961b39ef101ebad36686ee2c3cde94bb",
"date": "2024-April-12 18:28"
}
2024-04-15 20:56:13,536 INFO
2024-04-15 20:56:13,536 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-15 20:56:13,536 INFO
2024-04-15 20:56:13,536 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-15 20:56:13,536 INFO
2024-04-15 20:56:13,536 INFO ---------------------- Get IDM software version ----------------------
2024-04-15 20:56:13,536 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/version
2024-04-15 20:56:13,537 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/version"
2024-04-15 20:56:13,752 INFO [http_cmd]: http status code OK
2024-04-15 20:56:13,752 DEBUG --- status code ---
2024-04-15 20:56:13,752 DEBUG http status code is 200 (expected 200)
2024-04-15 20:56:13,752 DEBUG --- http response ---
2024-04-15 20:56:13,752 DEBUG {
"_id": "version",
"productVersion": "7.6.0-SNAPSHOT",
"productBuildDate": "20240411202127",
"productRevision": "8554e0d1c5"
}
2024-04-15 20:56:13,757 INFO
2024-04-15 20:56:13,757 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-15 20:56:13,757 INFO
2024-04-15 20:56:13,757 INFO ------------------ Get END-USER-UI software version ------------------
2024-04-15 20:56:13,757 INFO
2024-04-15 20:56:13,757 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec end-user-ui-58878d7668-zg4h2 --container end-user-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-15 20:56:13,757 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 20:56:14,539 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:14,539 DEBUG --- stdout ---
2024-04-15 20:56:14,540 DEBUG /usr/share/nginx/html/js/chunk-vendors.0380a414.js
2024-04-15 20:56:14,540 DEBUG --- stderr ---
2024-04-15 20:56:14,540 DEBUG
2024-04-15 20:56:14,540 INFO
2024-04-15 20:56:14,540 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp end-user-ui-58878d7668-zg4h2:/usr/share/nginx/html/js/chunk-vendors.0380a414.js /tmp/end-user-ui_info/chunk-vendors.0380a414.js --container end-user-ui
2024-04-15 20:56:14,540 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:15,694 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:15,695 DEBUG --- stdout ---
2024-04-15 20:56:15,695 DEBUG tar: Removing leading `/' from member names
2024-04-15 20:56:15,695 DEBUG --- stderr ---
2024-04-15 20:56:15,695 DEBUG
2024-04-15 20:56:15,734 INFO
2024-04-15 20:56:15,734 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-15 20:56:15,734 INFO
2024-04-15 20:56:15,734 INFO -------------------- Get LOGIN-UI software version --------------------
2024-04-15 20:56:15,734 INFO
2024-04-15 20:56:15,734 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec login-ui-5fdf49fc66-2wnx7 --container login-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-15 20:56:15,734 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 20:56:16,511 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:16,512 DEBUG --- stdout ---
2024-04-15 20:56:16,512 DEBUG /usr/share/nginx/html/js/chunk-vendors.f2fa5365.js
2024-04-15 20:56:16,512 DEBUG --- stderr ---
2024-04-15 20:56:16,512 DEBUG
2024-04-15 20:56:16,513 INFO
2024-04-15 20:56:16,513 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp login-ui-5fdf49fc66-2wnx7:/usr/share/nginx/html/js/chunk-vendors.f2fa5365.js /tmp/login-ui_info/chunk-vendors.f2fa5365.js --container login-ui
2024-04-15 20:56:16,513 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:17,653 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:17,654 DEBUG --- stdout ---
2024-04-15 20:56:17,654 DEBUG tar: Removing leading `/' from member names
2024-04-15 20:56:17,654 DEBUG --- stderr ---
2024-04-15 20:56:17,654 DEBUG
2024-04-15 20:56:17,677 INFO
2024-04-15 20:56:17,678 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-15 20:56:17,678 INFO
2024-04-15 20:56:17,678 INFO -------------------- Get ADMIN-UI software version --------------------
2024-04-15 20:56:17,678 INFO
2024-04-15 20:56:17,678 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec admin-ui-c4c7bf584-mbzlq --container admin-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-15 20:56:17,678 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 20:56:18,449 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:18,450 DEBUG --- stdout ---
2024-04-15 20:56:18,451 DEBUG /usr/share/nginx/html/js/chunk-vendors.b3d879ac.js
2024-04-15 20:56:18,451 DEBUG --- stderr ---
2024-04-15 20:56:18,451 DEBUG
2024-04-15 20:56:18,451 INFO
2024-04-15 20:56:18,451 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp admin-ui-c4c7bf584-mbzlq:/usr/share/nginx/html/js/chunk-vendors.b3d879ac.js /tmp/admin-ui_info/chunk-vendors.b3d879ac.js --container admin-ui
2024-04-15 20:56:18,452 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:19,664 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:19,665 DEBUG --- stdout ---
2024-04-15 20:56:19,665 DEBUG tar: Removing leading `/' from member names
2024-04-15 20:56:19,665 DEBUG --- stderr ---
2024-04-15 20:56:19,665 DEBUG
2024-04-15 20:56:19,716 INFO
2024-04-15 20:56:19,716 INFO
2024-04-15 20:56:19,716 INFO ====================================================================================================
2024-04-15 20:56:19,716 INFO ================ Admin password for DS-CTS is: b2Et9XuIRcBwntYUi7J7f7OASS9YbQqx ================
2024-04-15 20:56:19,716 INFO ====================================================================================================
2024-04-15 20:56:19,716 INFO
2024-04-15 20:56:19,716 INFO ====================================================================================================
2024-04-15 20:56:19,716 INFO ============== Admin password for DS-IDREPO is: b2Et9XuIRcBwntYUi7J7f7OASS9YbQqx ==============
2024-04-15 20:56:19,716 INFO ====================================================================================================
2024-04-15 20:56:19,716 INFO
2024-04-15 20:56:19,716 INFO ====================================================================================================
2024-04-15 20:56:19,716 INFO ====================== Admin password for AM is: YNwb14OS5H3c83lWKq9jblnT ======================
2024-04-15 20:56:19,716 INFO ====================================================================================================
2024-04-15 20:56:19,716 INFO
2024-04-15 20:56:19,716 INFO ====================================================================================================
2024-04-15 20:56:19,716 INFO ===================== Admin password for IDM is: 1UXqmgfePB4NHghbGcuasRXC =====================
2024-04-15 20:56:19,716 INFO ====================================================================================================
2024-04-15 20:56:19,717 INFO
2024-04-15 20:56:19,717 INFO *************************************** Dumping pod list ***************************************
2024-04-15 20:56:19,717 INFO Dumping pod list to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/_pod-list.txt
2024-04-15 20:56:20,355 INFO Dumping platform-config configmap to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/_platform-config_configmap.txt
2024-04-15 20:56:20,726 INFO
2024-04-15 20:56:20,726 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-15 20:56:20,727 INFO
2024-04-15 20:56:20,727 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:56:20,727 INFO
2024-04-15 20:56:20,727 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:56:20,727 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:21,152 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:21,152 DEBUG --- stdout ---
2024-04-15 20:56:21,152 DEBUG 3
2024-04-15 20:56:21,152 DEBUG --- stderr ---
2024-04-15 20:56:21,152 DEBUG
2024-04-15 20:56:21,152 INFO
2024-04-15 20:56:21,152 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:56:21,153 INFO
2024-04-15 20:56:21,153 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-cts --output jsonpath={.items[*].metadata.name}
2024-04-15 20:56:21,153 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:56:21,634 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:21,635 DEBUG --- stdout ---
2024-04-15 20:56:21,635 DEBUG ds-cts-0 ds-cts-1 ds-cts-2
2024-04-15 20:56:21,635 DEBUG --- stderr ---
2024-04-15 20:56:21,635 DEBUG
2024-04-15 20:56:21,635 INFO
2024-04-15 20:56:21,635 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-15 20:56:21,635 INFO
2024-04-15 20:56:21,635 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:56:21,635 INFO
2024-04-15 20:56:21,635 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:56:21,635 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:22,037 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:22,037 DEBUG --- stdout ---
2024-04-15 20:56:22,037 DEBUG 3
2024-04-15 20:56:22,037 DEBUG --- stderr ---
2024-04-15 20:56:22,037 DEBUG
2024-04-15 20:56:22,037 INFO
2024-04-15 20:56:22,037 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:56:22,037 INFO
2024-04-15 20:56:22,038 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-idrepo --output jsonpath={.items[*].metadata.name}
2024-04-15 20:56:22,038 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:56:22,502 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:22,502 DEBUG --- stdout ---
2024-04-15 20:56:22,502 DEBUG ds-idrepo-0 ds-idrepo-1 ds-idrepo-2
2024-04-15 20:56:22,502 DEBUG --- stderr ---
2024-04-15 20:56:22,502 DEBUG
2024-04-15 20:56:22,502 INFO
2024-04-15 20:56:22,502 INFO ****************************** Initializing component pods for AM ******************************
2024-04-15 20:56:22,502 INFO
2024-04-15 20:56:22,502 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:56:22,502 INFO
2024-04-15 20:56:22,502 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:56:22,503 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:22,885 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:22,885 DEBUG --- stdout ---
2024-04-15 20:56:22,885 DEBUG 3
2024-04-15 20:56:22,886 DEBUG --- stderr ---
2024-04-15 20:56:22,886 DEBUG
2024-04-15 20:56:22,886 INFO
2024-04-15 20:56:22,886 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:56:22,886 INFO
2024-04-15 20:56:22,886 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}
2024-04-15 20:56:22,886 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:56:23,331 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:23,331 DEBUG --- stdout ---
2024-04-15 20:56:23,331 DEBUG am-7d597cf7d6-8ctft am-7d597cf7d6-8vr8f am-7d597cf7d6-9wz9h
2024-04-15 20:56:23,331 DEBUG --- stderr ---
2024-04-15 20:56:23,331 DEBUG
2024-04-15 20:56:23,331 INFO
2024-04-15 20:56:23,331 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-15 20:56:23,331 INFO
2024-04-15 20:56:23,331 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:56:23,332 INFO
2024-04-15 20:56:23,332 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-15 20:56:23,332 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:56:23,724 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:23,724 DEBUG --- stdout ---
2024-04-15 20:56:23,724 DEBUG amster-qv96p
2024-04-15 20:56:23,724 DEBUG --- stderr ---
2024-04-15 20:56:23,724 DEBUG
2024-04-15 20:56:23,724 INFO
2024-04-15 20:56:23,724 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-15 20:56:23,724 INFO
2024-04-15 20:56:23,724 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:56:23,725 INFO
2024-04-15 20:56:23,725 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:56:23,725 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:24,119 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:24,119 DEBUG --- stdout ---
2024-04-15 20:56:24,119 DEBUG 2
2024-04-15 20:56:24,119 DEBUG --- stderr ---
2024-04-15 20:56:24,119 DEBUG
2024-04-15 20:56:24,119 INFO
2024-04-15 20:56:24,119 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:56:24,120 INFO
2024-04-15 20:56:24,120 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=idm --output jsonpath={.items[*].metadata.name}
2024-04-15 20:56:24,120 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:56:24,550 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:24,550 DEBUG --- stdout ---
2024-04-15 20:56:24,550 DEBUG idm-b85f9d978-j5pbm idm-b85f9d978-rl76f
2024-04-15 20:56:24,550 DEBUG --- stderr ---
2024-04-15 20:56:24,550 DEBUG
2024-04-15 20:56:24,550 INFO
2024-04-15 20:56:24,550 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-15 20:56:24,550 INFO
2024-04-15 20:56:24,550 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:56:24,551 INFO
2024-04-15 20:56:24,551 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:56:24,551 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:24,872 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:24,872 DEBUG --- stdout ---
2024-04-15 20:56:24,873 DEBUG 1
2024-04-15 20:56:24,873 DEBUG --- stderr ---
2024-04-15 20:56:24,873 DEBUG
2024-04-15 20:56:24,873 INFO
2024-04-15 20:56:24,873 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:56:24,873 INFO
2024-04-15 20:56:24,873 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].metadata.name}
2024-04-15 20:56:24,873 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:56:25,215 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:25,215 DEBUG --- stdout ---
2024-04-15 20:56:25,216 DEBUG end-user-ui-58878d7668-zg4h2
2024-04-15 20:56:25,216 DEBUG --- stderr ---
2024-04-15 20:56:25,216 DEBUG
2024-04-15 20:56:25,216 INFO
2024-04-15 20:56:25,216 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-15 20:56:25,216 INFO
2024-04-15 20:56:25,216 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:56:25,216 INFO
2024-04-15 20:56:25,216 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:56:25,216 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:25,557 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:25,557 DEBUG --- stdout ---
2024-04-15 20:56:25,557 DEBUG 1
2024-04-15 20:56:25,557 DEBUG --- stderr ---
2024-04-15 20:56:25,557 DEBUG
2024-04-15 20:56:25,557 INFO
2024-04-15 20:56:25,557 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:56:25,558 INFO
2024-04-15 20:56:25,558 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].metadata.name}
2024-04-15 20:56:25,558 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:56:25,902 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:25,903 DEBUG --- stdout ---
2024-04-15 20:56:25,903 DEBUG login-ui-5fdf49fc66-2wnx7
2024-04-15 20:56:25,903 DEBUG --- stderr ---
2024-04-15 20:56:25,903 DEBUG
2024-04-15 20:56:25,903 INFO
2024-04-15 20:56:25,903 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-15 20:56:25,903 INFO
2024-04-15 20:56:25,903 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 20:56:25,903 INFO
2024-04-15 20:56:25,903 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 20:56:25,903 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 20:56:26,242 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:26,242 DEBUG --- stdout ---
2024-04-15 20:56:26,242 DEBUG 1
2024-04-15 20:56:26,242 DEBUG --- stderr ---
2024-04-15 20:56:26,242 DEBUG
2024-04-15 20:56:26,243 INFO
2024-04-15 20:56:26,243 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 20:56:26,243 INFO
2024-04-15 20:56:26,243 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].metadata.name}
2024-04-15 20:56:26,243 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 20:56:26,586 INFO [loop_until]: OK (rc = 0)
2024-04-15 20:56:26,587 DEBUG --- stdout ---
2024-04-15 20:56:26,587 DEBUG admin-ui-c4c7bf584-mbzlq
2024-04-15 20:56:26,587 DEBUG --- stderr ---
2024-04-15 20:56:26,587 DEBUG
2024-04-15 20:56:26,587 INFO
2024-04-15 20:56:26,587 INFO *********************************** Dumping components logs ***********************************
2024-04-15 20:56:26,587 INFO
2024-04-15 20:56:26,587 INFO ----------------------- Dumping logs for DS-CTS -----------------------
2024-04-15 20:56:27,753 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-cts-0.txt
2024-04-15 20:56:27,753 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-cts-0.txt
2024-04-15 20:56:27,754 INFO Check pod logs for errors
2024-04-15 20:56:29,206 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-cts-1.txt
2024-04-15 20:56:29,207 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-cts-1.txt
2024-04-15 20:56:29,209 INFO Check pod logs for errors
2024-04-15 20:56:30,576 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-cts-2.txt
2024-04-15 20:56:30,577 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-cts-2.txt
2024-04-15 20:56:30,578 INFO Check pod logs for errors
2024-04-15 20:56:30,579 INFO
2024-04-15 20:56:30,579 INFO --------------------- Dumping logs for DS-IDREPO ---------------------
2024-04-15 20:56:32,155 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-idrepo-0.txt
2024-04-15 20:56:32,156 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-idrepo-0.txt
2024-04-15 20:56:32,178 INFO Check pod logs for errors
2024-04-15 20:56:33,321 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-idrepo-1.txt
2024-04-15 20:56:33,322 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-idrepo-1.txt
2024-04-15 20:56:33,322 INFO Check pod logs for errors
2024-04-15 20:56:34,467 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-idrepo-2.txt
2024-04-15 20:56:34,469 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/ds-idrepo-2.txt
2024-04-15 20:56:34,470 INFO Check pod logs for errors
2024-04-15 20:56:34,470 INFO
2024-04-15 20:56:34,470 INFO ------------------------- Dumping logs for AM -------------------------
2024-04-15 20:56:35,873 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/am-7d597cf7d6-8ctft.txt
2024-04-15 20:56:35,874 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/am-7d597cf7d6-8ctft.txt
2024-04-15 20:56:35,875 INFO Check pod logs for errors
2024-04-15 20:56:37,566 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/am-7d597cf7d6-8vr8f.txt
2024-04-15 20:56:37,566 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/am-7d597cf7d6-8vr8f.txt
2024-04-15 20:56:37,586 INFO Check pod logs for errors
2024-04-15 20:56:39,346 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/am-7d597cf7d6-9wz9h.txt
2024-04-15 20:56:39,348 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/am-7d597cf7d6-9wz9h.txt
2024-04-15 20:56:39,408 INFO Check pod logs for errors
2024-04-15 20:56:39,437 INFO
2024-04-15 20:56:39,438 INFO ----------------------- Dumping logs for AMSTER -----------------------
2024-04-15 20:56:40,600 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/amster-qv96p.txt
2024-04-15 20:56:40,605 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/amster-qv96p.txt
2024-04-15 20:56:40,605 INFO Check pod logs for errors
2024-04-15 20:56:40,607 INFO
2024-04-15 20:56:40,608 INFO ------------------------ Dumping logs for IDM ------------------------
2024-04-15 20:56:42,455 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/idm-b85f9d978-j5pbm.txt
2024-04-15 20:56:42,456 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/idm-b85f9d978-j5pbm.txt
2024-04-15 20:56:42,494 INFO Check pod logs for errors
2024-04-15 20:56:44,361 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/idm-b85f9d978-rl76f.txt
2024-04-15 20:56:44,362 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/idm-b85f9d978-rl76f.txt
2024-04-15 20:56:44,390 INFO Check pod logs for errors
2024-04-15 20:56:44,408 INFO
2024-04-15 20:56:44,408 INFO -------------------- Dumping logs for END-USER-UI --------------------
2024-04-15 20:56:45,871 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/end-user-ui-58878d7668-zg4h2.txt
2024-04-15 20:56:45,872 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/end-user-ui-58878d7668-zg4h2.txt
2024-04-15 20:56:45,885 INFO Check pod logs for errors
2024-04-15 20:56:45,900 INFO
2024-04-15 20:56:45,901 INFO ---------------------- Dumping logs for LOGIN-UI ----------------------
2024-04-15 20:56:47,293 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/login-ui-5fdf49fc66-2wnx7.txt
2024-04-15 20:56:47,294 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/login-ui-5fdf49fc66-2wnx7.txt
2024-04-15 20:56:47,312 INFO Check pod logs for errors
2024-04-15 20:56:47,326 INFO
2024-04-15 20:56:47,326 INFO ---------------------- Dumping logs for ADMIN-UI ----------------------
2024-04-15 20:56:48,681 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/admin-ui-c4c7bf584-mbzlq.txt
2024-04-15 20:56:48,682 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_205619-after-deployment/admin-ui-c4c7bf584-mbzlq.txt
2024-04-15 20:56:48,695 INFO Check pod logs for errors
[2024-04-15 20:56:48] - INFO: Deployment ready
________________________________________________________________________________
[2024-04-15 20:56:48] Discover_all_components post : Post method
________________________________________________________________________________
Setting result to PASS
Task has been successfully stopped