--Task--
name: Discover_all_components
enabled: True
class_name: DiscoverComponentsTask
source_name: controller
source_namespace: >default<
target_name: controller
target_namespace: >default<
start: 0.0
stop: None
timeout: no timeout
loop: False
interval: None
dependencies: []
wait_for: []
preceding_task: None
options: {}
group_name: None
Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock
________________________________________________________________________________
[2024-04-19 21:03:36] Discover_all_components pre : Initialising task parameters
________________________________________________________________________________
task will be executed on controller (localhost)
________________________________________________________________________________
[2024-04-19 21:03:36] Discover_all_components step1 : Discover components
________________________________________________________________________________
2024-04-19 21:03:36,998 INFO Deployment was skipped.
2024-04-19 21:03:39,198 INFO
2024-04-19 21:03:39,199 INFO ************************************* Creating deployment *************************************
2024-04-19 21:03:39,199 INFO Creating normal (forgeops) type deployment for deployment: stack
2024-04-19 21:03:39,228 INFO
2024-04-19 21:03:39,228 INFO ------- Custom component configuration present. Loading values -------
2024-04-19 21:03:39,228 INFO
2024-04-19 21:03:39,229 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker ds cts
2024-04-19 21:03:39,355 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:03:39,356 DEBUG --- stdout ---
2024-04-19 21:03:39,356 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/cts
2024-04-19 21:03:39,356 DEBUG --- stderr ---
2024-04-19 21:03:39,356 DEBUG
2024-04-19 21:03:39,356 INFO
2024-04-19 21:03:39,357 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/cts/Dockerfile: gcr.io/forgerock-io/ds/pit1:7.6.0-db97906b8eff882e42d79d5b86aca43562470e8d
2024-04-19 21:03:39,357 INFO
2024-04-19 21:03:39,357 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker ds idrepo
2024-04-19 21:03:39,475 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:03:39,475 DEBUG --- stdout ---
2024-04-19 21:03:39,475 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/idrepo
2024-04-19 21:03:39,475 DEBUG --- stderr ---
2024-04-19 21:03:39,475 DEBUG
2024-04-19 21:03:39,475 INFO
2024-04-19 21:03:39,476 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/idrepo/Dockerfile: gcr.io/forgerock-io/ds/pit1:7.6.0-db97906b8eff882e42d79d5b86aca43562470e8d
2024-04-19 21:03:39,476 INFO
2024-04-19 21:03:39,476 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker am
2024-04-19 21:03:39,575 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:03:39,575 DEBUG --- stdout ---
2024-04-19 21:03:39,575 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/am
2024-04-19 21:03:39,575 DEBUG --- stderr ---
2024-04-19 21:03:39,576 DEBUG
2024-04-19 21:03:39,576 INFO
2024-04-19 21:03:39,576 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/am/Dockerfile: gcr.io/forgerock-io/am-cdk/pit1:7.6.0-b3f32bcf2825f595a381aae2f4ddab14a92da8c1
2024-04-19 21:03:39,576 INFO
2024-04-19 21:03:39,576 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker amster
2024-04-19 21:03:39,689 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:03:39,689 DEBUG --- stdout ---
2024-04-19 21:03:39,690 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/amster
2024-04-19 21:03:39,690 DEBUG --- stderr ---
2024-04-19 21:03:39,690 DEBUG
2024-04-19 21:03:39,690 INFO
2024-04-19 21:03:39,690 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/amster/Dockerfile: gcr.io/forgerock-io/amster/pit1:7.6.0-b3f32bcf2825f595a381aae2f4ddab14a92da8c1
2024-04-19 21:03:39,690 INFO
2024-04-19 21:03:39,690 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker idm
2024-04-19 21:03:39,772 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:03:39,773 DEBUG --- stdout ---
2024-04-19 21:03:39,773 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/idm
2024-04-19 21:03:39,773 DEBUG --- stderr ---
2024-04-19 21:03:39,773 DEBUG
2024-04-19 21:03:39,773 INFO
2024-04-19 21:03:39,773 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/idm/Dockerfile: gcr.io/forgerock-io/idm-cdk/pit1:7.6.0-8554e0d1c5044f4f6c6504cd366196848874e695
2024-04-19 21:03:39,773 INFO
2024-04-19 21:03:39,773 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base end-user-ui
2024-04-19 21:03:39,854 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:03:39,854 DEBUG --- stdout ---
2024-04-19 21:03:39,855 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/end-user-ui
2024-04-19 21:03:39,855 DEBUG --- stderr ---
2024-04-19 21:03:39,855 DEBUG
2024-04-19 21:03:39,855 INFO
2024-04-19 21:03:39,858 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/end-user-ui/kustomization.yaml: gcr.io/forgerock-io/platform-enduser-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-19 21:03:39,858 INFO
2024-04-19 21:03:39,858 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base login-ui
2024-04-19 21:03:39,938 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:03:39,938 DEBUG --- stdout ---
2024-04-19 21:03:39,939 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/login-ui
2024-04-19 21:03:39,939 DEBUG --- stderr ---
2024-04-19 21:03:39,939 DEBUG
2024-04-19 21:03:39,939 INFO
2024-04-19 21:03:39,942 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/login-ui/kustomization.yaml: gcr.io/forgerock-io/platform-login-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-19 21:03:39,942 INFO
2024-04-19 21:03:39,942 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base admin-ui
2024-04-19 21:03:40,025 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:03:40,025 DEBUG --- stdout ---
2024-04-19 21:03:40,025 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/admin-ui
2024-04-19 21:03:40,025 DEBUG --- stderr ---
2024-04-19 21:03:40,025 DEBUG
2024-04-19 21:03:40,025 INFO
2024-04-19 21:03:40,028 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/admin-ui/kustomization.yaml: gcr.io/forgerock-io/platform-admin-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-19 21:03:40,028 INFO Loading external components from config.yaml
2024-04-19 21:03:40,153 INFO No external components defined.
2024-04-19 21:03:40,278 INFO Setting context for components to: gke_engineeringpit_us-east1-d_xlou-cdm
2024-04-19 21:03:40,278 INFO
2024-04-19 21:03:40,278 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-19 21:03:40,278 INFO
2024-04-19 21:03:40,278 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:03:40,278 INFO
2024-04-19 21:03:40,278 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:03:40,278 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:40,676 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:40,676 DEBUG --- stdout ---
2024-04-19 21:03:40,677 DEBUG 3
2024-04-19 21:03:40,677 DEBUG --- stderr ---
2024-04-19 21:03:40,677 DEBUG
2024-04-19 21:03:40,677 INFO
2024-04-19 21:03:40,677 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:03:40,677 INFO
2024-04-19 21:03:40,677 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-cts --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-19 21:03:40,677 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:03:41,127 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-19 21:03:41,127 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:41,127 DEBUG --- stdout ---
2024-04-19 21:03:41,127 DEBUG ds-cts-0 ds-cts-1 ds-cts-2
2024-04-19 21:03:41,127 DEBUG --- stderr ---
2024-04-19 21:03:41,127 DEBUG
2024-04-19 21:03:41,127 INFO
2024-04-19 21:03:41,127 INFO -------------------- Check pod ds-cts-0 is running --------------------
2024-04-19 21:03:41,128 INFO
2024-04-19 21:03:41,128 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-0 -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:03:41,128 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:41,496 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:41,496 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:41,496 DEBUG --- stdout ---
2024-04-19 21:03:41,496 DEBUG Running
2024-04-19 21:03:41,497 DEBUG --- stderr ---
2024-04-19 21:03:41,497 DEBUG
2024-04-19 21:03:41,497 INFO
2024-04-19 21:03:41,497 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-0 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:03:41,497 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:41,887 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:41,887 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:41,887 DEBUG --- stdout ---
2024-04-19 21:03:41,887 DEBUG true
2024-04-19 21:03:41,887 DEBUG --- stderr ---
2024-04-19 21:03:41,887 DEBUG
2024-04-19 21:03:41,888 INFO
2024-04-19 21:03:41,888 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-0 --output jsonpath={.status.startTime}
2024-04-19 21:03:41,888 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:42,267 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:42,267 DEBUG --- stdout ---
2024-04-19 21:03:42,267 DEBUG 2024-04-17T18:33:21Z
2024-04-19 21:03:42,267 DEBUG --- stderr ---
2024-04-19 21:03:42,267 DEBUG
2024-04-19 21:03:42,268 INFO
2024-04-19 21:03:42,268 INFO ------------- Check pod ds-cts-0 filesystem is accessible -------------
2024-04-19 21:03:43,160 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:43,161 INFO
2024-04-19 21:03:43,161 INFO ------------------ Check pod ds-cts-0 restart count ------------------
2024-04-19 21:03:43,161 INFO
2024-04-19 21:03:43,161 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-0 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:03:43,161 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:43,687 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:43,687 DEBUG --- stdout ---
2024-04-19 21:03:43,687 DEBUG 0
2024-04-19 21:03:43,688 DEBUG --- stderr ---
2024-04-19 21:03:43,688 DEBUG
2024-04-19 21:03:43,688 INFO Pod ds-cts-0 has been restarted 0 times.
2024-04-19 21:03:43,688 INFO
2024-04-19 21:03:43,688 INFO -------------------- Check pod ds-cts-1 is running --------------------
2024-04-19 21:03:43,688 INFO
2024-04-19 21:03:43,688 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-1 -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:03:43,688 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:44,225 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:44,225 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:44,225 DEBUG --- stdout ---
2024-04-19 21:03:44,226 DEBUG Running
2024-04-19 21:03:44,226 DEBUG --- stderr ---
2024-04-19 21:03:44,226 DEBUG
2024-04-19 21:03:44,227 INFO
2024-04-19 21:03:44,227 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-1 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:03:44,227 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:44,771 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:44,773 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:44,773 DEBUG --- stdout ---
2024-04-19 21:03:44,773 DEBUG true
2024-04-19 21:03:44,773 DEBUG --- stderr ---
2024-04-19 21:03:44,773 DEBUG
2024-04-19 21:03:44,774 INFO
2024-04-19 21:03:44,774 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-1 --output jsonpath={.status.startTime}
2024-04-19 21:03:44,774 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:45,320 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:45,320 DEBUG --- stdout ---
2024-04-19 21:03:45,320 DEBUG 2024-04-17T18:33:00Z
2024-04-19 21:03:45,321 DEBUG --- stderr ---
2024-04-19 21:03:45,321 DEBUG
2024-04-19 21:03:45,322 INFO
2024-04-19 21:03:45,322 INFO ------------- Check pod ds-cts-1 filesystem is accessible -------------
2024-04-19 21:03:46,345 INFO [loop_until]: Function succeeded after 1s (rc=0) - expected pattern found
2024-04-19 21:03:46,346 INFO
2024-04-19 21:03:46,346 INFO ------------------ Check pod ds-cts-1 restart count ------------------
2024-04-19 21:03:46,346 INFO
2024-04-19 21:03:46,346 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-1 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:03:46,346 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:46,834 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:46,835 DEBUG --- stdout ---
2024-04-19 21:03:46,835 DEBUG 0
2024-04-19 21:03:46,835 DEBUG --- stderr ---
2024-04-19 21:03:46,838 DEBUG
2024-04-19 21:03:46,838 INFO Pod ds-cts-1 has been restarted 0 times.
2024-04-19 21:03:46,839 INFO
2024-04-19 21:03:46,839 INFO -------------------- Check pod ds-cts-2 is running --------------------
2024-04-19 21:03:46,840 INFO
2024-04-19 21:03:46,840 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:03:46,840 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:47,371 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:47,371 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:47,371 DEBUG --- stdout ---
2024-04-19 21:03:47,371 DEBUG Running
2024-04-19 21:03:47,371 DEBUG --- stderr ---
2024-04-19 21:03:47,371 DEBUG
2024-04-19 21:03:47,372 INFO
2024-04-19 21:03:47,372 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:03:47,372 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:47,877 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:47,877 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:47,877 DEBUG --- stdout ---
2024-04-19 21:03:47,877 DEBUG true
2024-04-19 21:03:47,877 DEBUG --- stderr ---
2024-04-19 21:03:47,877 DEBUG
2024-04-19 21:03:47,878 INFO
2024-04-19 21:03:47,878 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-2 --output jsonpath={.status.startTime}
2024-04-19 21:03:47,878 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:48,368 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:48,368 DEBUG --- stdout ---
2024-04-19 21:03:48,368 DEBUG 2024-04-17T18:32:38Z
2024-04-19 21:03:48,368 DEBUG --- stderr ---
2024-04-19 21:03:48,368 DEBUG
2024-04-19 21:03:48,368 INFO
2024-04-19 21:03:48,369 INFO ------------- Check pod ds-cts-2 filesystem is accessible -------------
2024-04-19 21:03:49,337 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:49,339 INFO
2024-04-19 21:03:49,344 INFO ------------------ Check pod ds-cts-2 restart count ------------------
2024-04-19 21:03:49,345 INFO
2024-04-19 21:03:49,345 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:03:49,345 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:49,824 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:49,824 DEBUG --- stdout ---
2024-04-19 21:03:49,824 DEBUG 0
2024-04-19 21:03:49,824 DEBUG --- stderr ---
2024-04-19 21:03:49,824 DEBUG
2024-04-19 21:03:49,824 INFO Pod ds-cts-2 has been restarted 0 times.
2024-04-19 21:03:49,824 INFO
2024-04-19 21:03:49,825 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-19 21:03:49,825 INFO
2024-04-19 21:03:49,825 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:03:49,825 INFO
2024-04-19 21:03:49,825 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:03:49,825 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:50,323 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:50,323 DEBUG --- stdout ---
2024-04-19 21:03:50,323 DEBUG 3
2024-04-19 21:03:50,323 DEBUG --- stderr ---
2024-04-19 21:03:50,323 DEBUG
2024-04-19 21:03:50,323 INFO
2024-04-19 21:03:50,323 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:03:50,323 INFO
2024-04-19 21:03:50,323 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-idrepo --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-19 21:03:50,323 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:03:50,884 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-19 21:03:50,884 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:50,884 DEBUG --- stdout ---
2024-04-19 21:03:50,884 DEBUG ds-idrepo-0 ds-idrepo-1 ds-idrepo-2
2024-04-19 21:03:50,884 DEBUG --- stderr ---
2024-04-19 21:03:50,884 DEBUG
2024-04-19 21:03:50,884 INFO
2024-04-19 21:03:50,884 INFO ------------------ Check pod ds-idrepo-0 is running ------------------
2024-04-19 21:03:50,884 INFO
2024-04-19 21:03:50,884 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-0 -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:03:50,884 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:51,404 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:51,405 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:51,405 DEBUG --- stdout ---
2024-04-19 21:03:51,405 DEBUG Running
2024-04-19 21:03:51,406 DEBUG --- stderr ---
2024-04-19 21:03:51,406 DEBUG
2024-04-19 21:03:51,406 INFO
2024-04-19 21:03:51,406 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-0 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:03:51,406 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:51,973 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:51,973 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:51,973 DEBUG --- stdout ---
2024-04-19 21:03:51,974 DEBUG true
2024-04-19 21:03:51,974 DEBUG --- stderr ---
2024-04-19 21:03:51,974 DEBUG
2024-04-19 21:03:51,974 INFO
2024-04-19 21:03:51,974 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-0 --output jsonpath={.status.startTime}
2024-04-19 21:03:51,974 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:52,508 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:52,508 DEBUG --- stdout ---
2024-04-19 21:03:52,508 DEBUG 2024-04-17T18:34:28Z
2024-04-19 21:03:52,508 DEBUG --- stderr ---
2024-04-19 21:03:52,508 DEBUG
2024-04-19 21:03:52,509 INFO
2024-04-19 21:03:52,509 INFO ----------- Check pod ds-idrepo-0 filesystem is accessible -----------
2024-04-19 21:03:53,475 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:53,475 INFO
2024-04-19 21:03:53,475 INFO ----------------- Check pod ds-idrepo-0 restart count -----------------
2024-04-19 21:03:53,475 INFO
2024-04-19 21:03:53,475 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-0 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:03:53,476 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:53,970 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:53,971 DEBUG --- stdout ---
2024-04-19 21:03:53,971 DEBUG 0
2024-04-19 21:03:53,971 DEBUG --- stderr ---
2024-04-19 21:03:53,971 DEBUG
2024-04-19 21:03:53,971 INFO Pod ds-idrepo-0 has been restarted 0 times.
2024-04-19 21:03:53,971 INFO
2024-04-19 21:03:53,971 INFO ------------------ Check pod ds-idrepo-1 is running ------------------
2024-04-19 21:03:53,972 INFO
2024-04-19 21:03:53,972 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-1 -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:03:53,972 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:54,443 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:54,445 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:54,445 DEBUG --- stdout ---
2024-04-19 21:03:54,445 DEBUG Running
2024-04-19 21:03:54,446 DEBUG --- stderr ---
2024-04-19 21:03:54,446 DEBUG
2024-04-19 21:03:54,447 INFO
2024-04-19 21:03:54,447 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-1 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:03:54,447 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:54,996 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:54,996 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:54,996 DEBUG --- stdout ---
2024-04-19 21:03:54,996 DEBUG true
2024-04-19 21:03:54,996 DEBUG --- stderr ---
2024-04-19 21:03:54,996 DEBUG
2024-04-19 21:03:54,997 INFO
2024-04-19 21:03:54,997 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-1 --output jsonpath={.status.startTime}
2024-04-19 21:03:54,997 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:55,544 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:55,545 DEBUG --- stdout ---
2024-04-19 21:03:55,545 DEBUG 2024-04-17T18:33:37Z
2024-04-19 21:03:55,545 DEBUG --- stderr ---
2024-04-19 21:03:55,545 DEBUG
2024-04-19 21:03:55,546 INFO
2024-04-19 21:03:55,546 INFO ----------- Check pod ds-idrepo-1 filesystem is accessible -----------
2024-04-19 21:03:56,494 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:56,494 INFO
2024-04-19 21:03:56,495 INFO ----------------- Check pod ds-idrepo-1 restart count -----------------
2024-04-19 21:03:56,495 INFO
2024-04-19 21:03:56,495 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-1 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:03:56,495 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:57,015 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:57,016 DEBUG --- stdout ---
2024-04-19 21:03:57,016 DEBUG 0
2024-04-19 21:03:57,016 DEBUG --- stderr ---
2024-04-19 21:03:57,016 DEBUG
2024-04-19 21:03:57,016 INFO Pod ds-idrepo-1 has been restarted 0 times.
2024-04-19 21:03:57,016 INFO
2024-04-19 21:03:57,016 INFO ------------------ Check pod ds-idrepo-2 is running ------------------
2024-04-19 21:03:57,017 INFO
2024-04-19 21:03:57,017 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:03:57,017 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:57,496 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:57,497 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:57,497 DEBUG --- stdout ---
2024-04-19 21:03:57,497 DEBUG Running
2024-04-19 21:03:57,498 DEBUG --- stderr ---
2024-04-19 21:03:57,498 DEBUG
2024-04-19 21:03:57,498 INFO
2024-04-19 21:03:57,498 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:03:57,498 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:03:57,988 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:57,988 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:57,988 DEBUG --- stdout ---
2024-04-19 21:03:57,989 DEBUG true
2024-04-19 21:03:57,989 DEBUG --- stderr ---
2024-04-19 21:03:57,989 DEBUG
2024-04-19 21:03:57,990 INFO
2024-04-19 21:03:57,991 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-2 --output jsonpath={.status.startTime}
2024-04-19 21:03:57,991 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:58,474 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:58,475 DEBUG --- stdout ---
2024-04-19 21:03:58,475 DEBUG 2024-04-17T18:32:49Z
2024-04-19 21:03:58,475 DEBUG --- stderr ---
2024-04-19 21:03:58,475 DEBUG
2024-04-19 21:03:58,477 INFO
2024-04-19 21:03:58,477 INFO ----------- Check pod ds-idrepo-2 filesystem is accessible -----------
2024-04-19 21:03:59,456 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:03:59,459 INFO
2024-04-19 21:03:59,460 INFO ----------------- Check pod ds-idrepo-2 restart count -----------------
2024-04-19 21:03:59,461 INFO
2024-04-19 21:03:59,461 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:03:59,461 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:03:59,946 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:03:59,947 DEBUG --- stdout ---
2024-04-19 21:03:59,947 DEBUG 0
2024-04-19 21:03:59,947 DEBUG --- stderr ---
2024-04-19 21:03:59,947 DEBUG
2024-04-19 21:03:59,947 INFO Pod ds-idrepo-2 has been restarted 0 times.
2024-04-19 21:03:59,947 INFO
2024-04-19 21:03:59,947 INFO ****************************** Initializing component pods for AM ******************************
2024-04-19 21:03:59,947 INFO
2024-04-19 21:03:59,947 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:03:59,948 INFO
2024-04-19 21:03:59,949 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:03:59,949 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:00,535 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:00,535 DEBUG --- stdout ---
2024-04-19 21:04:00,535 DEBUG 3
2024-04-19 21:04:00,535 DEBUG --- stderr ---
2024-04-19 21:04:00,535 DEBUG
2024-04-19 21:04:00,535 INFO
2024-04-19 21:04:00,535 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:04:00,536 INFO
2024-04-19 21:04:00,536 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-19 21:04:00,536 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:04:01,111 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-19 21:04:01,111 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:01,111 DEBUG --- stdout ---
2024-04-19 21:04:01,111 DEBUG am-767f6b67b5-9zw22 am-767f6b67b5-qcqz2 am-767f6b67b5-vbltz
2024-04-19 21:04:01,111 DEBUG --- stderr ---
2024-04-19 21:04:01,111 DEBUG
2024-04-19 21:04:01,112 INFO
2024-04-19 21:04:01,112 INFO -------------- Check pod am-767f6b67b5-9zw22 is running --------------
2024-04-19 21:04:01,112 INFO
2024-04-19 21:04:01,112 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-767f6b67b5-9zw22 -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:04:01,113 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:01,614 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:01,614 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:01,614 DEBUG --- stdout ---
2024-04-19 21:04:01,614 DEBUG Running
2024-04-19 21:04:01,615 DEBUG --- stderr ---
2024-04-19 21:04:01,615 DEBUG
2024-04-19 21:04:01,615 INFO
2024-04-19 21:04:01,615 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-767f6b67b5-9zw22 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:04:01,615 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:02,094 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:02,094 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:02,095 DEBUG --- stdout ---
2024-04-19 21:04:02,095 DEBUG true
2024-04-19 21:04:02,095 DEBUG --- stderr ---
2024-04-19 21:04:02,095 DEBUG
2024-04-19 21:04:02,096 INFO
2024-04-19 21:04:02,096 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-767f6b67b5-9zw22 --output jsonpath={.status.startTime}
2024-04-19 21:04:02,096 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:02,607 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:02,607 DEBUG --- stdout ---
2024-04-19 21:04:02,607 DEBUG 2024-04-19T18:05:54Z
2024-04-19 21:04:02,607 DEBUG --- stderr ---
2024-04-19 21:04:02,607 DEBUG
2024-04-19 21:04:02,608 INFO
2024-04-19 21:04:02,608 INFO ------- Check pod am-767f6b67b5-9zw22 filesystem is accessible -------
2024-04-19 21:04:03,573 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:03,573 INFO
2024-04-19 21:04:03,573 INFO ------------- Check pod am-767f6b67b5-9zw22 restart count -------------
2024-04-19 21:04:03,574 INFO
2024-04-19 21:04:03,574 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-767f6b67b5-9zw22 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:04:03,574 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:04,102 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:04,103 DEBUG --- stdout ---
2024-04-19 21:04:04,103 DEBUG 0
2024-04-19 21:04:04,103 DEBUG --- stderr ---
2024-04-19 21:04:04,103 DEBUG
2024-04-19 21:04:04,103 INFO Pod am-767f6b67b5-9zw22 has been restarted 0 times.
2024-04-19 21:04:04,103 INFO
2024-04-19 21:04:04,103 INFO -------------- Check pod am-767f6b67b5-qcqz2 is running --------------
2024-04-19 21:04:04,103 INFO
2024-04-19 21:04:04,103 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-767f6b67b5-qcqz2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:04:04,103 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:04,577 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:04,577 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:04,578 DEBUG --- stdout ---
2024-04-19 21:04:04,578 DEBUG Running
2024-04-19 21:04:04,578 DEBUG --- stderr ---
2024-04-19 21:04:04,578 DEBUG
2024-04-19 21:04:04,578 INFO
2024-04-19 21:04:04,578 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-767f6b67b5-qcqz2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:04:04,578 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:05,084 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:05,085 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:05,085 DEBUG --- stdout ---
2024-04-19 21:04:05,085 DEBUG true
2024-04-19 21:04:05,085 DEBUG --- stderr ---
2024-04-19 21:04:05,085 DEBUG
2024-04-19 21:04:05,085 INFO
2024-04-19 21:04:05,085 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-767f6b67b5-qcqz2 --output jsonpath={.status.startTime}
2024-04-19 21:04:05,085 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:05,580 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:05,580 DEBUG --- stdout ---
2024-04-19 21:04:05,580 DEBUG 2024-04-19T18:05:54Z
2024-04-19 21:04:05,580 DEBUG --- stderr ---
2024-04-19 21:04:05,580 DEBUG
2024-04-19 21:04:05,581 INFO
2024-04-19 21:04:05,581 INFO ------- Check pod am-767f6b67b5-qcqz2 filesystem is accessible -------
2024-04-19 21:04:06,546 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:06,547 INFO
2024-04-19 21:04:06,547 INFO ------------- Check pod am-767f6b67b5-qcqz2 restart count -------------
2024-04-19 21:04:06,547 INFO
2024-04-19 21:04:06,547 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-767f6b67b5-qcqz2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:04:06,547 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:07,028 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:07,028 DEBUG --- stdout ---
2024-04-19 21:04:07,028 DEBUG 0
2024-04-19 21:04:07,028 DEBUG --- stderr ---
2024-04-19 21:04:07,028 DEBUG
2024-04-19 21:04:07,028 INFO Pod am-767f6b67b5-qcqz2 has been restarted 0 times.
2024-04-19 21:04:07,028 INFO
2024-04-19 21:04:07,028 INFO -------------- Check pod am-767f6b67b5-vbltz is running --------------
2024-04-19 21:04:07,029 INFO
2024-04-19 21:04:07,029 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-767f6b67b5-vbltz -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:04:07,029 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:07,541 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:07,541 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:07,541 DEBUG --- stdout ---
2024-04-19 21:04:07,542 DEBUG Running
2024-04-19 21:04:07,542 DEBUG --- stderr ---
2024-04-19 21:04:07,542 DEBUG
2024-04-19 21:04:07,542 INFO
2024-04-19 21:04:07,542 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-767f6b67b5-vbltz -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:04:07,542 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:08,005 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:08,005 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:08,006 DEBUG --- stdout ---
2024-04-19 21:04:08,006 DEBUG true
2024-04-19 21:04:08,006 DEBUG --- stderr ---
2024-04-19 21:04:08,006 DEBUG
2024-04-19 21:04:08,007 INFO
2024-04-19 21:04:08,008 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-767f6b67b5-vbltz --output jsonpath={.status.startTime}
2024-04-19 21:04:08,008 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:08,511 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:08,512 DEBUG --- stdout ---
2024-04-19 21:04:08,512 DEBUG 2024-04-19T18:03:58Z
2024-04-19 21:04:08,512 DEBUG --- stderr ---
2024-04-19 21:04:08,512 DEBUG
2024-04-19 21:04:08,514 INFO
2024-04-19 21:04:08,514 INFO ------- Check pod am-767f6b67b5-vbltz filesystem is accessible -------
2024-04-19 21:04:09,455 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:09,455 INFO
2024-04-19 21:04:09,455 INFO ------------- Check pod am-767f6b67b5-vbltz restart count -------------
2024-04-19 21:04:09,456 INFO
2024-04-19 21:04:09,456 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-767f6b67b5-vbltz --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:04:09,457 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:10,045 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:10,047 DEBUG --- stdout ---
2024-04-19 21:04:10,047 DEBUG 0
2024-04-19 21:04:10,048 DEBUG --- stderr ---
2024-04-19 21:04:10,048 DEBUG
2024-04-19 21:04:10,048 INFO Pod am-767f6b67b5-vbltz has been restarted 0 times.
2024-04-19 21:04:10,049 INFO
2024-04-19 21:04:10,049 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-19 21:04:10,050 INFO
2024-04-19 21:04:10,050 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:04:10,050 INFO
2024-04-19 21:04:10,050 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm --field-selector status.phase!=Failed get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-19 21:04:10,050 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:04:10,577 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:10,577 DEBUG --- stdout ---
2024-04-19 21:04:10,577 DEBUG amster-grkh7
2024-04-19 21:04:10,577 DEBUG --- stderr ---
2024-04-19 21:04:10,577 DEBUG
2024-04-19 21:04:10,577 INFO
2024-04-19 21:04:10,577 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-19 21:04:10,577 INFO
2024-04-19 21:04:10,577 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:10,578 INFO
2024-04-19 21:04:10,578 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:10,578 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:11,081 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:11,081 DEBUG --- stdout ---
2024-04-19 21:04:11,081 DEBUG 2
2024-04-19 21:04:11,081 DEBUG --- stderr ---
2024-04-19 21:04:11,081 DEBUG
2024-04-19 21:04:11,081 INFO
2024-04-19 21:04:11,081 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:04:11,081 INFO
2024-04-19 21:04:11,081 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=idm --output jsonpath={.items[*].metadata.name}` | grep 2
2024-04-19 21:04:11,081 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:04:11,607 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-19 21:04:11,607 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:11,607 DEBUG --- stdout ---
2024-04-19 21:04:11,607 DEBUG idm-b85f9d978-txpwv idm-b85f9d978-wnhxb
2024-04-19 21:04:11,607 DEBUG --- stderr ---
2024-04-19 21:04:11,607 DEBUG
2024-04-19 21:04:11,607 INFO
2024-04-19 21:04:11,607 INFO -------------- Check pod idm-b85f9d978-txpwv is running --------------
2024-04-19 21:04:11,608 INFO
2024-04-19 21:04:11,608 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-txpwv -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:04:11,608 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:12,113 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:12,114 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:12,114 DEBUG --- stdout ---
2024-04-19 21:04:12,114 DEBUG Running
2024-04-19 21:04:12,114 DEBUG --- stderr ---
2024-04-19 21:04:12,114 DEBUG
2024-04-19 21:04:12,115 INFO
2024-04-19 21:04:12,115 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-txpwv -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:04:12,116 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:12,548 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:12,549 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:12,549 DEBUG --- stdout ---
2024-04-19 21:04:12,549 DEBUG true
2024-04-19 21:04:12,549 DEBUG --- stderr ---
2024-04-19 21:04:12,549 DEBUG
2024-04-19 21:04:12,549 INFO
2024-04-19 21:04:12,549 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-txpwv --output jsonpath={.status.startTime}
2024-04-19 21:04:12,549 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:13,039 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:13,040 DEBUG --- stdout ---
2024-04-19 21:04:13,040 DEBUG 2024-04-17T16:00:45Z
2024-04-19 21:04:13,040 DEBUG --- stderr ---
2024-04-19 21:04:13,040 DEBUG
2024-04-19 21:04:13,041 INFO
2024-04-19 21:04:13,041 INFO ------- Check pod idm-b85f9d978-txpwv filesystem is accessible -------
2024-04-19 21:04:14,117 INFO [loop_until]: Function succeeded after 1s (rc=0) - expected pattern found
2024-04-19 21:04:14,117 INFO
2024-04-19 21:04:14,117 INFO ------------- Check pod idm-b85f9d978-txpwv restart count -------------
2024-04-19 21:04:14,117 INFO
2024-04-19 21:04:14,118 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-txpwv --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:04:14,118 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:14,610 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:14,610 DEBUG --- stdout ---
2024-04-19 21:04:14,610 DEBUG 0
2024-04-19 21:04:14,610 DEBUG --- stderr ---
2024-04-19 21:04:14,610 DEBUG
2024-04-19 21:04:14,610 INFO Pod idm-b85f9d978-txpwv has been restarted 0 times.
2024-04-19 21:04:14,610 INFO
2024-04-19 21:04:14,611 INFO -------------- Check pod idm-b85f9d978-wnhxb is running --------------
2024-04-19 21:04:14,611 INFO
2024-04-19 21:04:14,611 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-wnhxb -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:04:14,611 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:15,150 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:15,151 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:15,153 DEBUG --- stdout ---
2024-04-19 21:04:15,153 DEBUG Running
2024-04-19 21:04:15,154 DEBUG --- stderr ---
2024-04-19 21:04:15,154 DEBUG
2024-04-19 21:04:15,154 INFO
2024-04-19 21:04:15,155 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-wnhxb -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:04:15,155 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:15,756 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:15,756 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:15,756 DEBUG --- stdout ---
2024-04-19 21:04:15,756 DEBUG true
2024-04-19 21:04:15,757 DEBUG --- stderr ---
2024-04-19 21:04:15,757 DEBUG
2024-04-19 21:04:15,758 INFO
2024-04-19 21:04:15,758 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-wnhxb --output jsonpath={.status.startTime}
2024-04-19 21:04:15,758 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:16,363 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:16,363 DEBUG --- stdout ---
2024-04-19 21:04:16,364 DEBUG 2024-04-17T16:00:45Z
2024-04-19 21:04:16,364 DEBUG --- stderr ---
2024-04-19 21:04:16,364 DEBUG
2024-04-19 21:04:16,366 INFO
2024-04-19 21:04:16,366 INFO ------- Check pod idm-b85f9d978-wnhxb filesystem is accessible -------
2024-04-19 21:04:17,431 INFO [loop_until]: Function succeeded after 1s (rc=0) - expected pattern found
2024-04-19 21:04:17,432 INFO
2024-04-19 21:04:17,432 INFO ------------- Check pod idm-b85f9d978-wnhxb restart count -------------
2024-04-19 21:04:17,432 INFO
2024-04-19 21:04:17,432 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-wnhxb --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:04:17,432 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:17,833 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:17,833 DEBUG --- stdout ---
2024-04-19 21:04:17,833 DEBUG 0
2024-04-19 21:04:17,833 DEBUG --- stderr ---
2024-04-19 21:04:17,833 DEBUG
2024-04-19 21:04:17,834 INFO Pod idm-b85f9d978-wnhxb has been restarted 0 times.
2024-04-19 21:04:17,834 INFO
2024-04-19 21:04:17,834 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-19 21:04:17,834 INFO
2024-04-19 21:04:17,834 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:17,834 INFO
2024-04-19 21:04:17,834 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:17,834 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:18,165 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:18,165 DEBUG --- stdout ---
2024-04-19 21:04:18,165 DEBUG 1
2024-04-19 21:04:18,166 DEBUG --- stderr ---
2024-04-19 21:04:18,166 DEBUG
2024-04-19 21:04:18,166 INFO
2024-04-19 21:04:18,166 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:04:18,166 INFO
2024-04-19 21:04:18,166 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-19 21:04:18,166 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:04:18,499 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-19 21:04:18,499 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:18,499 DEBUG --- stdout ---
2024-04-19 21:04:18,499 DEBUG end-user-ui-58878d7668-z8s6j
2024-04-19 21:04:18,499 DEBUG --- stderr ---
2024-04-19 21:04:18,499 DEBUG
2024-04-19 21:04:18,499 INFO
2024-04-19 21:04:18,499 INFO ---------- Check pod end-user-ui-58878d7668-z8s6j is running ----------
2024-04-19 21:04:18,499 INFO
2024-04-19 21:04:18,499 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods end-user-ui-58878d7668-z8s6j -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:04:18,499 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:18,819 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:18,819 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:18,819 DEBUG --- stdout ---
2024-04-19 21:04:18,819 DEBUG Running
2024-04-19 21:04:18,819 DEBUG --- stderr ---
2024-04-19 21:04:18,820 DEBUG
2024-04-19 21:04:18,820 INFO
2024-04-19 21:04:18,820 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods end-user-ui-58878d7668-z8s6j -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:04:18,820 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:19,126 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:19,126 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:19,127 DEBUG --- stdout ---
2024-04-19 21:04:19,127 DEBUG true
2024-04-19 21:04:19,127 DEBUG --- stderr ---
2024-04-19 21:04:19,127 DEBUG
2024-04-19 21:04:19,127 INFO
2024-04-19 21:04:19,127 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod end-user-ui-58878d7668-z8s6j --output jsonpath={.status.startTime}
2024-04-19 21:04:19,127 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:19,418 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:19,418 DEBUG --- stdout ---
2024-04-19 21:04:19,419 DEBUG 2024-04-17T16:01:55Z
2024-04-19 21:04:19,419 DEBUG --- stderr ---
2024-04-19 21:04:19,419 DEBUG
2024-04-19 21:04:19,419 INFO
2024-04-19 21:04:19,419 INFO --- Check pod end-user-ui-58878d7668-z8s6j filesystem is accessible ---
2024-04-19 21:04:20,179 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:20,179 INFO
2024-04-19 21:04:20,180 INFO -------- Check pod end-user-ui-58878d7668-z8s6j restart count --------
2024-04-19 21:04:20,180 INFO
2024-04-19 21:04:20,180 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod end-user-ui-58878d7668-z8s6j --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:04:20,180 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:20,479 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:20,480 DEBUG --- stdout ---
2024-04-19 21:04:20,480 DEBUG 0
2024-04-19 21:04:20,480 DEBUG --- stderr ---
2024-04-19 21:04:20,480 DEBUG
2024-04-19 21:04:20,480 INFO Pod end-user-ui-58878d7668-z8s6j has been restarted 0 times.
2024-04-19 21:04:20,480 INFO
2024-04-19 21:04:20,480 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-19 21:04:20,480 INFO
2024-04-19 21:04:20,480 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:20,480 INFO
2024-04-19 21:04:20,480 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:20,480 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:20,784 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:20,784 DEBUG --- stdout ---
2024-04-19 21:04:20,785 DEBUG 1
2024-04-19 21:04:20,785 DEBUG --- stderr ---
2024-04-19 21:04:20,785 DEBUG
2024-04-19 21:04:20,785 INFO
2024-04-19 21:04:20,785 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:04:20,785 INFO
2024-04-19 21:04:20,785 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-19 21:04:20,785 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:04:21,092 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-19 21:04:21,092 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:21,092 DEBUG --- stdout ---
2024-04-19 21:04:21,092 DEBUG login-ui-5fdf49fc66-dlhjd
2024-04-19 21:04:21,092 DEBUG --- stderr ---
2024-04-19 21:04:21,092 DEBUG
2024-04-19 21:04:21,092 INFO
2024-04-19 21:04:21,092 INFO ----------- Check pod login-ui-5fdf49fc66-dlhjd is running -----------
2024-04-19 21:04:21,093 INFO
2024-04-19 21:04:21,093 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods login-ui-5fdf49fc66-dlhjd -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:04:21,093 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:21,422 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:21,422 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:21,422 DEBUG --- stdout ---
2024-04-19 21:04:21,422 DEBUG Running
2024-04-19 21:04:21,422 DEBUG --- stderr ---
2024-04-19 21:04:21,422 DEBUG
2024-04-19 21:04:21,423 INFO
2024-04-19 21:04:21,423 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods login-ui-5fdf49fc66-dlhjd -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:04:21,423 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:21,738 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:21,739 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:21,739 DEBUG --- stdout ---
2024-04-19 21:04:21,739 DEBUG true
2024-04-19 21:04:21,739 DEBUG --- stderr ---
2024-04-19 21:04:21,739 DEBUG
2024-04-19 21:04:21,739 INFO
2024-04-19 21:04:21,739 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod login-ui-5fdf49fc66-dlhjd --output jsonpath={.status.startTime}
2024-04-19 21:04:21,739 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:22,049 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:22,050 DEBUG --- stdout ---
2024-04-19 21:04:22,050 DEBUG 2024-04-17T16:01:55Z
2024-04-19 21:04:22,050 DEBUG --- stderr ---
2024-04-19 21:04:22,050 DEBUG
2024-04-19 21:04:22,051 INFO
2024-04-19 21:04:22,051 INFO ---- Check pod login-ui-5fdf49fc66-dlhjd filesystem is accessible ----
2024-04-19 21:04:22,813 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:22,813 INFO
2024-04-19 21:04:22,813 INFO ---------- Check pod login-ui-5fdf49fc66-dlhjd restart count ----------
2024-04-19 21:04:22,813 INFO
2024-04-19 21:04:22,813 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod login-ui-5fdf49fc66-dlhjd --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:04:22,813 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:23,107 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:23,107 DEBUG --- stdout ---
2024-04-19 21:04:23,108 DEBUG 0
2024-04-19 21:04:23,108 DEBUG --- stderr ---
2024-04-19 21:04:23,108 DEBUG
2024-04-19 21:04:23,108 INFO Pod login-ui-5fdf49fc66-dlhjd has been restarted 0 times.
2024-04-19 21:04:23,108 INFO
2024-04-19 21:04:23,108 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-19 21:04:23,108 INFO
2024-04-19 21:04:23,108 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:23,108 INFO
2024-04-19 21:04:23,108 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:23,108 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:23,407 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:23,408 DEBUG --- stdout ---
2024-04-19 21:04:23,408 DEBUG 1
2024-04-19 21:04:23,408 DEBUG --- stderr ---
2024-04-19 21:04:23,408 DEBUG
2024-04-19 21:04:23,408 INFO
2024-04-19 21:04:23,408 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:04:23,408 INFO
2024-04-19 21:04:23,408 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-19 21:04:23,408 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:04:23,707 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-19 21:04:23,708 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:23,708 DEBUG --- stdout ---
2024-04-19 21:04:23,708 DEBUG admin-ui-c4c7bf584-tlgkc
2024-04-19 21:04:23,708 DEBUG --- stderr ---
2024-04-19 21:04:23,708 DEBUG
2024-04-19 21:04:23,708 INFO
2024-04-19 21:04:23,708 INFO ------------ Check pod admin-ui-c4c7bf584-tlgkc is running ------------
2024-04-19 21:04:23,708 INFO
2024-04-19 21:04:23,708 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods admin-ui-c4c7bf584-tlgkc -o=jsonpath={.status.phase} | grep "Running"
2024-04-19 21:04:23,708 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:24,018 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:24,018 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:24,018 DEBUG --- stdout ---
2024-04-19 21:04:24,019 DEBUG Running
2024-04-19 21:04:24,019 DEBUG --- stderr ---
2024-04-19 21:04:24,019 DEBUG
2024-04-19 21:04:24,019 INFO
2024-04-19 21:04:24,019 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods admin-ui-c4c7bf584-tlgkc -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-19 21:04:24,019 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-19 21:04:24,323 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:24,323 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:24,323 DEBUG --- stdout ---
2024-04-19 21:04:24,323 DEBUG true
2024-04-19 21:04:24,323 DEBUG --- stderr ---
2024-04-19 21:04:24,323 DEBUG
2024-04-19 21:04:24,324 INFO
2024-04-19 21:04:24,324 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod admin-ui-c4c7bf584-tlgkc --output jsonpath={.status.startTime}
2024-04-19 21:04:24,324 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:24,638 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:24,638 DEBUG --- stdout ---
2024-04-19 21:04:24,638 DEBUG 2024-04-17T16:01:54Z
2024-04-19 21:04:24,638 DEBUG --- stderr ---
2024-04-19 21:04:24,638 DEBUG
2024-04-19 21:04:24,639 INFO
2024-04-19 21:04:24,639 INFO ----- Check pod admin-ui-c4c7bf584-tlgkc filesystem is accessible -----
2024-04-19 21:04:25,403 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:25,403 INFO
2024-04-19 21:04:25,404 INFO ---------- Check pod admin-ui-c4c7bf584-tlgkc restart count ----------
2024-04-19 21:04:25,405 INFO
2024-04-19 21:04:25,405 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod admin-ui-c4c7bf584-tlgkc --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-19 21:04:25,405 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:25,713 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:25,713 DEBUG --- stdout ---
2024-04-19 21:04:25,713 DEBUG 0
2024-04-19 21:04:25,713 DEBUG --- stderr ---
2024-04-19 21:04:25,713 DEBUG
2024-04-19 21:04:25,713 INFO Pod admin-ui-c4c7bf584-tlgkc has been restarted 0 times.
2024-04-19 21:04:25,713 INFO
2024-04-19 21:04:25,713 INFO ***************************** Checking DS-CTS component is running *****************************
2024-04-19 21:04:25,713 INFO
2024-04-19 21:04:25,713 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:25,714 INFO
2024-04-19 21:04:25,714 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:25,714 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:26,097 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:26,097 DEBUG --- stdout ---
2024-04-19 21:04:26,097 DEBUG 3
2024-04-19 21:04:26,097 DEBUG --- stderr ---
2024-04-19 21:04:26,098 DEBUG
2024-04-19 21:04:26,098 INFO
2024-04-19 21:04:26,098 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-19 21:04:26,098 INFO
2024-04-19 21:04:26,098 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-cts --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3"
2024-04-19 21:04:26,098 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-19 21:04:26,490 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:26,490 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:26,490 DEBUG --- stdout ---
2024-04-19 21:04:26,490 DEBUG current:3 ready:3 replicas:3
2024-04-19 21:04:26,490 DEBUG --- stderr ---
2024-04-19 21:04:26,490 DEBUG
2024-04-19 21:04:26,490 INFO
2024-04-19 21:04:26,491 INFO *************************** Checking DS-IDREPO component is running ***************************
2024-04-19 21:04:26,491 INFO
2024-04-19 21:04:26,491 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:26,491 INFO
2024-04-19 21:04:26,491 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:26,491 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:26,867 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:26,868 DEBUG --- stdout ---
2024-04-19 21:04:26,868 DEBUG 3
2024-04-19 21:04:26,868 DEBUG --- stderr ---
2024-04-19 21:04:26,868 DEBUG
2024-04-19 21:04:26,868 INFO
2024-04-19 21:04:26,868 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-19 21:04:26,868 INFO
2024-04-19 21:04:26,868 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-idrepo --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3"
2024-04-19 21:04:26,868 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-19 21:04:27,267 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:27,267 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:27,267 DEBUG --- stdout ---
2024-04-19 21:04:27,267 DEBUG current:3 ready:3 replicas:3
2024-04-19 21:04:27,267 DEBUG --- stderr ---
2024-04-19 21:04:27,267 DEBUG
2024-04-19 21:04:27,267 INFO
2024-04-19 21:04:27,267 INFO ******************************* Checking AM component is running *******************************
2024-04-19 21:04:27,267 INFO
2024-04-19 21:04:27,267 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:27,268 INFO
2024-04-19 21:04:27,268 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:27,268 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:27,653 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:27,653 DEBUG --- stdout ---
2024-04-19 21:04:27,653 DEBUG 3
2024-04-19 21:04:27,653 DEBUG --- stderr ---
2024-04-19 21:04:27,653 DEBUG
2024-04-19 21:04:27,653 INFO
2024-04-19 21:04:27,653 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-19 21:04:27,653 INFO
2024-04-19 21:04:27,654 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments am --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:3 replicas:3"
2024-04-19 21:04:27,654 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-19 21:04:28,049 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:28,050 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:28,050 DEBUG --- stdout ---
2024-04-19 21:04:28,050 DEBUG ready:3 replicas:3
2024-04-19 21:04:28,050 DEBUG --- stderr ---
2024-04-19 21:04:28,050 DEBUG
2024-04-19 21:04:28,050 INFO
2024-04-19 21:04:28,050 INFO ***************************** Checking AMSTER component is running *****************************
2024-04-19 21:04:28,050 INFO
2024-04-19 21:04:28,050 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:28,050 INFO
2024-04-19 21:04:28,050 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-19 21:04:28,050 INFO
2024-04-19 21:04:28,050 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get jobs amster --output jsonpath="{.status.succeeded}" | grep "1"
2024-04-19 21:04:28,050 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-19 21:04:28,419 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:28,419 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:28,419 DEBUG --- stdout ---
2024-04-19 21:04:28,420 DEBUG 1
2024-04-19 21:04:28,420 DEBUG --- stderr ---
2024-04-19 21:04:28,420 DEBUG
2024-04-19 21:04:28,420 INFO
2024-04-19 21:04:28,420 INFO ****************************** Checking IDM component is running ******************************
2024-04-19 21:04:28,420 INFO
2024-04-19 21:04:28,420 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:28,420 INFO
2024-04-19 21:04:28,420 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:28,420 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:28,795 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:28,795 DEBUG --- stdout ---
2024-04-19 21:04:28,795 DEBUG 2
2024-04-19 21:04:28,795 DEBUG --- stderr ---
2024-04-19 21:04:28,795 DEBUG
2024-04-19 21:04:28,795 INFO
2024-04-19 21:04:28,795 INFO -------------- Waiting for 2 expected pod(s) to be ready --------------
2024-04-19 21:04:28,795 INFO
2024-04-19 21:04:28,795 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment idm --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:2 replicas:2"
2024-04-19 21:04:28,795 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-19 21:04:29,180 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:29,180 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:29,180 DEBUG --- stdout ---
2024-04-19 21:04:29,180 DEBUG ready:2 replicas:2
2024-04-19 21:04:29,180 DEBUG --- stderr ---
2024-04-19 21:04:29,180 DEBUG
2024-04-19 21:04:29,180 INFO
2024-04-19 21:04:29,180 INFO ************************** Checking END-USER-UI component is running **************************
2024-04-19 21:04:29,180 INFO
2024-04-19 21:04:29,180 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:29,181 INFO
2024-04-19 21:04:29,181 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:29,181 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:29,488 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:29,488 DEBUG --- stdout ---
2024-04-19 21:04:29,488 DEBUG 1
2024-04-19 21:04:29,488 DEBUG --- stderr ---
2024-04-19 21:04:29,488 DEBUG
2024-04-19 21:04:29,488 INFO
2024-04-19 21:04:29,488 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-19 21:04:29,489 INFO
2024-04-19 21:04:29,489 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments end-user-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-19 21:04:29,489 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-19 21:04:29,807 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:29,807 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:29,807 DEBUG --- stdout ---
2024-04-19 21:04:29,807 DEBUG ready:1 replicas:1
2024-04-19 21:04:29,807 DEBUG --- stderr ---
2024-04-19 21:04:29,807 DEBUG
2024-04-19 21:04:29,807 INFO
2024-04-19 21:04:29,807 INFO **************************** Checking LOGIN-UI component is running ****************************
2024-04-19 21:04:29,807 INFO
2024-04-19 21:04:29,807 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:29,808 INFO
2024-04-19 21:04:29,808 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:29,808 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:30,106 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:30,106 DEBUG --- stdout ---
2024-04-19 21:04:30,106 DEBUG 1
2024-04-19 21:04:30,106 DEBUG --- stderr ---
2024-04-19 21:04:30,106 DEBUG
2024-04-19 21:04:30,106 INFO
2024-04-19 21:04:30,106 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-19 21:04:30,107 INFO
2024-04-19 21:04:30,107 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments login-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-19 21:04:30,107 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-19 21:04:30,420 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:30,422 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:30,422 DEBUG --- stdout ---
2024-04-19 21:04:30,422 DEBUG ready:1 replicas:1
2024-04-19 21:04:30,422 DEBUG --- stderr ---
2024-04-19 21:04:30,422 DEBUG
2024-04-19 21:04:30,422 INFO
2024-04-19 21:04:30,423 INFO **************************** Checking ADMIN-UI component is running ****************************
2024-04-19 21:04:30,423 INFO
2024-04-19 21:04:30,423 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:30,423 INFO
2024-04-19 21:04:30,423 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:30,423 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:30,734 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:30,734 DEBUG --- stdout ---
2024-04-19 21:04:30,734 DEBUG 1
2024-04-19 21:04:30,734 DEBUG --- stderr ---
2024-04-19 21:04:30,734 DEBUG
2024-04-19 21:04:30,734 INFO
2024-04-19 21:04:30,734 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-19 21:04:30,734 INFO
2024-04-19 21:04:30,735 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments admin-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-19 21:04:30,735 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-19 21:04:31,062 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-19 21:04:31,063 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:31,063 DEBUG --- stdout ---
2024-04-19 21:04:31,063 DEBUG ready:1 replicas:1
2024-04-19 21:04:31,063 DEBUG --- stderr ---
2024-04-19 21:04:31,063 DEBUG
2024-04-19 21:04:31,063 INFO
2024-04-19 21:04:31,063 INFO ****************************** Livecheck stage: After deployment ******************************
2024-04-19 21:04:31,063 INFO
2024-04-19 21:04:31,063 INFO ---------------------- Running DS-CTS livecheck ----------------------
2024-04-19 21:04:31,063 INFO Livecheck to ds-cts-0
2024-04-19 21:04:31,063 INFO
2024-04-19 21:04:31,063 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.dirmanager\.pw}"
2024-04-19 21:04:31,063 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-19 21:04:31,386 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:31,386 DEBUG --- stdout ---
2024-04-19 21:04:31,386 DEBUG RGlLaTRoUUY2dlkyc3hUWTM0enhpbUt0dW1yU2QxelM=
2024-04-19 21:04:31,386 DEBUG --- stderr ---
2024-04-19 21:04:31,386 DEBUG
2024-04-19 21:04:31,386 INFO
2024-04-19 21:04:31,387 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-19 21:04:33,255 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:04:33,256 DEBUG --- stdout ---
2024-04-19 21:04:33,256 DEBUG dn:
alive: true
2024-04-19 21:04:33,256 DEBUG --- stderr ---
2024-04-19 21:04:33,256 DEBUG
2024-04-19 21:04:33,256 INFO
2024-04-19 21:04:33,256 INFO Livecheck to ds-cts-1
2024-04-19 21:04:33,256 INFO
2024-04-19 21:04:33,256 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-19 21:04:35,013 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:04:35,013 DEBUG --- stdout ---
2024-04-19 21:04:35,013 DEBUG dn:
alive: true
2024-04-19 21:04:35,013 DEBUG --- stderr ---
2024-04-19 21:04:35,013 DEBUG
2024-04-19 21:04:35,013 INFO
2024-04-19 21:04:35,013 INFO Livecheck to ds-cts-2
2024-04-19 21:04:35,013 INFO
2024-04-19 21:04:35,014 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-19 21:04:36,763 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:04:36,763 DEBUG --- stdout ---
2024-04-19 21:04:36,764 DEBUG dn:
alive: true
2024-04-19 21:04:36,764 DEBUG --- stderr ---
2024-04-19 21:04:36,764 DEBUG
2024-04-19 21:04:36,764 INFO
2024-04-19 21:04:36,764 INFO
2024-04-19 21:04:36,764 INFO --------------------- Running DS-IDREPO livecheck ---------------------
2024-04-19 21:04:36,764 INFO Livecheck to ds-idrepo-0
2024-04-19 21:04:36,764 INFO
2024-04-19 21:04:36,764 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.dirmanager\.pw}"
2024-04-19 21:04:36,764 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-19 21:04:37,057 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:37,057 DEBUG --- stdout ---
2024-04-19 21:04:37,057 DEBUG RGlLaTRoUUY2dlkyc3hUWTM0enhpbUt0dW1yU2QxelM=
2024-04-19 21:04:37,058 DEBUG --- stderr ---
2024-04-19 21:04:37,058 DEBUG
2024-04-19 21:04:37,058 INFO
2024-04-19 21:04:37,058 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-19 21:04:38,808 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:04:38,808 DEBUG --- stdout ---
2024-04-19 21:04:38,808 DEBUG dn:
alive: true
2024-04-19 21:04:38,808 DEBUG --- stderr ---
2024-04-19 21:04:38,808 DEBUG
2024-04-19 21:04:38,808 INFO
2024-04-19 21:04:38,808 INFO Livecheck to ds-idrepo-1
2024-04-19 21:04:38,808 INFO
2024-04-19 21:04:38,809 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-19 21:04:40,640 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:04:40,640 DEBUG --- stdout ---
2024-04-19 21:04:40,640 DEBUG dn:
alive: true
2024-04-19 21:04:40,640 DEBUG --- stderr ---
2024-04-19 21:04:40,640 DEBUG
2024-04-19 21:04:40,640 INFO
2024-04-19 21:04:40,640 INFO Livecheck to ds-idrepo-2
2024-04-19 21:04:40,640 INFO
2024-04-19 21:04:40,641 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-19 21:04:42,419 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-19 21:04:42,421 DEBUG --- stdout ---
2024-04-19 21:04:42,421 DEBUG dn:
alive: true
2024-04-19 21:04:42,421 DEBUG --- stderr ---
2024-04-19 21:04:42,421 DEBUG
2024-04-19 21:04:42,421 INFO
2024-04-19 21:04:42,421 INFO
2024-04-19 21:04:42,421 INFO ------------------------ Running AM livecheck ------------------------
2024-04-19 21:04:42,422 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready
2024-04-19 21:04:42,422 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready"
2024-04-19 21:04:42,706 INFO [http_cmd]: http status code OK
2024-04-19 21:04:42,706 DEBUG --- status code ---
2024-04-19 21:04:42,706 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:42,706 DEBUG --- http response ---
2024-04-19 21:04:42,706 DEBUG
2024-04-19 21:04:42,707 INFO
2024-04-19 21:04:42,707 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.AM_PASSWORDS_AMADMIN_CLEAR}"
2024-04-19 21:04:42,707 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-19 21:04:43,036 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:43,036 DEBUG --- stdout ---
2024-04-19 21:04:43,037 DEBUG dUl1ZjBBejNacjd5Nk94akxLYjF6eWZj
2024-04-19 21:04:43,037 DEBUG --- stderr ---
2024-04-19 21:04:43,037 DEBUG
2024-04-19 21:04:43,037 INFO Authenticate user amadmin via REST
2024-04-19 21:04:43,038 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/"
2024-04-19 21:04:43,268 INFO [http_cmd]: http status code OK
2024-04-19 21:04:43,269 DEBUG --- status code ---
2024-04-19 21:04:43,269 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:43,269 DEBUG --- http response ---
2024-04-19 21:04:43,270 DEBUG {
"tokenId": "g1UZ7Pt5sIrG2uXeO3-tH1rMD0c.*AAJTSQACMDIAAlNLABxpOVlTL0pvanJaV2lIVkJWQWVhV1JaRklNVE09AAR0eXBlAANDVFMAAlMxAAIwMQ..*",
"successUrl": "/am/console",
"realm": "/"
}
2024-04-19 21:04:43,275 INFO
2024-04-19 21:04:43,277 INFO ---------------------- Running AMSTER livecheck ----------------------
2024-04-19 21:04:43,277 INFO
2024-04-19 21:04:43,277 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:04:43,278 INFO
2024-04-19 21:04:43,278 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm --field-selector status.phase!=Failed get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-19 21:04:43,278 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:04:43,708 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:43,708 DEBUG --- stdout ---
2024-04-19 21:04:43,708 DEBUG amster-grkh7
2024-04-19 21:04:43,708 DEBUG --- stderr ---
2024-04-19 21:04:43,708 DEBUG
2024-04-19 21:04:44,384 INFO Amster livecheck is passed
2024-04-19 21:04:44,385 INFO
2024-04-19 21:04:44,385 INFO ------------------------ Running IDM livecheck ------------------------
2024-04-19 21:04:44,385 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/ping
2024-04-19 21:04:44,385 INFO
2024-04-19 21:04:44,385 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.OPENIDM_ADMIN_PASSWORD}"
2024-04-19 21:04:44,385 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-19 21:04:44,735 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:44,735 DEBUG --- stdout ---
2024-04-19 21:04:44,735 DEBUG TzlLMVZYWVZKaXlpbVNtbTRWUU54clFV
2024-04-19 21:04:44,735 DEBUG --- stderr ---
2024-04-19 21:04:44,735 DEBUG
2024-04-19 21:04:44,735 INFO Set admin password: O9K1VXYVJiyimSmm4VQNxrQU
2024-04-19 21:04:44,736 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/ping"
2024-04-19 21:04:44,954 INFO [http_cmd]: http status code OK
2024-04-19 21:04:44,954 DEBUG --- status code ---
2024-04-19 21:04:44,954 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:44,954 DEBUG --- http response ---
2024-04-19 21:04:44,954 DEBUG {
"_id": "",
"_rev": "",
"shortDesc": "OpenIDM ready",
"state": "ACTIVE_READY"
}
2024-04-19 21:04:44,955 INFO Create new LoginSession for user "amadmin"
2024-04-19 21:04:44,955 INFO [LoginSession] Get "access_token" token for user "amadmin"
2024-04-19 21:04:44,955 INFO [LoginSession] Obtaining new "access_token" token from server because it was not obtained yet for the user
2024-04-19 21:04:44,955 INFO [LoginSession] Get "token_id" token for user "amadmin"
2024-04-19 21:04:44,955 INFO [LoginSession] Obtaining new "token_id" token from server because it was not obtained yet for the user
2024-04-19 21:04:44,955 INFO Authenticate user amadmin via REST
2024-04-19 21:04:44,955 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/"
2024-04-19 21:04:45,170 INFO [http_cmd]: http status code OK
2024-04-19 21:04:45,170 DEBUG --- status code ---
2024-04-19 21:04:45,170 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:45,170 DEBUG --- http response ---
2024-04-19 21:04:45,171 DEBUG {
"tokenId": "_sSPLOQsczoaDxOkfOZPXNEn8cU.*AAJTSQACMDIAAlNLABxTWENiYnNoMlBRYTJFODBaeTBKVDFTN2dnbmc9AAR0eXBlAANDVFMAAlMxAAIwMQ..*",
"successUrl": "/am/console",
"realm": "/"
}
2024-04-19 21:04:45,171 INFO [LoginSession] Obtaining session info
2024-04-19 21:04:45,171 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/realms/root/sessions?_action=getSessionInfo"
2024-04-19 21:04:45,388 INFO [http_cmd]: http status code OK
2024-04-19 21:04:45,388 DEBUG --- status code ---
2024-04-19 21:04:45,388 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:45,388 DEBUG --- http response ---
2024-04-19 21:04:45,388 DEBUG {
"username": "amadmin",
"universalId": "id=amadmin,ou=user,ou=am-config",
"realm": "/",
"latestAccessTime": "2024-04-19T21:04:45Z",
"maxIdleExpirationTime": "2024-04-19T21:07:45Z",
"maxSessionExpirationTime": "2024-04-19T21:07:44Z",
"properties": {
"AMCtxId": "ea2ffb82-dff1-4fbb-a5c5-c826431ff3d8-46975798"
}
}
2024-04-19 21:04:45,392 INFO Obtain Oauth2 authz code via REST
2024-04-19 21:04:45,392 DEBUG Oauth2 authorize for access token
2024-04-19 21:04:45,392 INFO [http_cmd]: curl --request POST --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=_sSPLOQsczoaDxOkfOZPXNEn8cU.*AAJTSQACMDIAAlNLABxTWENiYnNoMlBRYTJFODBaeTBKVDFTN2dnbmc9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" --data '{"decision": "Allow", "csrf": "_sSPLOQsczoaDxOkfOZPXNEn8cU.*AAJTSQACMDIAAlNLABxTWENiYnNoMlBRYTJFODBaeTBKVDFTN2dnbmc9AAR0eXBlAANDVFMAAlMxAAIwMQ..*"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/am/oauth2/authorize?client_id=smokeclient&scope=fr:idm:*&redirect_uri=https://fake.com&response_type=code&realm=/"
2024-04-19 21:04:45,619 INFO [http_cmd]: http status code OK
2024-04-19 21:04:45,619 DEBUG --- status code ---
2024-04-19 21:04:45,619 DEBUG http status code is 302 (expected 302)
2024-04-19 21:04:45,619 DEBUG --- http response ---
2024-04-19 21:04:45,619 DEBUG {'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'cache-control': 'no-store', 'location': 'https://fake.com?code=DvojcaH59gdiOvHyGsx4gEEkUws&iss=https%3A%2F%2Fxlou.iam.xlou-cdm.engineeringpit.com%2Fam%2Foauth2&client_id=smokeclient', 'pragma': 'no-cache', 'set-cookie': 'OAUTH_REQUEST_ATTRIBUTES=DELETED; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Path=/; Secure; HttpOnly; SameSite=none', 'content-length': '0', 'date': 'Fri, 19 Apr 2024 21:04:44 GMT'}
2024-04-19 21:04:45,619 DEBUG Oauth2 exchange authorize code for access token
2024-04-19 21:04:45,619 INFO [http_cmd]: curl -L --request POST --data '{"grant_type": "authorization_code", "redirect_uri": "https://fake.com", "client_id": "smokeclient", "code": "DvojcaH59gdiOvHyGsx4gEEkUws"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/am/oauth2/access_token?realm=/"
2024-04-19 21:04:45,699 INFO [http_cmd]: http status code OK
2024-04-19 21:04:45,699 DEBUG --- status code ---
2024-04-19 21:04:45,699 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:45,699 DEBUG --- http response ---
2024-04-19 21:04:45,700 DEBUG {
"access_token": "Lg2fctNjEQo2X3mqWwCYti7kzbQ",
"scope": "fr:idm:*",
"token_type": "Bearer",
"expires_in": 3599
}
2024-04-19 21:04:45,701 INFO Additional check to sync/queue to ensure sync with DS is ok (see LODESTAR-1245)
2024-04-19 21:04:45,702 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/sync/queue?_queryFilter=true&_pageSize=1"
2024-04-19 21:04:45,927 INFO [http_cmd]: http status code OK
2024-04-19 21:04:45,927 DEBUG --- status code ---
2024-04-19 21:04:45,927 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:45,927 DEBUG --- http response ---
2024-04-19 21:04:45,927 DEBUG {
"result": [],
"resultCount": 0,
"pagedResultsCookie": null,
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}
2024-04-19 21:04:45,929 INFO Try to create, query and delete a user
2024-04-19 21:04:45,929 INFO Create user idm_49c269b9lb in IDM
2024-04-19 21:04:45,930 INFO [http_cmd]: curl --header "Authorization: Bearer Lg2fctNjEQo2X3mqWwCYti7kzbQ" --header "Content-Type: application/json" --insecure -L --request POST --data '{"userName": "idm_49c269b9lb", "givenName": "Peter", "sn": "Parker", "password": "T35tr0ck123", "telephoneNumber": "6669876987", "description": "IDM Test User", "roles": [], "mail": "idm_49c269b9lb@forgerock.com"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user?_action=create"
2024-04-19 21:04:46,185 INFO [http_cmd]: http status code OK
2024-04-19 21:04:46,185 DEBUG --- status code ---
2024-04-19 21:04:46,185 DEBUG http status code is 201 (expected [200, 201])
2024-04-19 21:04:46,185 DEBUG --- http response ---
2024-04-19 21:04:46,185 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-19 21:04:46,185 DEBUG ----- output -----
2024-04-19 21:04:46,185 DEBUG {
2024-04-19 21:04:46,185 DEBUG "_id": "4aac4ac5-6a8d-4af1-b40b-63521d6afa0e",
2024-04-19 21:04:46,185 DEBUG "_rev": "34651f31-1c43-4774-828c-270d64a997a7-72770",
2024-04-19 21:04:46,185 DEBUG "country": null,
2024-04-19 21:04:46,185 DEBUG "mail": "idm_49c269b9lb@forgerock.com",
2024-04-19 21:04:46,185 DEBUG "memberOfOrgIDs": [],
2024-04-19 21:04:46,185 DEBUG "postalCode": null,
2024-04-19 21:04:46,185 DEBUG "profileImage": null,
2024-04-19 21:04:46,185 DEBUG "expireAccount": null,
2024-04-19 21:04:46,185 DEBUG "activeDate": null,
2024-04-19 21:04:46,186 DEBUG
2024-04-19 21:04:46,186 DEBUG
2024-04-19 21:04:46,186 DEBUG
2024-04-19 21:04:46,186 DEBUG [...]
2024-04-19 21:04:46,186 DEBUG
2024-04-19 21:04:46,186 DEBUG
2024-04-19 21:04:46,186 DEBUG
2024-04-19 21:04:46,186 DEBUG "aliasList": [],
2024-04-19 21:04:46,186 DEBUG "kbaInfo": [],
2024-04-19 21:04:46,186 DEBUG "inactiveDate": null,
2024-04-19 21:04:46,186 DEBUG "sn": "Parker",
2024-04-19 21:04:46,186 DEBUG "preferences": null,
2024-04-19 21:04:46,186 DEBUG "userName": "idm_49c269b9lb",
2024-04-19 21:04:46,186 DEBUG "passwordLastChangedTime": "2024-04-19T21:04:45.968Z",
2024-04-19 21:04:46,186 DEBUG "effectiveRoles": [],
2024-04-19 21:04:46,186 DEBUG "activateAccount": null
2024-04-19 21:04:46,186 DEBUG }
2024-04-19 21:04:46,186 INFO Query IDM to verify UserID 4aac4ac5-6a8d-4af1-b40b-63521d6afa0e is visible
2024-04-19 21:04:46,186 INFO [http_cmd]: curl --insecure -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user/4aac4ac5-6a8d-4af1-b40b-63521d6afa0e"
2024-04-19 21:04:46,394 INFO [http_cmd]: http status code OK
2024-04-19 21:04:46,394 DEBUG --- status code ---
2024-04-19 21:04:46,394 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:46,394 DEBUG --- http response ---
2024-04-19 21:04:46,394 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-19 21:04:46,395 DEBUG ----- output -----
2024-04-19 21:04:46,395 DEBUG {
2024-04-19 21:04:46,395 DEBUG "_id": "4aac4ac5-6a8d-4af1-b40b-63521d6afa0e",
2024-04-19 21:04:46,395 DEBUG "_rev": "34651f31-1c43-4774-828c-270d64a997a7-72770",
2024-04-19 21:04:46,395 DEBUG "country": null,
2024-04-19 21:04:46,395 DEBUG "mail": "idm_49c269b9lb@forgerock.com",
2024-04-19 21:04:46,395 DEBUG "memberOfOrgIDs": [],
2024-04-19 21:04:46,395 DEBUG "postalCode": null,
2024-04-19 21:04:46,395 DEBUG "profileImage": null,
2024-04-19 21:04:46,395 DEBUG "expireAccount": null,
2024-04-19 21:04:46,395 DEBUG "activeDate": null,
2024-04-19 21:04:46,395 DEBUG
2024-04-19 21:04:46,395 DEBUG
2024-04-19 21:04:46,395 DEBUG
2024-04-19 21:04:46,395 DEBUG [...]
2024-04-19 21:04:46,395 DEBUG
2024-04-19 21:04:46,395 DEBUG
2024-04-19 21:04:46,395 DEBUG
2024-04-19 21:04:46,395 DEBUG "aliasList": [],
2024-04-19 21:04:46,395 DEBUG "kbaInfo": [],
2024-04-19 21:04:46,395 DEBUG "inactiveDate": null,
2024-04-19 21:04:46,395 DEBUG "sn": "Parker",
2024-04-19 21:04:46,395 DEBUG "preferences": null,
2024-04-19 21:04:46,395 DEBUG "userName": "idm_49c269b9lb",
2024-04-19 21:04:46,395 DEBUG "passwordLastChangedTime": "2024-04-19T21:04:45.968Z",
2024-04-19 21:04:46,395 DEBUG "effectiveRoles": [],
2024-04-19 21:04:46,395 DEBUG "activateAccount": null
2024-04-19 21:04:46,395 DEBUG }
2024-04-19 21:04:46,396 INFO Authenticate as user idm_49c269b9lb
2024-04-19 21:04:46,396 INFO [http_cmd]: curl --insecure -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/authentication?_action=login"
2024-04-19 21:04:46,603 INFO [http_cmd]: http status code OK
2024-04-19 21:04:46,604 DEBUG --- status code ---
2024-04-19 21:04:46,604 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:46,604 DEBUG --- http response ---
2024-04-19 21:04:46,604 DEBUG {
"_id": "login",
"authorization": {
"id": "anonymous",
"roles": [
"internal/role/openidm-reg"
],
"component": "internal/user"
},
"authenticationId": "anonymous"
}
2024-04-19 21:04:46,605 INFO [http_cmd]: curl --insecure -L --request DELETE "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user/4aac4ac5-6a8d-4af1-b40b-63521d6afa0e"
2024-04-19 21:04:46,829 INFO [http_cmd]: http status code OK
2024-04-19 21:04:46,829 DEBUG --- status code ---
2024-04-19 21:04:46,829 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:46,829 DEBUG --- http response ---
2024-04-19 21:04:46,829 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-19 21:04:46,829 DEBUG ----- output -----
2024-04-19 21:04:46,829 DEBUG {
2024-04-19 21:04:46,829 DEBUG "_id": "4aac4ac5-6a8d-4af1-b40b-63521d6afa0e",
2024-04-19 21:04:46,829 DEBUG "_rev": "34651f31-1c43-4774-828c-270d64a997a7-72770",
2024-04-19 21:04:46,829 DEBUG "country": null,
2024-04-19 21:04:46,829 DEBUG "mail": "idm_49c269b9lb@forgerock.com",
2024-04-19 21:04:46,829 DEBUG "memberOfOrgIDs": [],
2024-04-19 21:04:46,829 DEBUG "postalCode": null,
2024-04-19 21:04:46,829 DEBUG "profileImage": null,
2024-04-19 21:04:46,829 DEBUG "expireAccount": null,
2024-04-19 21:04:46,829 DEBUG "activeDate": null,
2024-04-19 21:04:46,830 DEBUG
2024-04-19 21:04:46,830 DEBUG
2024-04-19 21:04:46,830 DEBUG
2024-04-19 21:04:46,830 DEBUG [...]
2024-04-19 21:04:46,830 DEBUG
2024-04-19 21:04:46,830 DEBUG
2024-04-19 21:04:46,830 DEBUG
2024-04-19 21:04:46,830 DEBUG "aliasList": [],
2024-04-19 21:04:46,830 DEBUG "kbaInfo": [],
2024-04-19 21:04:46,830 DEBUG "inactiveDate": null,
2024-04-19 21:04:46,830 DEBUG "sn": "Parker",
2024-04-19 21:04:46,830 DEBUG "preferences": null,
2024-04-19 21:04:46,830 DEBUG "userName": "idm_49c269b9lb",
2024-04-19 21:04:46,830 DEBUG "passwordLastChangedTime": "2024-04-19T21:04:45.968Z",
2024-04-19 21:04:46,830 DEBUG "effectiveRoles": [],
2024-04-19 21:04:46,830 DEBUG "activateAccount": null
2024-04-19 21:04:46,830 DEBUG }
2024-04-19 21:04:46,832 INFO
2024-04-19 21:04:46,832 INFO -------------------- Running END-USER-UI livecheck --------------------
2024-04-19 21:04:46,832 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/enduser
2024-04-19 21:04:46,833 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/enduser"
2024-04-19 21:04:47,109 INFO [http_cmd]: http status code OK
2024-04-19 21:04:47,109 DEBUG --- status code ---
2024-04-19 21:04:47,109 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:47,109 DEBUG --- http response ---
2024-04-19 21:04:47,109 DEBUG Identity Management
[]
2024-04-19 21:04:47,111 INFO
2024-04-19 21:04:47,111 INFO --------------------- Running LOGIN-UI livecheck ---------------------
2024-04-19 21:04:47,111 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/XUI
2024-04-19 21:04:47,112 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/XUI"
2024-04-19 21:04:47,391 INFO [http_cmd]: http status code OK
2024-04-19 21:04:47,391 DEBUG --- status code ---
2024-04-19 21:04:47,391 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:47,391 DEBUG --- http response ---
2024-04-19 21:04:47,391 DEBUG Login
[]
2024-04-19 21:04:47,393 INFO
2024-04-19 21:04:47,393 INFO --------------------- Running ADMIN-UI livecheck ---------------------
2024-04-19 21:04:47,393 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/platform
2024-04-19 21:04:47,394 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/platform"
2024-04-19 21:04:47,671 INFO [http_cmd]: http status code OK
2024-04-19 21:04:47,672 DEBUG --- status code ---
2024-04-19 21:04:47,672 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:47,672 DEBUG --- http response ---
2024-04-19 21:04:47,672 DEBUG Platform Admin
[]
2024-04-19 21:04:47,673 INFO LIVECHECK SUCCEEDED
2024-04-19 21:04:47,674 INFO
2024-04-19 21:04:47,674 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-19 21:04:47,674 INFO
2024-04-19 21:04:47,674 INFO --------------------- Get DS-CTS software version ---------------------
2024-04-19 21:04:47,674 INFO
2024-04-19 21:04:47,675 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- find /opt/opendj -name opendj-core.jar
2024-04-19 21:04:47,675 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-19 21:04:48,512 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:48,513 DEBUG --- stdout ---
2024-04-19 21:04:48,513 DEBUG /opt/opendj/lib/opendj-core.jar
2024-04-19 21:04:48,513 DEBUG --- stderr ---
2024-04-19 21:04:48,513 DEBUG
2024-04-19 21:04:48,513 INFO
2024-04-19 21:04:48,513 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-cts-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-cts_info/opendj-core.jar --container ds
2024-04-19 21:04:48,513 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:49,784 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:49,784 DEBUG --- stdout ---
2024-04-19 21:04:49,784 DEBUG tar: Removing leading `/' from member names
2024-04-19 21:04:49,784 DEBUG --- stderr ---
2024-04-19 21:04:49,784 DEBUG
2024-04-19 21:04:49,794 INFO
2024-04-19 21:04:49,794 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-19 21:04:49,794 INFO
2024-04-19 21:04:49,794 INFO ------------------- Get DS-IDREPO software version -------------------
2024-04-19 21:04:49,794 INFO
2024-04-19 21:04:49,794 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- find /opt/opendj -name opendj-core.jar
2024-04-19 21:04:49,795 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-19 21:04:50,594 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:50,595 DEBUG --- stdout ---
2024-04-19 21:04:50,595 DEBUG /opt/opendj/lib/opendj-core.jar
2024-04-19 21:04:50,595 DEBUG --- stderr ---
2024-04-19 21:04:50,595 DEBUG
2024-04-19 21:04:50,595 INFO
2024-04-19 21:04:50,595 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-idrepo-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-idrepo_info/opendj-core.jar --container ds
2024-04-19 21:04:50,595 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:51,816 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:51,816 DEBUG --- stdout ---
2024-04-19 21:04:51,816 DEBUG tar: Removing leading `/' from member names
2024-04-19 21:04:51,816 DEBUG --- stderr ---
2024-04-19 21:04:51,816 DEBUG
2024-04-19 21:04:51,825 INFO
2024-04-19 21:04:51,825 INFO ****************************** Initializing component pods for AM ******************************
2024-04-19 21:04:51,825 INFO
2024-04-19 21:04:51,825 INFO ----------------------- Get AM software version -----------------------
2024-04-19 21:04:51,825 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version
2024-04-19 21:04:51,825 INFO [LoginSession] Get "token_id" token for user "amadmin"
2024-04-19 21:04:51,825 DEBUG [LoginSession] re-using "token_id" token that remains valid for 2m 42s
2024-04-19 21:04:51,826 INFO [http_cmd]: curl -L --request GET --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=_sSPLOQsczoaDxOkfOZPXNEn8cU.*AAJTSQACMDIAAlNLABxTWENiYnNoMlBRYTJFODBaeTBKVDFTN2dnbmc9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version"
2024-04-19 21:04:52,041 INFO [http_cmd]: http status code OK
2024-04-19 21:04:52,041 DEBUG --- status code ---
2024-04-19 21:04:52,042 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:52,042 DEBUG --- http response ---
2024-04-19 21:04:52,042 DEBUG {
"_id": "version",
"_rev": "-1298450700",
"version": "7.6.0-SNAPSHOT",
"fullVersion": "ForgeRock Access Management 7.6.0-SNAPSHOT Build b3f32bcf2825f595a381aae2f4ddab14a92da8c1 (2024-April-16 16:19)",
"revision": "b3f32bcf2825f595a381aae2f4ddab14a92da8c1",
"date": "2024-April-16 16:19"
}
2024-04-19 21:04:52,044 INFO
2024-04-19 21:04:52,045 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-19 21:04:52,045 INFO
2024-04-19 21:04:52,045 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-19 21:04:52,045 INFO
2024-04-19 21:04:52,045 INFO ---------------------- Get IDM software version ----------------------
2024-04-19 21:04:52,045 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/version
2024-04-19 21:04:52,045 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/version"
2024-04-19 21:04:52,255 INFO [http_cmd]: http status code OK
2024-04-19 21:04:52,255 DEBUG --- status code ---
2024-04-19 21:04:52,255 DEBUG http status code is 200 (expected 200)
2024-04-19 21:04:52,255 DEBUG --- http response ---
2024-04-19 21:04:52,255 DEBUG {
"_id": "version",
"productVersion": "7.6.0-SNAPSHOT",
"productBuildDate": "20240411202127",
"productRevision": "8554e0d1c5"
}
2024-04-19 21:04:52,257 INFO
2024-04-19 21:04:52,257 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-19 21:04:52,257 INFO
2024-04-19 21:04:52,257 INFO ------------------ Get END-USER-UI software version ------------------
2024-04-19 21:04:52,258 INFO
2024-04-19 21:04:52,258 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec end-user-ui-58878d7668-z8s6j --container end-user-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-19 21:04:52,258 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-19 21:04:52,987 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:52,989 DEBUG --- stdout ---
2024-04-19 21:04:52,989 DEBUG /usr/share/nginx/html/js/chunk-vendors.0380a414.js
2024-04-19 21:04:52,989 DEBUG --- stderr ---
2024-04-19 21:04:52,989 DEBUG
2024-04-19 21:04:52,990 INFO
2024-04-19 21:04:52,990 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp end-user-ui-58878d7668-z8s6j:/usr/share/nginx/html/js/chunk-vendors.0380a414.js /tmp/end-user-ui_info/chunk-vendors.0380a414.js --container end-user-ui
2024-04-19 21:04:52,990 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:54,119 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:54,119 DEBUG --- stdout ---
2024-04-19 21:04:54,119 DEBUG tar: Removing leading `/' from member names
2024-04-19 21:04:54,119 DEBUG --- stderr ---
2024-04-19 21:04:54,119 DEBUG
2024-04-19 21:04:54,153 INFO
2024-04-19 21:04:54,153 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-19 21:04:54,153 INFO
2024-04-19 21:04:54,153 INFO -------------------- Get LOGIN-UI software version --------------------
2024-04-19 21:04:54,154 INFO
2024-04-19 21:04:54,154 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec login-ui-5fdf49fc66-dlhjd --container login-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-19 21:04:54,154 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-19 21:04:54,876 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:54,877 DEBUG --- stdout ---
2024-04-19 21:04:54,877 DEBUG /usr/share/nginx/html/js/chunk-vendors.f2fa5365.js
2024-04-19 21:04:54,877 DEBUG --- stderr ---
2024-04-19 21:04:54,877 DEBUG
2024-04-19 21:04:54,877 INFO
2024-04-19 21:04:54,877 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp login-ui-5fdf49fc66-dlhjd:/usr/share/nginx/html/js/chunk-vendors.f2fa5365.js /tmp/login-ui_info/chunk-vendors.f2fa5365.js --container login-ui
2024-04-19 21:04:54,877 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:55,993 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:55,993 DEBUG --- stdout ---
2024-04-19 21:04:55,993 DEBUG tar: Removing leading `/' from member names
2024-04-19 21:04:55,994 DEBUG --- stderr ---
2024-04-19 21:04:55,994 DEBUG
2024-04-19 21:04:56,014 INFO
2024-04-19 21:04:56,014 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-19 21:04:56,015 INFO
2024-04-19 21:04:56,015 INFO -------------------- Get ADMIN-UI software version --------------------
2024-04-19 21:04:56,015 INFO
2024-04-19 21:04:56,015 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec admin-ui-c4c7bf584-tlgkc --container admin-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-19 21:04:56,015 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-19 21:04:56,751 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:56,752 DEBUG --- stdout ---
2024-04-19 21:04:56,752 DEBUG /usr/share/nginx/html/js/chunk-vendors.b3d879ac.js
2024-04-19 21:04:56,752 DEBUG --- stderr ---
2024-04-19 21:04:56,752 DEBUG
2024-04-19 21:04:56,752 INFO
2024-04-19 21:04:56,752 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp admin-ui-c4c7bf584-tlgkc:/usr/share/nginx/html/js/chunk-vendors.b3d879ac.js /tmp/admin-ui_info/chunk-vendors.b3d879ac.js --container admin-ui
2024-04-19 21:04:56,752 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:57,926 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:57,927 DEBUG --- stdout ---
2024-04-19 21:04:57,927 DEBUG tar: Removing leading `/' from member names
2024-04-19 21:04:57,927 DEBUG --- stderr ---
2024-04-19 21:04:57,927 DEBUG
2024-04-19 21:04:57,970 INFO
2024-04-19 21:04:57,970 INFO
2024-04-19 21:04:57,970 INFO ====================================================================================================
2024-04-19 21:04:57,970 INFO ================ Admin password for DS-CTS is: DiKi4hQF6vY2sxTY34zximKtumrSd1zS ================
2024-04-19 21:04:57,970 INFO ====================================================================================================
2024-04-19 21:04:57,970 INFO
2024-04-19 21:04:57,970 INFO ====================================================================================================
2024-04-19 21:04:57,970 INFO ============== Admin password for DS-IDREPO is: DiKi4hQF6vY2sxTY34zximKtumrSd1zS ==============
2024-04-19 21:04:57,970 INFO ====================================================================================================
2024-04-19 21:04:57,970 INFO
2024-04-19 21:04:57,970 INFO ====================================================================================================
2024-04-19 21:04:57,970 INFO ====================== Admin password for AM is: uIuf0Az3Zr7y6OxjLKb1zyfc ======================
2024-04-19 21:04:57,970 INFO ====================================================================================================
2024-04-19 21:04:57,970 INFO
2024-04-19 21:04:57,970 INFO ====================================================================================================
2024-04-19 21:04:57,970 INFO ===================== Admin password for IDM is: O9K1VXYVJiyimSmm4VQNxrQU =====================
2024-04-19 21:04:57,970 INFO ====================================================================================================
2024-04-19 21:04:57,970 INFO
2024-04-19 21:04:57,971 INFO *************************************** Dumping pod list ***************************************
2024-04-19 21:04:57,971 INFO Dumping pod list to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/_pod-list.txt
2024-04-19 21:04:58,468 INFO Dumping platform-config configmap to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/_platform-config_configmap.txt
2024-04-19 21:04:58,922 INFO
2024-04-19 21:04:58,922 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-19 21:04:58,922 INFO
2024-04-19 21:04:58,922 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:58,923 INFO
2024-04-19 21:04:58,923 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:58,923 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:04:59,289 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:59,289 DEBUG --- stdout ---
2024-04-19 21:04:59,289 DEBUG 3
2024-04-19 21:04:59,289 DEBUG --- stderr ---
2024-04-19 21:04:59,289 DEBUG
2024-04-19 21:04:59,289 INFO
2024-04-19 21:04:59,289 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:04:59,290 INFO
2024-04-19 21:04:59,290 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-cts --output jsonpath={.items[*].metadata.name}
2024-04-19 21:04:59,290 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:04:59,726 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:04:59,726 DEBUG --- stdout ---
2024-04-19 21:04:59,726 DEBUG ds-cts-0 ds-cts-1 ds-cts-2
2024-04-19 21:04:59,726 DEBUG --- stderr ---
2024-04-19 21:04:59,726 DEBUG
2024-04-19 21:04:59,726 INFO
2024-04-19 21:04:59,726 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-19 21:04:59,726 INFO
2024-04-19 21:04:59,726 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:04:59,727 INFO
2024-04-19 21:04:59,727 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:04:59,727 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:05:00,142 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:00,143 DEBUG --- stdout ---
2024-04-19 21:05:00,143 DEBUG 3
2024-04-19 21:05:00,143 DEBUG --- stderr ---
2024-04-19 21:05:00,143 DEBUG
2024-04-19 21:05:00,143 INFO
2024-04-19 21:05:00,143 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:05:00,143 INFO
2024-04-19 21:05:00,143 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-idrepo --output jsonpath={.items[*].metadata.name}
2024-04-19 21:05:00,143 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:05:00,637 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:00,637 DEBUG --- stdout ---
2024-04-19 21:05:00,637 DEBUG ds-idrepo-0 ds-idrepo-1 ds-idrepo-2
2024-04-19 21:05:00,638 DEBUG --- stderr ---
2024-04-19 21:05:00,638 DEBUG
2024-04-19 21:05:00,638 INFO
2024-04-19 21:05:00,638 INFO ****************************** Initializing component pods for AM ******************************
2024-04-19 21:05:00,638 INFO
2024-04-19 21:05:00,638 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:05:00,638 INFO
2024-04-19 21:05:00,638 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:05:00,639 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:05:01,075 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:01,075 DEBUG --- stdout ---
2024-04-19 21:05:01,076 DEBUG 3
2024-04-19 21:05:01,076 DEBUG --- stderr ---
2024-04-19 21:05:01,076 DEBUG
2024-04-19 21:05:01,076 INFO
2024-04-19 21:05:01,076 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:05:01,076 INFO
2024-04-19 21:05:01,076 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}
2024-04-19 21:05:01,076 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:05:01,524 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:01,524 DEBUG --- stdout ---
2024-04-19 21:05:01,524 DEBUG am-767f6b67b5-9zw22 am-767f6b67b5-qcqz2 am-767f6b67b5-vbltz
2024-04-19 21:05:01,524 DEBUG --- stderr ---
2024-04-19 21:05:01,524 DEBUG
2024-04-19 21:05:01,525 INFO
2024-04-19 21:05:01,525 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-19 21:05:01,525 INFO
2024-04-19 21:05:01,525 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:05:01,525 INFO
2024-04-19 21:05:01,525 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-19 21:05:01,525 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:05:01,882 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:01,882 DEBUG --- stdout ---
2024-04-19 21:05:01,882 DEBUG amster-grkh7
2024-04-19 21:05:01,882 DEBUG --- stderr ---
2024-04-19 21:05:01,882 DEBUG
2024-04-19 21:05:01,882 INFO
2024-04-19 21:05:01,882 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-19 21:05:01,882 INFO
2024-04-19 21:05:01,883 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:05:01,883 INFO
2024-04-19 21:05:01,883 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:05:01,883 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:05:02,244 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:02,244 DEBUG --- stdout ---
2024-04-19 21:05:02,244 DEBUG 2
2024-04-19 21:05:02,244 DEBUG --- stderr ---
2024-04-19 21:05:02,244 DEBUG
2024-04-19 21:05:02,244 INFO
2024-04-19 21:05:02,244 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:05:02,244 INFO
2024-04-19 21:05:02,244 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=idm --output jsonpath={.items[*].metadata.name}
2024-04-19 21:05:02,244 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:05:02,638 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:02,638 DEBUG --- stdout ---
2024-04-19 21:05:02,638 DEBUG idm-b85f9d978-txpwv idm-b85f9d978-wnhxb
2024-04-19 21:05:02,638 DEBUG --- stderr ---
2024-04-19 21:05:02,638 DEBUG
2024-04-19 21:05:02,638 INFO
2024-04-19 21:05:02,639 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-19 21:05:02,639 INFO
2024-04-19 21:05:02,639 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:05:02,639 INFO
2024-04-19 21:05:02,639 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:05:02,639 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:05:02,952 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:02,952 DEBUG --- stdout ---
2024-04-19 21:05:02,952 DEBUG 1
2024-04-19 21:05:02,952 DEBUG --- stderr ---
2024-04-19 21:05:02,952 DEBUG
2024-04-19 21:05:02,952 INFO
2024-04-19 21:05:02,952 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:05:02,953 INFO
2024-04-19 21:05:02,953 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].metadata.name}
2024-04-19 21:05:02,953 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:05:03,262 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:03,262 DEBUG --- stdout ---
2024-04-19 21:05:03,263 DEBUG end-user-ui-58878d7668-z8s6j
2024-04-19 21:05:03,263 DEBUG --- stderr ---
2024-04-19 21:05:03,263 DEBUG
2024-04-19 21:05:03,263 INFO
2024-04-19 21:05:03,263 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-19 21:05:03,263 INFO
2024-04-19 21:05:03,263 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:05:03,263 INFO
2024-04-19 21:05:03,263 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:05:03,263 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:05:03,562 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:03,564 DEBUG --- stdout ---
2024-04-19 21:05:03,564 DEBUG 1
2024-04-19 21:05:03,564 DEBUG --- stderr ---
2024-04-19 21:05:03,564 DEBUG
2024-04-19 21:05:03,564 INFO
2024-04-19 21:05:03,564 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:05:03,564 INFO
2024-04-19 21:05:03,564 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].metadata.name}
2024-04-19 21:05:03,564 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:05:03,863 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:03,864 DEBUG --- stdout ---
2024-04-19 21:05:03,864 DEBUG login-ui-5fdf49fc66-dlhjd
2024-04-19 21:05:03,864 DEBUG --- stderr ---
2024-04-19 21:05:03,864 DEBUG
2024-04-19 21:05:03,864 INFO
2024-04-19 21:05:03,864 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-19 21:05:03,864 INFO
2024-04-19 21:05:03,864 INFO --------------------- Get expected number of pods ---------------------
2024-04-19 21:05:03,864 INFO
2024-04-19 21:05:03,864 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-19 21:05:03,864 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-19 21:05:04,162 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:04,162 DEBUG --- stdout ---
2024-04-19 21:05:04,162 DEBUG 1
2024-04-19 21:05:04,162 DEBUG --- stderr ---
2024-04-19 21:05:04,162 DEBUG
2024-04-19 21:05:04,162 INFO
2024-04-19 21:05:04,162 INFO ---------------------------- Get pod list ----------------------------
2024-04-19 21:05:04,163 INFO
2024-04-19 21:05:04,163 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].metadata.name}
2024-04-19 21:05:04,163 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-19 21:05:04,479 INFO [loop_until]: OK (rc = 0)
2024-04-19 21:05:04,480 DEBUG --- stdout ---
2024-04-19 21:05:04,480 DEBUG admin-ui-c4c7bf584-tlgkc
2024-04-19 21:05:04,480 DEBUG --- stderr ---
2024-04-19 21:05:04,480 DEBUG
2024-04-19 21:05:04,480 INFO
2024-04-19 21:05:04,480 INFO *********************************** Dumping components logs ***********************************
2024-04-19 21:05:04,480 INFO
2024-04-19 21:05:04,480 INFO ----------------------- Dumping logs for DS-CTS -----------------------
2024-04-19 21:05:06,115 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-cts-0.txt
2024-04-19 21:05:06,115 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-cts-0.txt
2024-04-19 21:05:06,138 INFO Check pod logs for errors
2024-04-19 21:05:07,744 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-cts-1.txt
2024-04-19 21:05:07,744 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-cts-1.txt
2024-04-19 21:05:07,783 INFO Check pod logs for errors
2024-04-19 21:05:09,350 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-cts-2.txt
2024-04-19 21:05:09,352 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-cts-2.txt
2024-04-19 21:05:09,374 INFO Check pod logs for errors
2024-04-19 21:05:09,394 INFO
2024-04-19 21:05:09,394 INFO --------------------- Dumping logs for DS-IDREPO ---------------------
2024-04-19 21:05:10,954 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-idrepo-0.txt
2024-04-19 21:05:10,955 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-idrepo-0.txt
2024-04-19 21:05:10,984 INFO Check pod logs for errors
2024-04-19 21:05:12,566 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-idrepo-1.txt
2024-04-19 21:05:12,567 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-idrepo-1.txt
2024-04-19 21:05:12,581 INFO Check pod logs for errors
2024-04-19 21:05:14,114 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-idrepo-2.txt
2024-04-19 21:05:14,115 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/ds-idrepo-2.txt
2024-04-19 21:05:14,143 INFO Check pod logs for errors
2024-04-19 21:05:14,158 INFO
2024-04-19 21:05:14,159 INFO ------------------------- Dumping logs for AM -------------------------
2024-04-19 21:05:15,636 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/am-767f6b67b5-9zw22.txt
2024-04-19 21:05:15,636 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/am-767f6b67b5-9zw22.txt
2024-04-19 21:05:15,642 INFO Check pod logs for errors
2024-04-19 21:05:17,172 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/am-767f6b67b5-qcqz2.txt
2024-04-19 21:05:17,173 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/am-767f6b67b5-qcqz2.txt
2024-04-19 21:05:17,177 INFO Check pod logs for errors
2024-04-19 21:05:18,605 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/am-767f6b67b5-vbltz.txt
2024-04-19 21:05:18,606 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/am-767f6b67b5-vbltz.txt
2024-04-19 21:05:18,607 INFO Check pod logs for errors
2024-04-19 21:05:18,609 INFO
2024-04-19 21:05:18,609 INFO ----------------------- Dumping logs for AMSTER -----------------------
2024-04-19 21:05:19,676 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/amster-grkh7.txt
2024-04-19 21:05:19,677 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/amster-grkh7.txt
2024-04-19 21:05:19,677 INFO Check pod logs for errors
2024-04-19 21:05:19,677 INFO
2024-04-19 21:05:19,677 INFO ------------------------ Dumping logs for IDM ------------------------
2024-04-19 21:05:21,383 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/idm-b85f9d978-txpwv.txt
2024-04-19 21:05:21,384 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/idm-b85f9d978-txpwv.txt
2024-04-19 21:05:21,421 INFO Check pod logs for errors
2024-04-19 21:05:23,144 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/idm-b85f9d978-wnhxb.txt
2024-04-19 21:05:23,145 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/idm-b85f9d978-wnhxb.txt
2024-04-19 21:05:23,191 INFO Check pod logs for errors
2024-04-19 21:05:23,220 INFO
2024-04-19 21:05:23,220 INFO -------------------- Dumping logs for END-USER-UI --------------------
2024-04-19 21:05:24,567 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/end-user-ui-58878d7668-z8s6j.txt
2024-04-19 21:05:24,568 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/end-user-ui-58878d7668-z8s6j.txt
2024-04-19 21:05:24,579 INFO Check pod logs for errors
2024-04-19 21:05:24,591 INFO
2024-04-19 21:05:24,591 INFO ---------------------- Dumping logs for LOGIN-UI ----------------------
2024-04-19 21:05:25,897 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/login-ui-5fdf49fc66-dlhjd.txt
2024-04-19 21:05:25,898 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/login-ui-5fdf49fc66-dlhjd.txt
2024-04-19 21:05:25,907 INFO Check pod logs for errors
2024-04-19 21:05:25,919 INFO
2024-04-19 21:05:25,919 INFO ---------------------- Dumping logs for ADMIN-UI ----------------------
2024-04-19 21:05:27,272 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/admin-ui-c4c7bf584-tlgkc.txt
2024-04-19 21:05:27,273 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240419_210457-after-deployment/admin-ui-c4c7bf584-tlgkc.txt
2024-04-19 21:05:27,288 INFO Check pod logs for errors
[2024-04-19 21:05:27] - INFO: Deployment ready
________________________________________________________________________________
[2024-04-19 21:05:27] Discover_all_components post : Post method
________________________________________________________________________________
Setting result to PASS
Task has been successfully stopped