--Task--
name: Discover_all_components
enabled: True
class_name: DiscoverComponentsTask
source_name: controller
source_namespace: >default<
target_name: controller
target_namespace: >default<
start: 0.0
stop: None
timeout: no timeout
loop: False
interval: None
dependencies: []
wait_for: []
preceding_task: None
options: {}
group_name: None
Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock
________________________________________________________________________________
[2024-04-15 22:04:50] Discover_all_components pre : Initialising task parameters
________________________________________________________________________________
task will be executed on controller (localhost)
________________________________________________________________________________
[2024-04-15 22:04:50] Discover_all_components step1 : Discover components
________________________________________________________________________________
2024-04-15 22:04:50,440 INFO Deployment was skipped.
2024-04-15 22:04:50,779 INFO
2024-04-15 22:04:50,780 INFO ************************************* Creating deployment *************************************
2024-04-15 22:04:50,780 INFO Creating normal (forgeops) type deployment for deployment: stack
2024-04-15 22:04:50,796 INFO
2024-04-15 22:04:50,796 INFO ------- Custom component configuration present. Loading values -------
2024-04-15 22:04:50,796 INFO
2024-04-15 22:04:50,797 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker ds cts
2024-04-15 22:04:50,876 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:04:50,876 DEBUG --- stdout ---
2024-04-15 22:04:50,876 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/cts
2024-04-15 22:04:50,876 DEBUG --- stderr ---
2024-04-15 22:04:50,876 DEBUG
2024-04-15 22:04:50,876 INFO
2024-04-15 22:04:50,877 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/cts/Dockerfile: gcr.io/forgerock-io/ds/pit1:7.6.0-d49ad6a5879fab3530729d8773297783d042f903
2024-04-15 22:04:50,877 INFO
2024-04-15 22:04:50,877 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker ds idrepo
2024-04-15 22:04:50,956 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:04:50,956 DEBUG --- stdout ---
2024-04-15 22:04:50,956 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/idrepo
2024-04-15 22:04:50,956 DEBUG --- stderr ---
2024-04-15 22:04:50,956 DEBUG
2024-04-15 22:04:50,956 INFO
2024-04-15 22:04:50,956 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/ds/idrepo/Dockerfile: gcr.io/forgerock-io/ds/pit1:7.6.0-d49ad6a5879fab3530729d8773297783d042f903
2024-04-15 22:04:50,956 INFO
2024-04-15 22:04:50,957 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker am
2024-04-15 22:04:51,036 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:04:51,036 DEBUG --- stdout ---
2024-04-15 22:04:51,036 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/am
2024-04-15 22:04:51,036 DEBUG --- stderr ---
2024-04-15 22:04:51,036 DEBUG
2024-04-15 22:04:51,036 INFO
2024-04-15 22:04:51,037 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/am/Dockerfile: gcr.io/forgerock-io/am-cdk/pit1:7.6.0-bbf6791c961b39ef101ebad36686ee2c3cde94bb
2024-04-15 22:04:51,037 INFO
2024-04-15 22:04:51,037 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker amster
2024-04-15 22:04:51,114 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:04:51,114 DEBUG --- stdout ---
2024-04-15 22:04:51,114 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/amster
2024-04-15 22:04:51,114 DEBUG --- stderr ---
2024-04-15 22:04:51,114 DEBUG
2024-04-15 22:04:51,114 INFO
2024-04-15 22:04:51,114 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/amster/Dockerfile: gcr.io/forgerock-io/amster/pit1:7.6.0-bbf6791c961b39ef101ebad36686ee2c3cde94bb
2024-04-15 22:04:51,114 INFO
2024-04-15 22:04:51,115 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path docker idm
2024-04-15 22:04:51,189 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:04:51,190 DEBUG --- stdout ---
2024-04-15 22:04:51,190 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/idm
2024-04-15 22:04:51,190 DEBUG --- stderr ---
2024-04-15 22:04:51,190 DEBUG
2024-04-15 22:04:51,190 INFO
2024-04-15 22:04:51,190 INFO Read FROM line from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/docker/idm/Dockerfile: gcr.io/forgerock-io/idm-cdk/pit1:7.6.0-8554e0d1c5044f4f6c6504cd366196848874e695
2024-04-15 22:04:51,190 INFO
2024-04-15 22:04:51,191 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base end-user-ui
2024-04-15 22:04:51,271 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:04:51,272 DEBUG --- stdout ---
2024-04-15 22:04:51,272 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/end-user-ui
2024-04-15 22:04:51,272 DEBUG --- stderr ---
2024-04-15 22:04:51,272 DEBUG
2024-04-15 22:04:51,272 INFO
2024-04-15 22:04:51,275 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/end-user-ui/kustomization.yaml: gcr.io/forgerock-io/platform-enduser-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-15 22:04:51,275 INFO
2024-04-15 22:04:51,275 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base login-ui
2024-04-15 22:04:51,351 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:04:51,352 DEBUG --- stdout ---
2024-04-15 22:04:51,352 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/login-ui
2024-04-15 22:04:51,352 DEBUG --- stderr ---
2024-04-15 22:04:51,352 DEBUG
2024-04-15 22:04:51,352 INFO
2024-04-15 22:04:51,355 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/login-ui/kustomization.yaml: gcr.io/forgerock-io/platform-login-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-15 22:04:51,355 INFO
2024-04-15 22:04:51,355 INFO [run_command]: /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/bin/config path kustomize base admin-ui
2024-04-15 22:04:51,435 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:04:51,436 DEBUG --- stdout ---
2024-04-15 22:04:51,436 DEBUG /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/admin-ui
2024-04-15 22:04:51,436 DEBUG --- stderr ---
2024-04-15 22:04:51,436 DEBUG
2024-04-15 22:04:51,436 INFO
2024-04-15 22:04:51,439 INFO Read newTag info from /mnt/disks/data/xslou/lodestar-fork/ext/forgeops_stack/kustomize/base/admin-ui/kustomization.yaml: gcr.io/forgerock-io/platform-admin-ui/docker-build:7.6.0-SNAPSHOT-e28d3cbe3f9d1b58834dbaa041c4c5d0f2cae626
2024-04-15 22:04:51,439 INFO Loading external components from config.yaml
2024-04-15 22:04:51,559 INFO No external components defined.
2024-04-15 22:04:51,682 INFO Setting context for components to: gke_engineeringpit_us-east1-d_xlou-cdm
2024-04-15 22:04:51,682 INFO
2024-04-15 22:04:51,682 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-15 22:04:51,682 INFO
2024-04-15 22:04:51,682 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:04:51,682 INFO
2024-04-15 22:04:51,682 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:04:51,682 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:04:52,076 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:52,076 DEBUG --- stdout ---
2024-04-15 22:04:52,076 DEBUG 3
2024-04-15 22:04:52,076 DEBUG --- stderr ---
2024-04-15 22:04:52,076 DEBUG
2024-04-15 22:04:52,076 INFO
2024-04-15 22:04:52,076 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:04:52,076 INFO
2024-04-15 22:04:52,076 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-cts --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-15 22:04:52,077 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:04:52,514 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 22:04:52,514 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:52,514 DEBUG --- stdout ---
2024-04-15 22:04:52,515 DEBUG ds-cts-0 ds-cts-1 ds-cts-2
2024-04-15 22:04:52,515 DEBUG --- stderr ---
2024-04-15 22:04:52,515 DEBUG
2024-04-15 22:04:52,515 INFO
2024-04-15 22:04:52,515 INFO -------------------- Check pod ds-cts-0 is running --------------------
2024-04-15 22:04:52,515 INFO
2024-04-15 22:04:52,515 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-0 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:04:52,515 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:04:52,875 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:04:52,876 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:52,876 DEBUG --- stdout ---
2024-04-15 22:04:52,876 DEBUG Running
2024-04-15 22:04:52,876 DEBUG --- stderr ---
2024-04-15 22:04:52,876 DEBUG
2024-04-15 22:04:52,876 INFO
2024-04-15 22:04:52,876 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-0 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:04:52,876 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:04:53,242 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:04:53,243 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:53,243 DEBUG --- stdout ---
2024-04-15 22:04:53,243 DEBUG true
2024-04-15 22:04:53,243 DEBUG --- stderr ---
2024-04-15 22:04:53,243 DEBUG
2024-04-15 22:04:53,243 INFO
2024-04-15 22:04:53,243 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-0 --output jsonpath={.status.startTime}
2024-04-15 22:04:53,243 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:04:53,632 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:53,632 DEBUG --- stdout ---
2024-04-15 22:04:53,633 DEBUG 2024-04-13T22:38:33Z
2024-04-15 22:04:53,633 DEBUG --- stderr ---
2024-04-15 22:04:53,633 DEBUG
2024-04-15 22:04:53,633 INFO
2024-04-15 22:04:53,633 INFO ------------- Check pod ds-cts-0 filesystem is accessible -------------
2024-04-15 22:04:54,476 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:04:54,478 INFO
2024-04-15 22:04:54,479 INFO ------------------ Check pod ds-cts-0 restart count ------------------
2024-04-15 22:04:54,479 INFO
2024-04-15 22:04:54,479 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-0 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:04:54,479 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:04:54,843 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:54,843 DEBUG --- stdout ---
2024-04-15 22:04:54,843 DEBUG 0
2024-04-15 22:04:54,843 DEBUG --- stderr ---
2024-04-15 22:04:54,843 DEBUG
2024-04-15 22:04:54,843 INFO Pod ds-cts-0 has been restarted 0 times.
2024-04-15 22:04:54,844 INFO
2024-04-15 22:04:54,844 INFO -------------------- Check pod ds-cts-1 is running --------------------
2024-04-15 22:04:54,844 INFO
2024-04-15 22:04:54,844 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-1 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:04:54,844 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:04:55,211 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:04:55,211 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:55,211 DEBUG --- stdout ---
2024-04-15 22:04:55,211 DEBUG Running
2024-04-15 22:04:55,211 DEBUG --- stderr ---
2024-04-15 22:04:55,211 DEBUG
2024-04-15 22:04:55,212 INFO
2024-04-15 22:04:55,212 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-1 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:04:55,212 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:04:55,572 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:04:55,572 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:55,572 DEBUG --- stdout ---
2024-04-15 22:04:55,572 DEBUG true
2024-04-15 22:04:55,572 DEBUG --- stderr ---
2024-04-15 22:04:55,573 DEBUG
2024-04-15 22:04:55,573 INFO
2024-04-15 22:04:55,573 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-1 --output jsonpath={.status.startTime}
2024-04-15 22:04:55,573 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:04:55,937 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:55,937 DEBUG --- stdout ---
2024-04-15 22:04:55,937 DEBUG 2024-04-15T16:51:45Z
2024-04-15 22:04:55,937 DEBUG --- stderr ---
2024-04-15 22:04:55,937 DEBUG
2024-04-15 22:04:55,938 INFO
2024-04-15 22:04:55,938 INFO ------------- Check pod ds-cts-1 filesystem is accessible -------------
2024-04-15 22:04:56,762 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:04:56,762 INFO
2024-04-15 22:04:56,762 INFO ------------------ Check pod ds-cts-1 restart count ------------------
2024-04-15 22:04:56,762 INFO
2024-04-15 22:04:56,762 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-1 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:04:56,762 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:04:57,128 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:57,128 DEBUG --- stdout ---
2024-04-15 22:04:57,128 DEBUG 0
2024-04-15 22:04:57,128 DEBUG --- stderr ---
2024-04-15 22:04:57,128 DEBUG
2024-04-15 22:04:57,128 INFO Pod ds-cts-1 has been restarted 0 times.
2024-04-15 22:04:57,128 INFO
2024-04-15 22:04:57,128 INFO -------------------- Check pod ds-cts-2 is running --------------------
2024-04-15 22:04:57,129 INFO
2024-04-15 22:04:57,129 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:04:57,129 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:04:57,491 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:04:57,491 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:57,491 DEBUG --- stdout ---
2024-04-15 22:04:57,491 DEBUG Running
2024-04-15 22:04:57,491 DEBUG --- stderr ---
2024-04-15 22:04:57,491 DEBUG
2024-04-15 22:04:57,492 INFO
2024-04-15 22:04:57,492 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-cts-2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:04:57,492 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:04:57,852 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:04:57,852 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:57,852 DEBUG --- stdout ---
2024-04-15 22:04:57,852 DEBUG true
2024-04-15 22:04:57,852 DEBUG --- stderr ---
2024-04-15 22:04:57,852 DEBUG
2024-04-15 22:04:57,852 INFO
2024-04-15 22:04:57,852 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-2 --output jsonpath={.status.startTime}
2024-04-15 22:04:57,852 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:04:58,218 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:58,218 DEBUG --- stdout ---
2024-04-15 22:04:58,218 DEBUG 2024-04-15T16:52:10Z
2024-04-15 22:04:58,218 DEBUG --- stderr ---
2024-04-15 22:04:58,218 DEBUG
2024-04-15 22:04:58,219 INFO
2024-04-15 22:04:58,219 INFO ------------- Check pod ds-cts-2 filesystem is accessible -------------
2024-04-15 22:04:59,222 INFO [loop_until]: Function succeeded after 1s (rc=0) - expected pattern found
2024-04-15 22:04:59,223 INFO
2024-04-15 22:04:59,223 INFO ------------------ Check pod ds-cts-2 restart count ------------------
2024-04-15 22:04:59,223 INFO
2024-04-15 22:04:59,223 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:04:59,223 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:04:59,598 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:59,598 DEBUG --- stdout ---
2024-04-15 22:04:59,598 DEBUG 0
2024-04-15 22:04:59,598 DEBUG --- stderr ---
2024-04-15 22:04:59,598 DEBUG
2024-04-15 22:04:59,598 INFO Pod ds-cts-2 has been restarted 0 times.
2024-04-15 22:04:59,598 INFO
2024-04-15 22:04:59,598 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-15 22:04:59,598 INFO
2024-04-15 22:04:59,598 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:04:59,599 INFO
2024-04-15 22:04:59,599 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:04:59,599 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:04:59,971 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:04:59,972 DEBUG --- stdout ---
2024-04-15 22:04:59,972 DEBUG 3
2024-04-15 22:04:59,972 DEBUG --- stderr ---
2024-04-15 22:04:59,972 DEBUG
2024-04-15 22:04:59,972 INFO
2024-04-15 22:04:59,972 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:04:59,972 INFO
2024-04-15 22:04:59,972 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-idrepo --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-15 22:04:59,972 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:05:00,409 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 22:05:00,409 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:00,409 DEBUG --- stdout ---
2024-04-15 22:05:00,409 DEBUG ds-idrepo-0 ds-idrepo-1 ds-idrepo-2
2024-04-15 22:05:00,410 DEBUG --- stderr ---
2024-04-15 22:05:00,410 DEBUG
2024-04-15 22:05:00,410 INFO
2024-04-15 22:05:00,410 INFO ------------------ Check pod ds-idrepo-0 is running ------------------
2024-04-15 22:05:00,410 INFO
2024-04-15 22:05:00,410 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-0 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:00,410 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:00,771 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:00,771 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:00,771 DEBUG --- stdout ---
2024-04-15 22:05:00,771 DEBUG Running
2024-04-15 22:05:00,771 DEBUG --- stderr ---
2024-04-15 22:05:00,771 DEBUG
2024-04-15 22:05:00,772 INFO
2024-04-15 22:05:00,772 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-0 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:00,772 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:01,135 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:01,135 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:01,135 DEBUG --- stdout ---
2024-04-15 22:05:01,135 DEBUG true
2024-04-15 22:05:01,135 DEBUG --- stderr ---
2024-04-15 22:05:01,135 DEBUG
2024-04-15 22:05:01,136 INFO
2024-04-15 22:05:01,136 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-0 --output jsonpath={.status.startTime}
2024-04-15 22:05:01,136 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:01,507 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:01,507 DEBUG --- stdout ---
2024-04-15 22:05:01,507 DEBUG 2024-04-13T22:38:34Z
2024-04-15 22:05:01,507 DEBUG --- stderr ---
2024-04-15 22:05:01,507 DEBUG
2024-04-15 22:05:01,508 INFO
2024-04-15 22:05:01,508 INFO ----------- Check pod ds-idrepo-0 filesystem is accessible -----------
2024-04-15 22:05:02,359 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:02,360 INFO
2024-04-15 22:05:02,360 INFO ----------------- Check pod ds-idrepo-0 restart count -----------------
2024-04-15 22:05:02,360 INFO
2024-04-15 22:05:02,360 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-0 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:02,360 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:02,724 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:02,725 DEBUG --- stdout ---
2024-04-15 22:05:02,725 DEBUG 0
2024-04-15 22:05:02,725 DEBUG --- stderr ---
2024-04-15 22:05:02,725 DEBUG
2024-04-15 22:05:02,725 INFO Pod ds-idrepo-0 has been restarted 0 times.
2024-04-15 22:05:02,725 INFO
2024-04-15 22:05:02,725 INFO ------------------ Check pod ds-idrepo-1 is running ------------------
2024-04-15 22:05:02,725 INFO
2024-04-15 22:05:02,725 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-1 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:02,725 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:03,092 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:03,092 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:03,092 DEBUG --- stdout ---
2024-04-15 22:05:03,092 DEBUG Running
2024-04-15 22:05:03,092 DEBUG --- stderr ---
2024-04-15 22:05:03,092 DEBUG
2024-04-15 22:05:03,092 INFO
2024-04-15 22:05:03,092 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-1 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:03,092 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:03,455 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:03,455 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:03,455 DEBUG --- stdout ---
2024-04-15 22:05:03,455 DEBUG true
2024-04-15 22:05:03,455 DEBUG --- stderr ---
2024-04-15 22:05:03,455 DEBUG
2024-04-15 22:05:03,456 INFO
2024-04-15 22:05:03,456 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-1 --output jsonpath={.status.startTime}
2024-04-15 22:05:03,456 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:03,845 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:03,845 DEBUG --- stdout ---
2024-04-15 22:05:03,845 DEBUG 2024-04-15T20:48:06Z
2024-04-15 22:05:03,845 DEBUG --- stderr ---
2024-04-15 22:05:03,845 DEBUG
2024-04-15 22:05:03,846 INFO
2024-04-15 22:05:03,846 INFO ----------- Check pod ds-idrepo-1 filesystem is accessible -----------
2024-04-15 22:05:04,683 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:04,684 INFO
2024-04-15 22:05:04,684 INFO ----------------- Check pod ds-idrepo-1 restart count -----------------
2024-04-15 22:05:04,685 INFO
2024-04-15 22:05:04,685 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-1 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:04,685 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:05,088 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:05,088 DEBUG --- stdout ---
2024-04-15 22:05:05,088 DEBUG 0
2024-04-15 22:05:05,088 DEBUG --- stderr ---
2024-04-15 22:05:05,088 DEBUG
2024-04-15 22:05:05,089 INFO Pod ds-idrepo-1 has been restarted 0 times.
2024-04-15 22:05:05,089 INFO
2024-04-15 22:05:05,089 INFO ------------------ Check pod ds-idrepo-2 is running ------------------
2024-04-15 22:05:05,089 INFO
2024-04-15 22:05:05,089 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:05,089 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:05,496 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:05,496 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:05,496 DEBUG --- stdout ---
2024-04-15 22:05:05,496 DEBUG Running
2024-04-15 22:05:05,496 DEBUG --- stderr ---
2024-04-15 22:05:05,496 DEBUG
2024-04-15 22:05:05,497 INFO
2024-04-15 22:05:05,497 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods ds-idrepo-2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:05,497 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:05,877 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:05,877 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:05,878 DEBUG --- stdout ---
2024-04-15 22:05:05,878 DEBUG true
2024-04-15 22:05:05,878 DEBUG --- stderr ---
2024-04-15 22:05:05,878 DEBUG
2024-04-15 22:05:05,878 INFO
2024-04-15 22:05:05,878 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-2 --output jsonpath={.status.startTime}
2024-04-15 22:05:05,878 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:06,269 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:06,269 DEBUG --- stdout ---
2024-04-15 22:05:06,270 DEBUG 2024-04-15T20:48:42Z
2024-04-15 22:05:06,270 DEBUG --- stderr ---
2024-04-15 22:05:06,270 DEBUG
2024-04-15 22:05:06,270 INFO
2024-04-15 22:05:06,270 INFO ----------- Check pod ds-idrepo-2 filesystem is accessible -----------
2024-04-15 22:05:07,132 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:07,132 INFO
2024-04-15 22:05:07,132 INFO ----------------- Check pod ds-idrepo-2 restart count -----------------
2024-04-15 22:05:07,132 INFO
2024-04-15 22:05:07,132 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:07,132 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:07,515 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:07,516 DEBUG --- stdout ---
2024-04-15 22:05:07,516 DEBUG 0
2024-04-15 22:05:07,516 DEBUG --- stderr ---
2024-04-15 22:05:07,516 DEBUG
2024-04-15 22:05:07,516 INFO Pod ds-idrepo-2 has been restarted 0 times.
2024-04-15 22:05:07,516 INFO
2024-04-15 22:05:07,516 INFO ****************************** Initializing component pods for AM ******************************
2024-04-15 22:05:07,516 INFO
2024-04-15 22:05:07,516 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:07,516 INFO
2024-04-15 22:05:07,516 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:07,516 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:07,908 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:07,908 DEBUG --- stdout ---
2024-04-15 22:05:07,908 DEBUG 3
2024-04-15 22:05:07,908 DEBUG --- stderr ---
2024-04-15 22:05:07,908 DEBUG
2024-04-15 22:05:07,908 INFO
2024-04-15 22:05:07,908 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:05:07,909 INFO
2024-04-15 22:05:07,909 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}` | grep 3
2024-04-15 22:05:07,909 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:05:08,354 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 22:05:08,354 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:08,354 DEBUG --- stdout ---
2024-04-15 22:05:08,354 DEBUG am-7d597cf7d6-8ctft am-7d597cf7d6-8vr8f am-7d597cf7d6-9wz9h
2024-04-15 22:05:08,354 DEBUG --- stderr ---
2024-04-15 22:05:08,354 DEBUG
2024-04-15 22:05:08,354 INFO
2024-04-15 22:05:08,354 INFO -------------- Check pod am-7d597cf7d6-8ctft is running --------------
2024-04-15 22:05:08,354 INFO
2024-04-15 22:05:08,354 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-8ctft -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:08,355 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:08,716 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:08,716 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:08,716 DEBUG --- stdout ---
2024-04-15 22:05:08,716 DEBUG Running
2024-04-15 22:05:08,716 DEBUG --- stderr ---
2024-04-15 22:05:08,716 DEBUG
2024-04-15 22:05:08,717 INFO
2024-04-15 22:05:08,717 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-8ctft -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:08,717 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:09,107 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:09,107 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:09,107 DEBUG --- stdout ---
2024-04-15 22:05:09,107 DEBUG true
2024-04-15 22:05:09,107 DEBUG --- stderr ---
2024-04-15 22:05:09,107 DEBUG
2024-04-15 22:05:09,107 INFO
2024-04-15 22:05:09,107 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8ctft --output jsonpath={.status.startTime}
2024-04-15 22:05:09,107 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:09,497 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:09,497 DEBUG --- stdout ---
2024-04-15 22:05:09,498 DEBUG 2024-04-13T22:40:34Z
2024-04-15 22:05:09,498 DEBUG --- stderr ---
2024-04-15 22:05:09,498 DEBUG
2024-04-15 22:05:09,499 INFO
2024-04-15 22:05:09,499 INFO ------- Check pod am-7d597cf7d6-8ctft filesystem is accessible -------
2024-04-15 22:05:10,327 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:10,328 INFO
2024-04-15 22:05:10,328 INFO ------------- Check pod am-7d597cf7d6-8ctft restart count -------------
2024-04-15 22:05:10,328 INFO
2024-04-15 22:05:10,328 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8ctft --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:10,328 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:10,699 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:10,699 DEBUG --- stdout ---
2024-04-15 22:05:10,700 DEBUG 0
2024-04-15 22:05:10,700 DEBUG --- stderr ---
2024-04-15 22:05:10,700 DEBUG
2024-04-15 22:05:10,700 INFO Pod am-7d597cf7d6-8ctft has been restarted 0 times.
2024-04-15 22:05:10,700 INFO
2024-04-15 22:05:10,700 INFO -------------- Check pod am-7d597cf7d6-8vr8f is running --------------
2024-04-15 22:05:10,700 INFO
2024-04-15 22:05:10,700 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-8vr8f -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:10,700 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:11,058 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:11,059 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:11,059 DEBUG --- stdout ---
2024-04-15 22:05:11,059 DEBUG Running
2024-04-15 22:05:11,059 DEBUG --- stderr ---
2024-04-15 22:05:11,059 DEBUG
2024-04-15 22:05:11,059 INFO
2024-04-15 22:05:11,059 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-8vr8f -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:11,059 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:11,423 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:11,423 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:11,423 DEBUG --- stdout ---
2024-04-15 22:05:11,423 DEBUG true
2024-04-15 22:05:11,423 DEBUG --- stderr ---
2024-04-15 22:05:11,423 DEBUG
2024-04-15 22:05:11,424 INFO
2024-04-15 22:05:11,424 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8vr8f --output jsonpath={.status.startTime}
2024-04-15 22:05:11,424 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:11,786 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:11,786 DEBUG --- stdout ---
2024-04-15 22:05:11,786 DEBUG 2024-04-13T22:40:34Z
2024-04-15 22:05:11,786 DEBUG --- stderr ---
2024-04-15 22:05:11,786 DEBUG
2024-04-15 22:05:11,787 INFO
2024-04-15 22:05:11,787 INFO ------- Check pod am-7d597cf7d6-8vr8f filesystem is accessible -------
2024-04-15 22:05:12,607 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:12,607 INFO
2024-04-15 22:05:12,607 INFO ------------- Check pod am-7d597cf7d6-8vr8f restart count -------------
2024-04-15 22:05:12,607 INFO
2024-04-15 22:05:12,607 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8vr8f --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:12,607 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:12,968 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:12,969 DEBUG --- stdout ---
2024-04-15 22:05:12,969 DEBUG 0
2024-04-15 22:05:12,969 DEBUG --- stderr ---
2024-04-15 22:05:12,969 DEBUG
2024-04-15 22:05:12,969 INFO Pod am-7d597cf7d6-8vr8f has been restarted 0 times.
2024-04-15 22:05:12,969 INFO
2024-04-15 22:05:12,969 INFO -------------- Check pod am-7d597cf7d6-9wz9h is running --------------
2024-04-15 22:05:12,969 INFO
2024-04-15 22:05:12,969 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-9wz9h -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:12,969 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:13,333 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:13,333 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:13,334 DEBUG --- stdout ---
2024-04-15 22:05:13,334 DEBUG Running
2024-04-15 22:05:13,334 DEBUG --- stderr ---
2024-04-15 22:05:13,334 DEBUG
2024-04-15 22:05:13,334 INFO
2024-04-15 22:05:13,334 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods am-7d597cf7d6-9wz9h -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:13,334 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:13,700 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:13,700 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:13,700 DEBUG --- stdout ---
2024-04-15 22:05:13,700 DEBUG true
2024-04-15 22:05:13,700 DEBUG --- stderr ---
2024-04-15 22:05:13,700 DEBUG
2024-04-15 22:05:13,701 INFO
2024-04-15 22:05:13,701 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-9wz9h --output jsonpath={.status.startTime}
2024-04-15 22:05:13,701 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:14,092 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:14,092 DEBUG --- stdout ---
2024-04-15 22:05:14,092 DEBUG 2024-04-13T22:40:34Z
2024-04-15 22:05:14,092 DEBUG --- stderr ---
2024-04-15 22:05:14,093 DEBUG
2024-04-15 22:05:14,093 INFO
2024-04-15 22:05:14,093 INFO ------- Check pod am-7d597cf7d6-9wz9h filesystem is accessible -------
2024-04-15 22:05:14,944 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:14,944 INFO
2024-04-15 22:05:14,944 INFO ------------- Check pod am-7d597cf7d6-9wz9h restart count -------------
2024-04-15 22:05:14,945 INFO
2024-04-15 22:05:14,945 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-9wz9h --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:14,945 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:15,337 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:15,337 DEBUG --- stdout ---
2024-04-15 22:05:15,338 DEBUG 0
2024-04-15 22:05:15,338 DEBUG --- stderr ---
2024-04-15 22:05:15,338 DEBUG
2024-04-15 22:05:15,338 INFO Pod am-7d597cf7d6-9wz9h has been restarted 0 times.
2024-04-15 22:05:15,338 INFO
2024-04-15 22:05:15,338 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-15 22:05:15,338 INFO
2024-04-15 22:05:15,338 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:05:15,338 INFO
2024-04-15 22:05:15,338 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm --field-selector status.phase!=Failed get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-15 22:05:15,338 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:05:15,731 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:15,733 DEBUG --- stdout ---
2024-04-15 22:05:15,733 DEBUG amster-qv96p
2024-04-15 22:05:15,733 DEBUG --- stderr ---
2024-04-15 22:05:15,733 DEBUG
2024-04-15 22:05:15,734 INFO
2024-04-15 22:05:15,734 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-15 22:05:15,734 INFO
2024-04-15 22:05:15,734 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:15,734 INFO
2024-04-15 22:05:15,734 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:15,734 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:16,132 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:16,132 DEBUG --- stdout ---
2024-04-15 22:05:16,132 DEBUG 2
2024-04-15 22:05:16,132 DEBUG --- stderr ---
2024-04-15 22:05:16,132 DEBUG
2024-04-15 22:05:16,133 INFO
2024-04-15 22:05:16,133 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:05:16,133 INFO
2024-04-15 22:05:16,133 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=idm --output jsonpath={.items[*].metadata.name}` | grep 2
2024-04-15 22:05:16,133 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:05:16,533 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 22:05:16,533 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:16,533 DEBUG --- stdout ---
2024-04-15 22:05:16,534 DEBUG idm-b85f9d978-j5pbm idm-b85f9d978-rl76f
2024-04-15 22:05:16,534 DEBUG --- stderr ---
2024-04-15 22:05:16,534 DEBUG
2024-04-15 22:05:16,534 INFO
2024-04-15 22:05:16,534 INFO -------------- Check pod idm-b85f9d978-j5pbm is running --------------
2024-04-15 22:05:16,534 INFO
2024-04-15 22:05:16,534 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-j5pbm -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:16,534 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:16,914 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:16,914 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:16,914 DEBUG --- stdout ---
2024-04-15 22:05:16,914 DEBUG Running
2024-04-15 22:05:16,914 DEBUG --- stderr ---
2024-04-15 22:05:16,915 DEBUG
2024-04-15 22:05:16,915 INFO
2024-04-15 22:05:16,915 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-j5pbm -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:16,915 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:17,301 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:17,301 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:17,301 DEBUG --- stdout ---
2024-04-15 22:05:17,301 DEBUG true
2024-04-15 22:05:17,301 DEBUG --- stderr ---
2024-04-15 22:05:17,301 DEBUG
2024-04-15 22:05:17,302 INFO
2024-04-15 22:05:17,302 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-j5pbm --output jsonpath={.status.startTime}
2024-04-15 22:05:17,302 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:17,678 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:17,679 DEBUG --- stdout ---
2024-04-15 22:05:17,679 DEBUG 2024-04-13T22:40:34Z
2024-04-15 22:05:17,679 DEBUG --- stderr ---
2024-04-15 22:05:17,679 DEBUG
2024-04-15 22:05:17,679 INFO
2024-04-15 22:05:17,679 INFO ------- Check pod idm-b85f9d978-j5pbm filesystem is accessible -------
2024-04-15 22:05:18,527 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:18,527 INFO
2024-04-15 22:05:18,528 INFO ------------- Check pod idm-b85f9d978-j5pbm restart count -------------
2024-04-15 22:05:18,528 INFO
2024-04-15 22:05:18,528 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-j5pbm --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:18,528 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:18,913 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:18,913 DEBUG --- stdout ---
2024-04-15 22:05:18,913 DEBUG 0
2024-04-15 22:05:18,913 DEBUG --- stderr ---
2024-04-15 22:05:18,913 DEBUG
2024-04-15 22:05:18,913 INFO Pod idm-b85f9d978-j5pbm has been restarted 0 times.
2024-04-15 22:05:18,913 INFO
2024-04-15 22:05:18,913 INFO -------------- Check pod idm-b85f9d978-rl76f is running --------------
2024-04-15 22:05:18,914 INFO
2024-04-15 22:05:18,914 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-rl76f -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:18,914 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:19,303 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:19,303 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:19,303 DEBUG --- stdout ---
2024-04-15 22:05:19,303 DEBUG Running
2024-04-15 22:05:19,303 DEBUG --- stderr ---
2024-04-15 22:05:19,303 DEBUG
2024-04-15 22:05:19,303 INFO
2024-04-15 22:05:19,303 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods idm-b85f9d978-rl76f -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:19,304 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:19,690 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:19,690 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:19,690 DEBUG --- stdout ---
2024-04-15 22:05:19,690 DEBUG true
2024-04-15 22:05:19,690 DEBUG --- stderr ---
2024-04-15 22:05:19,690 DEBUG
2024-04-15 22:05:19,691 INFO
2024-04-15 22:05:19,691 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-rl76f --output jsonpath={.status.startTime}
2024-04-15 22:05:19,691 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:20,090 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:20,090 DEBUG --- stdout ---
2024-04-15 22:05:20,090 DEBUG 2024-04-13T22:40:34Z
2024-04-15 22:05:20,090 DEBUG --- stderr ---
2024-04-15 22:05:20,090 DEBUG
2024-04-15 22:05:20,091 INFO
2024-04-15 22:05:20,091 INFO ------- Check pod idm-b85f9d978-rl76f filesystem is accessible -------
2024-04-15 22:05:20,939 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:20,939 INFO
2024-04-15 22:05:20,940 INFO ------------- Check pod idm-b85f9d978-rl76f restart count -------------
2024-04-15 22:05:20,941 INFO
2024-04-15 22:05:20,941 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod idm-b85f9d978-rl76f --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:20,941 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:21,327 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:21,328 DEBUG --- stdout ---
2024-04-15 22:05:21,328 DEBUG 0
2024-04-15 22:05:21,328 DEBUG --- stderr ---
2024-04-15 22:05:21,328 DEBUG
2024-04-15 22:05:21,328 INFO Pod idm-b85f9d978-rl76f has been restarted 0 times.
2024-04-15 22:05:21,328 INFO
2024-04-15 22:05:21,328 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-15 22:05:21,328 INFO
2024-04-15 22:05:21,328 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:21,328 INFO
2024-04-15 22:05:21,328 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:21,328 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:21,663 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:21,664 DEBUG --- stdout ---
2024-04-15 22:05:21,664 DEBUG 1
2024-04-15 22:05:21,664 DEBUG --- stderr ---
2024-04-15 22:05:21,664 DEBUG
2024-04-15 22:05:21,664 INFO
2024-04-15 22:05:21,664 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:05:21,664 INFO
2024-04-15 22:05:21,664 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-15 22:05:21,664 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:05:22,026 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 22:05:22,026 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:22,026 DEBUG --- stdout ---
2024-04-15 22:05:22,026 DEBUG end-user-ui-58878d7668-zg4h2
2024-04-15 22:05:22,027 DEBUG --- stderr ---
2024-04-15 22:05:22,027 DEBUG
2024-04-15 22:05:22,027 INFO
2024-04-15 22:05:22,027 INFO ---------- Check pod end-user-ui-58878d7668-zg4h2 is running ----------
2024-04-15 22:05:22,027 INFO
2024-04-15 22:05:22,027 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods end-user-ui-58878d7668-zg4h2 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:22,027 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:22,376 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:22,376 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:22,376 DEBUG --- stdout ---
2024-04-15 22:05:22,377 DEBUG Running
2024-04-15 22:05:22,377 DEBUG --- stderr ---
2024-04-15 22:05:22,377 DEBUG
2024-04-15 22:05:22,377 INFO
2024-04-15 22:05:22,377 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods end-user-ui-58878d7668-zg4h2 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:22,377 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:22,676 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:22,676 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:22,676 DEBUG --- stdout ---
2024-04-15 22:05:22,676 DEBUG true
2024-04-15 22:05:22,676 DEBUG --- stderr ---
2024-04-15 22:05:22,676 DEBUG
2024-04-15 22:05:22,677 INFO
2024-04-15 22:05:22,677 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod end-user-ui-58878d7668-zg4h2 --output jsonpath={.status.startTime}
2024-04-15 22:05:22,677 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:22,982 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:22,982 DEBUG --- stdout ---
2024-04-15 22:05:22,983 DEBUG 2024-04-13T22:41:43Z
2024-04-15 22:05:22,983 DEBUG --- stderr ---
2024-04-15 22:05:22,983 DEBUG
2024-04-15 22:05:22,983 INFO
2024-04-15 22:05:22,984 INFO --- Check pod end-user-ui-58878d7668-zg4h2 filesystem is accessible ---
2024-04-15 22:05:23,777 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:23,777 INFO
2024-04-15 22:05:23,777 INFO -------- Check pod end-user-ui-58878d7668-zg4h2 restart count --------
2024-04-15 22:05:23,777 INFO
2024-04-15 22:05:23,777 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod end-user-ui-58878d7668-zg4h2 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:23,777 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:24,096 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:24,096 DEBUG --- stdout ---
2024-04-15 22:05:24,096 DEBUG 0
2024-04-15 22:05:24,096 DEBUG --- stderr ---
2024-04-15 22:05:24,096 DEBUG
2024-04-15 22:05:24,096 INFO Pod end-user-ui-58878d7668-zg4h2 has been restarted 0 times.
2024-04-15 22:05:24,097 INFO
2024-04-15 22:05:24,097 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-15 22:05:24,097 INFO
2024-04-15 22:05:24,097 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:24,097 INFO
2024-04-15 22:05:24,097 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:24,097 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:24,406 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:24,406 DEBUG --- stdout ---
2024-04-15 22:05:24,406 DEBUG 1
2024-04-15 22:05:24,406 DEBUG --- stderr ---
2024-04-15 22:05:24,406 DEBUG
2024-04-15 22:05:24,406 INFO
2024-04-15 22:05:24,406 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:05:24,407 INFO
2024-04-15 22:05:24,407 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-15 22:05:24,407 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:05:24,718 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 22:05:24,718 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:24,718 DEBUG --- stdout ---
2024-04-15 22:05:24,718 DEBUG login-ui-5fdf49fc66-2wnx7
2024-04-15 22:05:24,718 DEBUG --- stderr ---
2024-04-15 22:05:24,718 DEBUG
2024-04-15 22:05:24,719 INFO
2024-04-15 22:05:24,719 INFO ----------- Check pod login-ui-5fdf49fc66-2wnx7 is running -----------
2024-04-15 22:05:24,719 INFO
2024-04-15 22:05:24,719 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods login-ui-5fdf49fc66-2wnx7 -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:24,719 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:25,048 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:25,048 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:25,048 DEBUG --- stdout ---
2024-04-15 22:05:25,048 DEBUG Running
2024-04-15 22:05:25,049 DEBUG --- stderr ---
2024-04-15 22:05:25,049 DEBUG
2024-04-15 22:05:25,049 INFO
2024-04-15 22:05:25,049 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods login-ui-5fdf49fc66-2wnx7 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:25,049 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:25,376 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:25,376 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:25,376 DEBUG --- stdout ---
2024-04-15 22:05:25,376 DEBUG true
2024-04-15 22:05:25,376 DEBUG --- stderr ---
2024-04-15 22:05:25,376 DEBUG
2024-04-15 22:05:25,377 INFO
2024-04-15 22:05:25,377 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod login-ui-5fdf49fc66-2wnx7 --output jsonpath={.status.startTime}
2024-04-15 22:05:25,377 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:25,698 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:25,699 DEBUG --- stdout ---
2024-04-15 22:05:25,699 DEBUG 2024-04-13T22:41:43Z
2024-04-15 22:05:25,699 DEBUG --- stderr ---
2024-04-15 22:05:25,699 DEBUG
2024-04-15 22:05:25,700 INFO
2024-04-15 22:05:25,700 INFO ---- Check pod login-ui-5fdf49fc66-2wnx7 filesystem is accessible ----
2024-04-15 22:05:26,485 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:26,490 INFO
2024-04-15 22:05:26,490 INFO ---------- Check pod login-ui-5fdf49fc66-2wnx7 restart count ----------
2024-04-15 22:05:26,491 INFO
2024-04-15 22:05:26,491 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod login-ui-5fdf49fc66-2wnx7 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:26,491 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:26,827 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:26,827 DEBUG --- stdout ---
2024-04-15 22:05:26,827 DEBUG 0
2024-04-15 22:05:26,827 DEBUG --- stderr ---
2024-04-15 22:05:26,828 DEBUG
2024-04-15 22:05:26,828 INFO Pod login-ui-5fdf49fc66-2wnx7 has been restarted 0 times.
2024-04-15 22:05:26,828 INFO
2024-04-15 22:05:26,828 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-15 22:05:26,828 INFO
2024-04-15 22:05:26,828 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:26,828 INFO
2024-04-15 22:05:26,828 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:26,828 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:27,154 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:27,154 DEBUG --- stdout ---
2024-04-15 22:05:27,154 DEBUG 1
2024-04-15 22:05:27,154 DEBUG --- stderr ---
2024-04-15 22:05:27,154 DEBUG
2024-04-15 22:05:27,154 INFO
2024-04-15 22:05:27,155 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:05:27,155 INFO
2024-04-15 22:05:27,155 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-15 22:05:27,155 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:05:27,484 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-15 22:05:27,484 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:27,484 DEBUG --- stdout ---
2024-04-15 22:05:27,484 DEBUG admin-ui-c4c7bf584-mbzlq
2024-04-15 22:05:27,484 DEBUG --- stderr ---
2024-04-15 22:05:27,484 DEBUG
2024-04-15 22:05:27,484 INFO
2024-04-15 22:05:27,484 INFO ------------ Check pod admin-ui-c4c7bf584-mbzlq is running ------------
2024-04-15 22:05:27,485 INFO
2024-04-15 22:05:27,485 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods admin-ui-c4c7bf584-mbzlq -o=jsonpath={.status.phase} | grep "Running"
2024-04-15 22:05:27,485 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:27,834 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:27,835 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:27,835 DEBUG --- stdout ---
2024-04-15 22:05:27,835 DEBUG Running
2024-04-15 22:05:27,835 DEBUG --- stderr ---
2024-04-15 22:05:27,835 DEBUG
2024-04-15 22:05:27,835 INFO
2024-04-15 22:05:27,835 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods admin-ui-c4c7bf584-mbzlq -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-15 22:05:27,835 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-15 22:05:28,153 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:28,154 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:28,154 DEBUG --- stdout ---
2024-04-15 22:05:28,154 DEBUG true
2024-04-15 22:05:28,154 DEBUG --- stderr ---
2024-04-15 22:05:28,154 DEBUG
2024-04-15 22:05:28,154 INFO
2024-04-15 22:05:28,155 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod admin-ui-c4c7bf584-mbzlq --output jsonpath={.status.startTime}
2024-04-15 22:05:28,155 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:28,460 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:28,460 DEBUG --- stdout ---
2024-04-15 22:05:28,460 DEBUG 2024-04-13T22:41:42Z
2024-04-15 22:05:28,460 DEBUG --- stderr ---
2024-04-15 22:05:28,460 DEBUG
2024-04-15 22:05:28,461 INFO
2024-04-15 22:05:28,461 INFO ----- Check pod admin-ui-c4c7bf584-mbzlq filesystem is accessible -----
2024-04-15 22:05:29,278 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:29,278 INFO
2024-04-15 22:05:29,278 INFO ---------- Check pod admin-ui-c4c7bf584-mbzlq restart count ----------
2024-04-15 22:05:29,279 INFO
2024-04-15 22:05:29,279 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod admin-ui-c4c7bf584-mbzlq --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-15 22:05:29,279 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:29,598 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:29,599 DEBUG --- stdout ---
2024-04-15 22:05:29,599 DEBUG 0
2024-04-15 22:05:29,599 DEBUG --- stderr ---
2024-04-15 22:05:29,599 DEBUG
2024-04-15 22:05:29,599 INFO Pod admin-ui-c4c7bf584-mbzlq has been restarted 0 times.
2024-04-15 22:05:29,599 INFO
2024-04-15 22:05:29,599 INFO ***************************** Checking DS-CTS component is running *****************************
2024-04-15 22:05:29,599 INFO
2024-04-15 22:05:29,599 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:29,599 INFO
2024-04-15 22:05:29,599 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:29,599 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:30,012 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:30,013 DEBUG --- stdout ---
2024-04-15 22:05:30,013 DEBUG 3
2024-04-15 22:05:30,013 DEBUG --- stderr ---
2024-04-15 22:05:30,013 DEBUG
2024-04-15 22:05:30,013 INFO
2024-04-15 22:05:30,013 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-15 22:05:30,014 INFO
2024-04-15 22:05:30,014 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-cts --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3"
2024-04-15 22:05:30,014 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 22:05:30,427 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:30,427 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:30,427 DEBUG --- stdout ---
2024-04-15 22:05:30,427 DEBUG current:3 ready:3 replicas:3
2024-04-15 22:05:30,427 DEBUG --- stderr ---
2024-04-15 22:05:30,427 DEBUG
2024-04-15 22:05:30,427 INFO
2024-04-15 22:05:30,427 INFO *************************** Checking DS-IDREPO component is running ***************************
2024-04-15 22:05:30,428 INFO
2024-04-15 22:05:30,428 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:30,428 INFO
2024-04-15 22:05:30,428 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:30,428 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:30,814 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:30,815 DEBUG --- stdout ---
2024-04-15 22:05:30,815 DEBUG 3
2024-04-15 22:05:30,815 DEBUG --- stderr ---
2024-04-15 22:05:30,815 DEBUG
2024-04-15 22:05:30,815 INFO
2024-04-15 22:05:30,815 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-15 22:05:30,815 INFO
2024-04-15 22:05:30,815 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-idrepo --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3"
2024-04-15 22:05:30,815 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 22:05:31,217 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:31,218 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:31,218 DEBUG --- stdout ---
2024-04-15 22:05:31,218 DEBUG current:3 ready:3 replicas:3
2024-04-15 22:05:31,218 DEBUG --- stderr ---
2024-04-15 22:05:31,218 DEBUG
2024-04-15 22:05:31,218 INFO
2024-04-15 22:05:31,218 INFO ******************************* Checking AM component is running *******************************
2024-04-15 22:05:31,218 INFO
2024-04-15 22:05:31,218 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:31,218 INFO
2024-04-15 22:05:31,219 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:31,219 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:31,608 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:31,609 DEBUG --- stdout ---
2024-04-15 22:05:31,610 DEBUG 3
2024-04-15 22:05:31,610 DEBUG --- stderr ---
2024-04-15 22:05:31,610 DEBUG
2024-04-15 22:05:31,611 INFO
2024-04-15 22:05:31,611 INFO -------------- Waiting for 3 expected pod(s) to be ready --------------
2024-04-15 22:05:31,611 INFO
2024-04-15 22:05:31,611 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments am --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:3 replicas:3"
2024-04-15 22:05:31,611 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 22:05:31,997 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:31,998 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:31,998 DEBUG --- stdout ---
2024-04-15 22:05:31,998 DEBUG ready:3 replicas:3
2024-04-15 22:05:31,998 DEBUG --- stderr ---
2024-04-15 22:05:31,998 DEBUG
2024-04-15 22:05:31,998 INFO
2024-04-15 22:05:31,998 INFO ***************************** Checking AMSTER component is running *****************************
2024-04-15 22:05:31,998 INFO
2024-04-15 22:05:31,998 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:31,998 INFO
2024-04-15 22:05:31,998 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-15 22:05:31,998 INFO
2024-04-15 22:05:31,999 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get jobs amster --output jsonpath="{.status.succeeded}" | grep "1"
2024-04-15 22:05:31,999 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 22:05:32,372 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:32,372 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:32,373 DEBUG --- stdout ---
2024-04-15 22:05:32,373 DEBUG 1
2024-04-15 22:05:32,373 DEBUG --- stderr ---
2024-04-15 22:05:32,373 DEBUG
2024-04-15 22:05:32,373 INFO
2024-04-15 22:05:32,373 INFO ****************************** Checking IDM component is running ******************************
2024-04-15 22:05:32,373 INFO
2024-04-15 22:05:32,373 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:32,373 INFO
2024-04-15 22:05:32,373 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:32,373 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:32,742 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:32,743 DEBUG --- stdout ---
2024-04-15 22:05:32,743 DEBUG 2
2024-04-15 22:05:32,743 DEBUG --- stderr ---
2024-04-15 22:05:32,743 DEBUG
2024-04-15 22:05:32,743 INFO
2024-04-15 22:05:32,743 INFO -------------- Waiting for 2 expected pod(s) to be ready --------------
2024-04-15 22:05:32,743 INFO
2024-04-15 22:05:32,743 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment idm --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:2 replicas:2"
2024-04-15 22:05:32,743 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 22:05:33,129 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:33,129 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:33,129 DEBUG --- stdout ---
2024-04-15 22:05:33,129 DEBUG ready:2 replicas:2
2024-04-15 22:05:33,129 DEBUG --- stderr ---
2024-04-15 22:05:33,129 DEBUG
2024-04-15 22:05:33,129 INFO
2024-04-15 22:05:33,129 INFO ************************** Checking END-USER-UI component is running **************************
2024-04-15 22:05:33,129 INFO
2024-04-15 22:05:33,129 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:33,130 INFO
2024-04-15 22:05:33,130 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:33,130 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:33,442 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:33,443 DEBUG --- stdout ---
2024-04-15 22:05:33,443 DEBUG 1
2024-04-15 22:05:33,443 DEBUG --- stderr ---
2024-04-15 22:05:33,443 DEBUG
2024-04-15 22:05:33,443 INFO
2024-04-15 22:05:33,443 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-15 22:05:33,443 INFO
2024-04-15 22:05:33,443 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments end-user-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-15 22:05:33,443 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 22:05:33,768 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:33,769 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:33,769 DEBUG --- stdout ---
2024-04-15 22:05:33,769 DEBUG ready:1 replicas:1
2024-04-15 22:05:33,769 DEBUG --- stderr ---
2024-04-15 22:05:33,769 DEBUG
2024-04-15 22:05:33,769 INFO
2024-04-15 22:05:33,769 INFO **************************** Checking LOGIN-UI component is running ****************************
2024-04-15 22:05:33,769 INFO
2024-04-15 22:05:33,769 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:33,770 INFO
2024-04-15 22:05:33,770 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:33,770 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:34,094 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:34,095 DEBUG --- stdout ---
2024-04-15 22:05:34,095 DEBUG 1
2024-04-15 22:05:34,095 DEBUG --- stderr ---
2024-04-15 22:05:34,095 DEBUG
2024-04-15 22:05:34,095 INFO
2024-04-15 22:05:34,095 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-15 22:05:34,096 INFO
2024-04-15 22:05:34,096 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments login-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-15 22:05:34,096 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 22:05:34,465 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:34,466 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:34,466 DEBUG --- stdout ---
2024-04-15 22:05:34,466 DEBUG ready:1 replicas:1
2024-04-15 22:05:34,466 DEBUG --- stderr ---
2024-04-15 22:05:34,466 DEBUG
2024-04-15 22:05:34,466 INFO
2024-04-15 22:05:34,466 INFO **************************** Checking ADMIN-UI component is running ****************************
2024-04-15 22:05:34,466 INFO
2024-04-15 22:05:34,466 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:05:34,466 INFO
2024-04-15 22:05:34,466 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:05:34,466 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:34,775 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:34,775 DEBUG --- stdout ---
2024-04-15 22:05:34,775 DEBUG 1
2024-04-15 22:05:34,775 DEBUG --- stderr ---
2024-04-15 22:05:34,776 DEBUG
2024-04-15 22:05:34,776 INFO
2024-04-15 22:05:34,776 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-15 22:05:34,776 INFO
2024-04-15 22:05:34,776 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments admin-ui --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-15 22:05:34,776 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-15 22:05:35,114 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-15 22:05:35,115 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:35,115 DEBUG --- stdout ---
2024-04-15 22:05:35,115 DEBUG ready:1 replicas:1
2024-04-15 22:05:35,115 DEBUG --- stderr ---
2024-04-15 22:05:35,115 DEBUG
2024-04-15 22:05:35,115 INFO
2024-04-15 22:05:35,115 INFO ****************************** Livecheck stage: After deployment ******************************
2024-04-15 22:05:35,115 INFO
2024-04-15 22:05:35,115 INFO ---------------------- Running DS-CTS livecheck ----------------------
2024-04-15 22:05:35,115 INFO Livecheck to ds-cts-0
2024-04-15 22:05:35,116 INFO
2024-04-15 22:05:35,116 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.dirmanager\.pw}"
2024-04-15 22:05:35,116 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-15 22:05:35,474 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:35,475 DEBUG --- stdout ---
2024-04-15 22:05:35,475 DEBUG YjJFdDlYdUlSY0J3bnRZVWk3SjdmN09BU1M5WWJRcXg=
2024-04-15 22:05:35,475 DEBUG --- stderr ---
2024-04-15 22:05:35,475 DEBUG
2024-04-15 22:05:35,475 INFO
2024-04-15 22:05:35,475 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 22:05:37,353 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:05:37,353 DEBUG --- stdout ---
2024-04-15 22:05:37,353 DEBUG dn:
alive: true
2024-04-15 22:05:37,353 DEBUG --- stderr ---
2024-04-15 22:05:37,353 DEBUG
2024-04-15 22:05:37,353 INFO
2024-04-15 22:05:37,354 INFO Livecheck to ds-cts-1
2024-04-15 22:05:37,354 INFO
2024-04-15 22:05:37,354 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 22:05:39,153 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:05:39,153 DEBUG --- stdout ---
2024-04-15 22:05:39,153 DEBUG dn:
alive: true
2024-04-15 22:05:39,153 DEBUG --- stderr ---
2024-04-15 22:05:39,153 DEBUG
2024-04-15 22:05:39,153 INFO
2024-04-15 22:05:39,153 INFO Livecheck to ds-cts-2
2024-04-15 22:05:39,153 INFO
2024-04-15 22:05:39,154 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 22:05:40,956 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:05:40,956 DEBUG --- stdout ---
2024-04-15 22:05:40,956 DEBUG dn:
alive: true
2024-04-15 22:05:40,956 DEBUG --- stderr ---
2024-04-15 22:05:40,956 DEBUG
2024-04-15 22:05:40,956 INFO
2024-04-15 22:05:40,956 INFO
2024-04-15 22:05:40,956 INFO --------------------- Running DS-IDREPO livecheck ---------------------
2024-04-15 22:05:40,957 INFO Livecheck to ds-idrepo-0
2024-04-15 22:05:40,957 INFO
2024-04-15 22:05:40,957 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.dirmanager\.pw}"
2024-04-15 22:05:40,957 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-15 22:05:41,290 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:41,291 DEBUG --- stdout ---
2024-04-15 22:05:41,291 DEBUG YjJFdDlYdUlSY0J3bnRZVWk3SjdmN09BU1M5WWJRcXg=
2024-04-15 22:05:41,291 DEBUG --- stderr ---
2024-04-15 22:05:41,291 DEBUG
2024-04-15 22:05:41,291 INFO
2024-04-15 22:05:41,291 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 22:05:43,068 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:05:43,069 DEBUG --- stdout ---
2024-04-15 22:05:43,069 DEBUG dn:
alive: true
2024-04-15 22:05:43,069 DEBUG --- stderr ---
2024-04-15 22:05:43,069 DEBUG
2024-04-15 22:05:43,069 INFO
2024-04-15 22:05:43,069 INFO Livecheck to ds-idrepo-1
2024-04-15 22:05:43,069 INFO
2024-04-15 22:05:43,070 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 22:05:44,834 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:05:44,834 DEBUG --- stdout ---
2024-04-15 22:05:44,835 DEBUG dn:
alive: true
2024-04-15 22:05:44,835 DEBUG --- stderr ---
2024-04-15 22:05:44,835 DEBUG
2024-04-15 22:05:44,835 INFO
2024-04-15 22:05:44,835 INFO Livecheck to ds-idrepo-2
2024-04-15 22:05:44,835 INFO
2024-04-15 22:05:44,836 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive
2024-04-15 22:05:46,676 INFO [run_command]: OK (rc = 0 - expected to be in [0])
2024-04-15 22:05:46,677 DEBUG --- stdout ---
2024-04-15 22:05:46,677 DEBUG dn:
alive: true
2024-04-15 22:05:46,677 DEBUG --- stderr ---
2024-04-15 22:05:46,677 DEBUG
2024-04-15 22:05:46,677 INFO
2024-04-15 22:05:46,677 INFO
2024-04-15 22:05:46,677 INFO ------------------------ Running AM livecheck ------------------------
2024-04-15 22:05:46,677 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready
2024-04-15 22:05:46,678 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready"
2024-04-15 22:05:46,965 INFO [http_cmd]: http status code OK
2024-04-15 22:05:46,965 DEBUG --- status code ---
2024-04-15 22:05:46,965 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:46,965 DEBUG --- http response ---
2024-04-15 22:05:46,966 DEBUG
2024-04-15 22:05:46,966 INFO
2024-04-15 22:05:46,966 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.AM_PASSWORDS_AMADMIN_CLEAR}"
2024-04-15 22:05:46,966 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-15 22:05:47,293 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:47,293 DEBUG --- stdout ---
2024-04-15 22:05:47,293 DEBUG WU53YjE0T1M1SDNjODNsV0txOWpibG5U
2024-04-15 22:05:47,293 DEBUG --- stderr ---
2024-04-15 22:05:47,293 DEBUG
2024-04-15 22:05:47,293 INFO Authenticate user amadmin via REST
2024-04-15 22:05:47,294 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/"
2024-04-15 22:05:47,516 INFO [http_cmd]: http status code OK
2024-04-15 22:05:47,516 DEBUG --- status code ---
2024-04-15 22:05:47,516 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:47,516 DEBUG --- http response ---
2024-04-15 22:05:47,517 DEBUG {
"tokenId": "bHZroViTVzgenzH5zsMBHXDQIes.*AAJTSQACMDIAAlNLABxIOHlsTUsrVytQdnVDK1BBOG1UNzBjWUdVQlU9AAR0eXBlAANDVFMAAlMxAAIwMQ..*",
"successUrl": "/am/console",
"realm": "/"
}
2024-04-15 22:05:47,520 INFO
2024-04-15 22:05:47,521 INFO ---------------------- Running AMSTER livecheck ----------------------
2024-04-15 22:05:47,522 INFO
2024-04-15 22:05:47,522 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:05:47,522 INFO
2024-04-15 22:05:47,522 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm --field-selector status.phase!=Failed get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-15 22:05:47,522 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:05:47,908 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:47,908 DEBUG --- stdout ---
2024-04-15 22:05:47,908 DEBUG amster-qv96p
2024-04-15 22:05:47,908 DEBUG --- stderr ---
2024-04-15 22:05:47,908 DEBUG
2024-04-15 22:05:48,543 INFO Amster livecheck is passed
2024-04-15 22:05:48,543 INFO
2024-04-15 22:05:48,543 INFO ------------------------ Running IDM livecheck ------------------------
2024-04-15 22:05:48,543 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/ping
2024-04-15 22:05:48,543 INFO
2024-04-15 22:05:48,543 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get secret **** --output jsonpath="{.data.OPENIDM_ADMIN_PASSWORD}"
2024-04-15 22:05:48,543 INFO [loop_until]: (max_time=60, interval=5, expected_rc=[0]
2024-04-15 22:05:48,882 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:48,882 DEBUG --- stdout ---
2024-04-15 22:05:48,882 DEBUG MVVYcW1nZmVQQjROSGdoYkdjdWFzUlhD
2024-04-15 22:05:48,882 DEBUG --- stderr ---
2024-04-15 22:05:48,882 DEBUG
2024-04-15 22:05:48,882 INFO Set admin password: 1UXqmgfePB4NHghbGcuasRXC
2024-04-15 22:05:48,883 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/ping"
2024-04-15 22:05:49,099 INFO [http_cmd]: http status code OK
2024-04-15 22:05:49,100 DEBUG --- status code ---
2024-04-15 22:05:49,100 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:49,100 DEBUG --- http response ---
2024-04-15 22:05:49,100 DEBUG {
"_id": "",
"_rev": "",
"shortDesc": "OpenIDM ready",
"state": "ACTIVE_READY"
}
2024-04-15 22:05:49,100 INFO Create new LoginSession for user "amadmin"
2024-04-15 22:05:49,100 INFO [LoginSession] Get "access_token" token for user "amadmin"
2024-04-15 22:05:49,100 INFO [LoginSession] Obtaining new "access_token" token from server because it was not obtained yet for the user
2024-04-15 22:05:49,100 INFO [LoginSession] Get "token_id" token for user "amadmin"
2024-04-15 22:05:49,100 INFO [LoginSession] Obtaining new "token_id" token from server because it was not obtained yet for the user
2024-04-15 22:05:49,100 INFO Authenticate user amadmin via REST
2024-04-15 22:05:49,101 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/"
2024-04-15 22:05:49,318 INFO [http_cmd]: http status code OK
2024-04-15 22:05:49,318 DEBUG --- status code ---
2024-04-15 22:05:49,318 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:49,318 DEBUG --- http response ---
2024-04-15 22:05:49,318 DEBUG {
"tokenId": "aK4I0fJUwXaw6q3QFD61NAXX_LQ.*AAJTSQACMDIAAlNLABxvd1hLNGJUcUl1SzNaYndjZGhRT2xydDVGZkE9AAR0eXBlAANDVFMAAlMxAAIwMQ..*",
"successUrl": "/am/console",
"realm": "/"
}
2024-04-15 22:05:49,319 INFO [LoginSession] Obtaining session info
2024-04-15 22:05:49,319 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/realms/root/sessions?_action=getSessionInfo"
2024-04-15 22:05:49,547 INFO [http_cmd]: http status code OK
2024-04-15 22:05:49,547 DEBUG --- status code ---
2024-04-15 22:05:49,547 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:49,547 DEBUG --- http response ---
2024-04-15 22:05:49,547 DEBUG {
"username": "amadmin",
"universalId": "id=amadmin,ou=user,ou=am-config",
"realm": "/",
"latestAccessTime": "2024-04-15T22:05:49Z",
"maxIdleExpirationTime": "2024-04-15T22:08:49Z",
"maxSessionExpirationTime": "2024-04-15T22:08:48Z",
"properties": {
"AMCtxId": "b5eef5cc-b013-4174-8d27-302dd8ef0203-1217420370"
}
}
2024-04-15 22:05:49,552 INFO Obtain Oauth2 authz code via REST
2024-04-15 22:05:49,553 DEBUG Oauth2 authorize for access token
2024-04-15 22:05:49,553 INFO [http_cmd]: curl --request POST --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=aK4I0fJUwXaw6q3QFD61NAXX_LQ.*AAJTSQACMDIAAlNLABxvd1hLNGJUcUl1SzNaYndjZGhRT2xydDVGZkE9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" --cookie "route=1713218750.278.48759.918359|f60edb382037eb2df1e800d563ad78a7" --data '{"decision": "Allow", "csrf": "aK4I0fJUwXaw6q3QFD61NAXX_LQ.*AAJTSQACMDIAAlNLABxvd1hLNGJUcUl1SzNaYndjZGhRT2xydDVGZkE9AAR0eXBlAANDVFMAAlMxAAIwMQ..*"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/am/oauth2/authorize?client_id=smokeclient&scope=fr:idm:*&redirect_uri=https://fake.com&response_type=code&realm=/"
2024-04-15 22:05:49,781 INFO [http_cmd]: http status code OK
2024-04-15 22:05:49,781 DEBUG --- status code ---
2024-04-15 22:05:49,782 DEBUG http status code is 302 (expected 302)
2024-04-15 22:05:49,782 DEBUG --- http response ---
2024-04-15 22:05:49,782 DEBUG {'Date': 'Mon, 15 Apr 2024 22:05:49 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store', 'Location': 'https://fake.com?code=r9kLWgFWJNHNaFtWF72HNWPbU6w&iss=https%3A%2F%2Fxlou.iam.xlou-cdm.engineeringpit.com%2Fam%2Foauth2&client_id=smokeclient', 'Pragma': 'no-cache', 'Set-Cookie': 'OAUTH_REQUEST_ATTRIBUTES=DELETED; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Path=/; Secure; HttpOnly; SameSite=none', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains'}
2024-04-15 22:05:49,783 DEBUG Oauth2 exchange authorize code for access token
2024-04-15 22:05:49,784 INFO [http_cmd]: curl -L --request POST --data '{"grant_type": "authorization_code", "redirect_uri": "https://fake.com", "client_id": "smokeclient", "code": "r9kLWgFWJNHNaFtWF72HNWPbU6w"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/am/oauth2/access_token?realm=/"
2024-04-15 22:05:49,867 INFO [http_cmd]: http status code OK
2024-04-15 22:05:49,868 DEBUG --- status code ---
2024-04-15 22:05:49,868 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:49,868 DEBUG --- http response ---
2024-04-15 22:05:49,868 DEBUG {
"access_token": "8qg7Tnf1e3DnLsV0hX2yI0kzga8",
"scope": "fr:idm:*",
"token_type": "Bearer",
"expires_in": 3599
}
2024-04-15 22:05:49,870 INFO Additional check to sync/queue to ensure sync with DS is ok (see LODESTAR-1245)
2024-04-15 22:05:49,871 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/sync/queue?_queryFilter=true&_pageSize=1"
2024-04-15 22:05:50,095 INFO [http_cmd]: http status code OK
2024-04-15 22:05:50,095 DEBUG --- status code ---
2024-04-15 22:05:50,095 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:50,095 DEBUG --- http response ---
2024-04-15 22:05:50,095 DEBUG {
"result": [],
"resultCount": 0,
"pagedResultsCookie": null,
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}
2024-04-15 22:05:50,096 INFO Try to create, query and delete a user
2024-04-15 22:05:50,097 INFO Create user idm_yemmwzf66f in IDM
2024-04-15 22:05:50,097 INFO [http_cmd]: curl --header "Authorization: Bearer 8qg7Tnf1e3DnLsV0hX2yI0kzga8" --header "Content-Type: application/json" --insecure -L --request POST --data '{"userName": "idm_yemmwzf66f", "givenName": "Peter", "sn": "Parker", "password": "T35tr0ck123", "telephoneNumber": "6669876987", "description": "IDM Test User", "roles": [], "mail": "idm_yemmwzf66f@forgerock.com"}' "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user?_action=create"
2024-04-15 22:05:50,346 INFO [http_cmd]: http status code OK
2024-04-15 22:05:50,346 DEBUG --- status code ---
2024-04-15 22:05:50,346 DEBUG http status code is 201 (expected [200, 201])
2024-04-15 22:05:50,346 DEBUG --- http response ---
2024-04-15 22:05:50,347 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-15 22:05:50,347 DEBUG ----- output -----
2024-04-15 22:05:50,347 DEBUG {
2024-04-15 22:05:50,347 DEBUG "_id": "267a9ae3-fcab-4366-8d0a-1b2b2bc8152e",
2024-04-15 22:05:50,347 DEBUG "_rev": "d2dd67f6-c775-4646-8c43-6112461729ba-1068427",
2024-04-15 22:05:50,347 DEBUG "country": null,
2024-04-15 22:05:50,347 DEBUG "mail": "idm_yemmwzf66f@forgerock.com",
2024-04-15 22:05:50,347 DEBUG "memberOfOrgIDs": [],
2024-04-15 22:05:50,347 DEBUG "postalCode": null,
2024-04-15 22:05:50,347 DEBUG "profileImage": null,
2024-04-15 22:05:50,347 DEBUG "expireAccount": null,
2024-04-15 22:05:50,347 DEBUG "activeDate": null,
2024-04-15 22:05:50,347 DEBUG
2024-04-15 22:05:50,347 DEBUG
2024-04-15 22:05:50,347 DEBUG
2024-04-15 22:05:50,347 DEBUG [...]
2024-04-15 22:05:50,347 DEBUG
2024-04-15 22:05:50,348 DEBUG
2024-04-15 22:05:50,348 DEBUG
2024-04-15 22:05:50,348 DEBUG "aliasList": [],
2024-04-15 22:05:50,348 DEBUG "kbaInfo": [],
2024-04-15 22:05:50,348 DEBUG "inactiveDate": null,
2024-04-15 22:05:50,348 DEBUG "sn": "Parker",
2024-04-15 22:05:50,348 DEBUG "preferences": null,
2024-04-15 22:05:50,348 DEBUG "userName": "idm_yemmwzf66f",
2024-04-15 22:05:50,348 DEBUG "passwordLastChangedTime": "2024-04-15T22:05:50.150Z",
2024-04-15 22:05:50,348 DEBUG "effectiveRoles": [],
2024-04-15 22:05:50,348 DEBUG "activateAccount": null
2024-04-15 22:05:50,348 DEBUG }
2024-04-15 22:05:50,348 INFO Query IDM to verify UserID 267a9ae3-fcab-4366-8d0a-1b2b2bc8152e is visible
2024-04-15 22:05:50,348 INFO [http_cmd]: curl --insecure -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user/267a9ae3-fcab-4366-8d0a-1b2b2bc8152e"
2024-04-15 22:05:50,562 INFO [http_cmd]: http status code OK
2024-04-15 22:05:50,562 DEBUG --- status code ---
2024-04-15 22:05:50,562 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:50,562 DEBUG --- http response ---
2024-04-15 22:05:50,563 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-15 22:05:50,563 DEBUG ----- output -----
2024-04-15 22:05:50,563 DEBUG {
2024-04-15 22:05:50,563 DEBUG "_id": "267a9ae3-fcab-4366-8d0a-1b2b2bc8152e",
2024-04-15 22:05:50,563 DEBUG "_rev": "d2dd67f6-c775-4646-8c43-6112461729ba-1068427",
2024-04-15 22:05:50,563 DEBUG "country": null,
2024-04-15 22:05:50,563 DEBUG "mail": "idm_yemmwzf66f@forgerock.com",
2024-04-15 22:05:50,563 DEBUG "memberOfOrgIDs": [],
2024-04-15 22:05:50,563 DEBUG "postalCode": null,
2024-04-15 22:05:50,563 DEBUG "profileImage": null,
2024-04-15 22:05:50,563 DEBUG "expireAccount": null,
2024-04-15 22:05:50,563 DEBUG "activeDate": null,
2024-04-15 22:05:50,563 DEBUG
2024-04-15 22:05:50,563 DEBUG
2024-04-15 22:05:50,563 DEBUG
2024-04-15 22:05:50,563 DEBUG [...]
2024-04-15 22:05:50,563 DEBUG
2024-04-15 22:05:50,563 DEBUG
2024-04-15 22:05:50,563 DEBUG
2024-04-15 22:05:50,563 DEBUG "aliasList": [],
2024-04-15 22:05:50,564 DEBUG "kbaInfo": [],
2024-04-15 22:05:50,564 DEBUG "inactiveDate": null,
2024-04-15 22:05:50,564 DEBUG "sn": "Parker",
2024-04-15 22:05:50,564 DEBUG "preferences": null,
2024-04-15 22:05:50,564 DEBUG "userName": "idm_yemmwzf66f",
2024-04-15 22:05:50,564 DEBUG "passwordLastChangedTime": "2024-04-15T22:05:50.150Z",
2024-04-15 22:05:50,564 DEBUG "effectiveRoles": [],
2024-04-15 22:05:50,564 DEBUG "activateAccount": null
2024-04-15 22:05:50,564 DEBUG }
2024-04-15 22:05:50,565 INFO Authenticate as user idm_yemmwzf66f
2024-04-15 22:05:50,565 INFO [http_cmd]: curl --insecure -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/authentication?_action=login"
2024-04-15 22:05:50,773 INFO [http_cmd]: http status code OK
2024-04-15 22:05:50,773 DEBUG --- status code ---
2024-04-15 22:05:50,773 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:50,773 DEBUG --- http response ---
2024-04-15 22:05:50,773 DEBUG {
"_id": "login",
"authorization": {
"id": "anonymous",
"roles": [
"internal/role/openidm-reg"
],
"component": "internal/user"
},
"authenticationId": "anonymous"
}
2024-04-15 22:05:50,775 INFO [http_cmd]: curl --insecure -L --request DELETE "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/managed/user/267a9ae3-fcab-4366-8d0a-1b2b2bc8152e"
2024-04-15 22:05:51,001 INFO [http_cmd]: http status code OK
2024-04-15 22:05:51,001 DEBUG --- status code ---
2024-04-15 22:05:51,001 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:51,001 DEBUG --- http response ---
2024-04-15 22:05:51,002 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (34 lines):
2024-04-15 22:05:51,002 DEBUG ----- output -----
2024-04-15 22:05:51,002 DEBUG {
2024-04-15 22:05:51,002 DEBUG "_id": "267a9ae3-fcab-4366-8d0a-1b2b2bc8152e",
2024-04-15 22:05:51,002 DEBUG "_rev": "d2dd67f6-c775-4646-8c43-6112461729ba-1068427",
2024-04-15 22:05:51,002 DEBUG "country": null,
2024-04-15 22:05:51,002 DEBUG "mail": "idm_yemmwzf66f@forgerock.com",
2024-04-15 22:05:51,002 DEBUG "memberOfOrgIDs": [],
2024-04-15 22:05:51,003 DEBUG "postalCode": null,
2024-04-15 22:05:51,003 DEBUG "profileImage": null,
2024-04-15 22:05:51,003 DEBUG "expireAccount": null,
2024-04-15 22:05:51,003 DEBUG "activeDate": null,
2024-04-15 22:05:51,003 DEBUG
2024-04-15 22:05:51,003 DEBUG
2024-04-15 22:05:51,003 DEBUG
2024-04-15 22:05:51,003 DEBUG [...]
2024-04-15 22:05:51,003 DEBUG
2024-04-15 22:05:51,003 DEBUG
2024-04-15 22:05:51,003 DEBUG
2024-04-15 22:05:51,003 DEBUG "aliasList": [],
2024-04-15 22:05:51,003 DEBUG "kbaInfo": [],
2024-04-15 22:05:51,003 DEBUG "inactiveDate": null,
2024-04-15 22:05:51,003 DEBUG "sn": "Parker",
2024-04-15 22:05:51,003 DEBUG "preferences": null,
2024-04-15 22:05:51,003 DEBUG "userName": "idm_yemmwzf66f",
2024-04-15 22:05:51,003 DEBUG "passwordLastChangedTime": "2024-04-15T22:05:50.150Z",
2024-04-15 22:05:51,004 DEBUG "effectiveRoles": [],
2024-04-15 22:05:51,004 DEBUG "activateAccount": null
2024-04-15 22:05:51,004 DEBUG }
2024-04-15 22:05:51,006 INFO
2024-04-15 22:05:51,007 INFO -------------------- Running END-USER-UI livecheck --------------------
2024-04-15 22:05:51,007 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/enduser
2024-04-15 22:05:51,008 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/enduser"
2024-04-15 22:05:51,289 INFO [http_cmd]: http status code OK
2024-04-15 22:05:51,289 DEBUG --- status code ---
2024-04-15 22:05:51,290 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:51,290 DEBUG --- http response ---
2024-04-15 22:05:51,290 DEBUG Identity Management
[]
2024-04-15 22:05:51,292 INFO
2024-04-15 22:05:51,292 INFO --------------------- Running LOGIN-UI livecheck ---------------------
2024-04-15 22:05:51,292 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/XUI
2024-04-15 22:05:51,293 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/XUI"
2024-04-15 22:05:51,574 INFO [http_cmd]: http status code OK
2024-04-15 22:05:51,574 DEBUG --- status code ---
2024-04-15 22:05:51,574 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:51,574 DEBUG --- http response ---
2024-04-15 22:05:51,574 DEBUG Login
[]
2024-04-15 22:05:51,576 INFO
2024-04-15 22:05:51,577 INFO --------------------- Running ADMIN-UI livecheck ---------------------
2024-04-15 22:05:51,577 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/platform
2024-04-15 22:05:51,577 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/platform"
2024-04-15 22:05:51,859 INFO [http_cmd]: http status code OK
2024-04-15 22:05:51,859 DEBUG --- status code ---
2024-04-15 22:05:51,859 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:51,859 DEBUG --- http response ---
2024-04-15 22:05:51,860 DEBUG Platform Admin
[]
2024-04-15 22:05:51,861 INFO LIVECHECK SUCCEEDED
2024-04-15 22:05:51,862 INFO
2024-04-15 22:05:51,862 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-15 22:05:51,862 INFO
2024-04-15 22:05:51,862 INFO --------------------- Get DS-CTS software version ---------------------
2024-04-15 22:05:51,863 INFO
2024-04-15 22:05:51,863 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- find /opt/opendj -name opendj-core.jar
2024-04-15 22:05:51,863 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 22:05:52,691 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:52,691 DEBUG --- stdout ---
2024-04-15 22:05:52,692 DEBUG /opt/opendj/lib/opendj-core.jar
2024-04-15 22:05:52,692 DEBUG --- stderr ---
2024-04-15 22:05:52,692 DEBUG
2024-04-15 22:05:52,692 INFO
2024-04-15 22:05:52,692 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-cts-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-cts_info/opendj-core.jar --container ds
2024-04-15 22:05:52,692 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:53,942 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:53,942 DEBUG --- stdout ---
2024-04-15 22:05:53,942 DEBUG tar: Removing leading `/' from member names
2024-04-15 22:05:53,942 DEBUG --- stderr ---
2024-04-15 22:05:53,942 DEBUG
2024-04-15 22:05:53,953 INFO
2024-04-15 22:05:53,953 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-15 22:05:53,953 INFO
2024-04-15 22:05:53,953 INFO ------------------- Get DS-IDREPO software version -------------------
2024-04-15 22:05:53,953 INFO
2024-04-15 22:05:53,953 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- find /opt/opendj -name opendj-core.jar
2024-04-15 22:05:53,953 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 22:05:54,757 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:54,757 DEBUG --- stdout ---
2024-04-15 22:05:54,757 DEBUG /opt/opendj/lib/opendj-core.jar
2024-04-15 22:05:54,757 DEBUG --- stderr ---
2024-04-15 22:05:54,757 DEBUG
2024-04-15 22:05:54,758 INFO
2024-04-15 22:05:54,758 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-idrepo-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-idrepo_info/opendj-core.jar --container ds
2024-04-15 22:05:54,758 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:56,004 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:56,004 DEBUG --- stdout ---
2024-04-15 22:05:56,004 DEBUG tar: Removing leading `/' from member names
2024-04-15 22:05:56,004 DEBUG --- stderr ---
2024-04-15 22:05:56,004 DEBUG
2024-04-15 22:05:56,023 INFO
2024-04-15 22:05:56,023 INFO ****************************** Initializing component pods for AM ******************************
2024-04-15 22:05:56,023 INFO
2024-04-15 22:05:56,023 INFO ----------------------- Get AM software version -----------------------
2024-04-15 22:05:56,024 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version
2024-04-15 22:05:56,024 INFO [LoginSession] Get "token_id" token for user "amadmin"
2024-04-15 22:05:56,024 DEBUG [LoginSession] re-using "token_id" token that remains valid for 2m 42s
2024-04-15 22:05:56,025 INFO [http_cmd]: curl -L --request GET --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=aK4I0fJUwXaw6q3QFD61NAXX_LQ.*AAJTSQACMDIAAlNLABxvd1hLNGJUcUl1SzNaYndjZGhRT2xydDVGZkE9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" --cookie "route=1713218750.278.48759.918359|f60edb382037eb2df1e800d563ad78a7" "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version"
2024-04-15 22:05:56,249 INFO [http_cmd]: http status code OK
2024-04-15 22:05:56,249 DEBUG --- status code ---
2024-04-15 22:05:56,249 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:56,249 DEBUG --- http response ---
2024-04-15 22:05:56,250 DEBUG {
"_id": "version",
"_rev": "509589602",
"version": "7.6.0-SNAPSHOT",
"fullVersion": "ForgeRock Access Management 7.6.0-SNAPSHOT Build bbf6791c961b39ef101ebad36686ee2c3cde94bb (2024-April-12 18:28)",
"revision": "bbf6791c961b39ef101ebad36686ee2c3cde94bb",
"date": "2024-April-12 18:28"
}
2024-04-15 22:05:56,253 INFO
2024-04-15 22:05:56,253 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-15 22:05:56,253 INFO
2024-04-15 22:05:56,253 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-15 22:05:56,253 INFO
2024-04-15 22:05:56,253 INFO ---------------------- Get IDM software version ----------------------
2024-04-15 22:05:56,253 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/version
2024-04-15 22:05:56,253 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/openidm/info/version"
2024-04-15 22:05:56,472 INFO [http_cmd]: http status code OK
2024-04-15 22:05:56,472 DEBUG --- status code ---
2024-04-15 22:05:56,472 DEBUG http status code is 200 (expected 200)
2024-04-15 22:05:56,472 DEBUG --- http response ---
2024-04-15 22:05:56,472 DEBUG {
"_id": "version",
"productVersion": "7.6.0-SNAPSHOT",
"productBuildDate": "20240411202127",
"productRevision": "8554e0d1c5"
}
2024-04-15 22:05:56,476 INFO
2024-04-15 22:05:56,476 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-15 22:05:56,476 INFO
2024-04-15 22:05:56,476 INFO ------------------ Get END-USER-UI software version ------------------
2024-04-15 22:05:56,476 INFO
2024-04-15 22:05:56,476 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec end-user-ui-58878d7668-zg4h2 --container end-user-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-15 22:05:56,476 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 22:05:57,221 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:57,221 DEBUG --- stdout ---
2024-04-15 22:05:57,221 DEBUG /usr/share/nginx/html/js/chunk-vendors.0380a414.js
2024-04-15 22:05:57,221 DEBUG --- stderr ---
2024-04-15 22:05:57,221 DEBUG
2024-04-15 22:05:57,222 INFO
2024-04-15 22:05:57,222 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp end-user-ui-58878d7668-zg4h2:/usr/share/nginx/html/js/chunk-vendors.0380a414.js /tmp/end-user-ui_info/chunk-vendors.0380a414.js --container end-user-ui
2024-04-15 22:05:57,222 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:05:58,369 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:58,369 DEBUG --- stdout ---
2024-04-15 22:05:58,369 DEBUG tar: Removing leading `/' from member names
2024-04-15 22:05:58,369 DEBUG --- stderr ---
2024-04-15 22:05:58,369 DEBUG
2024-04-15 22:05:58,398 INFO
2024-04-15 22:05:58,398 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-15 22:05:58,398 INFO
2024-04-15 22:05:58,398 INFO -------------------- Get LOGIN-UI software version --------------------
2024-04-15 22:05:58,399 INFO
2024-04-15 22:05:58,399 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec login-ui-5fdf49fc66-2wnx7 --container login-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-15 22:05:58,399 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 22:05:59,183 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:05:59,184 DEBUG --- stdout ---
2024-04-15 22:05:59,184 DEBUG /usr/share/nginx/html/js/chunk-vendors.f2fa5365.js
2024-04-15 22:05:59,184 DEBUG --- stderr ---
2024-04-15 22:05:59,184 DEBUG
2024-04-15 22:05:59,184 INFO
2024-04-15 22:05:59,184 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp login-ui-5fdf49fc66-2wnx7:/usr/share/nginx/html/js/chunk-vendors.f2fa5365.js /tmp/login-ui_info/chunk-vendors.f2fa5365.js --container login-ui
2024-04-15 22:05:59,184 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:06:00,307 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:00,307 DEBUG --- stdout ---
2024-04-15 22:06:00,308 DEBUG tar: Removing leading `/' from member names
2024-04-15 22:06:00,308 DEBUG --- stderr ---
2024-04-15 22:06:00,308 DEBUG
2024-04-15 22:06:00,333 INFO
2024-04-15 22:06:00,334 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-15 22:06:00,334 INFO
2024-04-15 22:06:00,334 INFO -------------------- Get ADMIN-UI software version --------------------
2024-04-15 22:06:00,335 INFO
2024-04-15 22:06:00,335 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec admin-ui-c4c7bf584-mbzlq --container admin-ui -- find /usr/share/nginx/html -name chunk-vendors.*.js
2024-04-15 22:06:00,335 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0]
2024-04-15 22:06:01,107 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:01,109 DEBUG --- stdout ---
2024-04-15 22:06:01,109 DEBUG /usr/share/nginx/html/js/chunk-vendors.b3d879ac.js
2024-04-15 22:06:01,109 DEBUG --- stderr ---
2024-04-15 22:06:01,109 DEBUG
2024-04-15 22:06:01,110 INFO
2024-04-15 22:06:01,110 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp admin-ui-c4c7bf584-mbzlq:/usr/share/nginx/html/js/chunk-vendors.b3d879ac.js /tmp/admin-ui_info/chunk-vendors.b3d879ac.js --container admin-ui
2024-04-15 22:06:01,110 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:06:02,329 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:02,329 DEBUG --- stdout ---
2024-04-15 22:06:02,329 DEBUG tar: Removing leading `/' from member names
2024-04-15 22:06:02,329 DEBUG --- stderr ---
2024-04-15 22:06:02,329 DEBUG
2024-04-15 22:06:02,387 INFO
2024-04-15 22:06:02,388 INFO
2024-04-15 22:06:02,388 INFO ====================================================================================================
2024-04-15 22:06:02,388 INFO ================ Admin password for DS-CTS is: b2Et9XuIRcBwntYUi7J7f7OASS9YbQqx ================
2024-04-15 22:06:02,388 INFO ====================================================================================================
2024-04-15 22:06:02,388 INFO
2024-04-15 22:06:02,388 INFO ====================================================================================================
2024-04-15 22:06:02,388 INFO ============== Admin password for DS-IDREPO is: b2Et9XuIRcBwntYUi7J7f7OASS9YbQqx ==============
2024-04-15 22:06:02,388 INFO ====================================================================================================
2024-04-15 22:06:02,388 INFO
2024-04-15 22:06:02,388 INFO ====================================================================================================
2024-04-15 22:06:02,388 INFO ====================== Admin password for AM is: YNwb14OS5H3c83lWKq9jblnT ======================
2024-04-15 22:06:02,388 INFO ====================================================================================================
2024-04-15 22:06:02,388 INFO
2024-04-15 22:06:02,388 INFO ====================================================================================================
2024-04-15 22:06:02,389 INFO ===================== Admin password for IDM is: 1UXqmgfePB4NHghbGcuasRXC =====================
2024-04-15 22:06:02,389 INFO ====================================================================================================
2024-04-15 22:06:02,389 INFO
2024-04-15 22:06:02,389 INFO *************************************** Dumping pod list ***************************************
2024-04-15 22:06:02,389 INFO Dumping pod list to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/_pod-list.txt
2024-04-15 22:06:02,922 INFO Dumping platform-config configmap to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/_platform-config_configmap.txt
2024-04-15 22:06:03,263 INFO
2024-04-15 22:06:03,263 INFO **************************** Initializing component pods for DS-CTS ****************************
2024-04-15 22:06:03,263 INFO
2024-04-15 22:06:03,263 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:06:03,264 INFO
2024-04-15 22:06:03,264 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:06:03,264 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:06:03,653 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:03,653 DEBUG --- stdout ---
2024-04-15 22:06:03,653 DEBUG 3
2024-04-15 22:06:03,653 DEBUG --- stderr ---
2024-04-15 22:06:03,653 DEBUG
2024-04-15 22:06:03,654 INFO
2024-04-15 22:06:03,654 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:06:03,654 INFO
2024-04-15 22:06:03,654 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-cts --output jsonpath={.items[*].metadata.name}
2024-04-15 22:06:03,654 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:06:04,116 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:04,117 DEBUG --- stdout ---
2024-04-15 22:06:04,117 DEBUG ds-cts-0 ds-cts-1 ds-cts-2
2024-04-15 22:06:04,117 DEBUG --- stderr ---
2024-04-15 22:06:04,117 DEBUG
2024-04-15 22:06:04,117 INFO
2024-04-15 22:06:04,117 INFO ************************** Initializing component pods for DS-IDREPO **************************
2024-04-15 22:06:04,117 INFO
2024-04-15 22:06:04,117 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:06:04,118 INFO
2024-04-15 22:06:04,118 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:06:04,118 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:06:04,516 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:04,516 DEBUG --- stdout ---
2024-04-15 22:06:04,517 DEBUG 3
2024-04-15 22:06:04,517 DEBUG --- stderr ---
2024-04-15 22:06:04,517 DEBUG
2024-04-15 22:06:04,517 INFO
2024-04-15 22:06:04,517 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:06:04,517 INFO
2024-04-15 22:06:04,517 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-idrepo --output jsonpath={.items[*].metadata.name}
2024-04-15 22:06:04,517 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:06:04,983 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:04,983 DEBUG --- stdout ---
2024-04-15 22:06:04,983 DEBUG ds-idrepo-0 ds-idrepo-1 ds-idrepo-2
2024-04-15 22:06:04,983 DEBUG --- stderr ---
2024-04-15 22:06:04,983 DEBUG
2024-04-15 22:06:04,983 INFO
2024-04-15 22:06:04,983 INFO ****************************** Initializing component pods for AM ******************************
2024-04-15 22:06:04,983 INFO
2024-04-15 22:06:04,983 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:06:04,984 INFO
2024-04-15 22:06:04,984 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:06:04,984 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:06:05,395 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:05,395 DEBUG --- stdout ---
2024-04-15 22:06:05,395 DEBUG 3
2024-04-15 22:06:05,395 DEBUG --- stderr ---
2024-04-15 22:06:05,395 DEBUG
2024-04-15 22:06:05,396 INFO
2024-04-15 22:06:05,396 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:06:05,396 INFO
2024-04-15 22:06:05,396 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}
2024-04-15 22:06:05,396 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:06:05,865 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:05,866 DEBUG --- stdout ---
2024-04-15 22:06:05,866 DEBUG am-7d597cf7d6-8ctft am-7d597cf7d6-8vr8f am-7d597cf7d6-9wz9h
2024-04-15 22:06:05,866 DEBUG --- stderr ---
2024-04-15 22:06:05,866 DEBUG
2024-04-15 22:06:05,866 INFO
2024-04-15 22:06:05,866 INFO **************************** Initializing component pods for AMSTER ****************************
2024-04-15 22:06:05,866 INFO
2024-04-15 22:06:05,866 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:06:05,866 INFO
2024-04-15 22:06:05,866 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=amster --output jsonpath={.items[*].metadata.name}
2024-04-15 22:06:05,866 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:06:06,257 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:06,259 DEBUG --- stdout ---
2024-04-15 22:06:06,260 DEBUG amster-qv96p
2024-04-15 22:06:06,260 DEBUG --- stderr ---
2024-04-15 22:06:06,260 DEBUG
2024-04-15 22:06:06,260 INFO
2024-04-15 22:06:06,260 INFO ***************************** Initializing component pods for IDM *****************************
2024-04-15 22:06:06,260 INFO
2024-04-15 22:06:06,260 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:06:06,260 INFO
2024-04-15 22:06:06,260 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployment --selector app=idm --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:06:06,260 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:06:06,670 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:06,670 DEBUG --- stdout ---
2024-04-15 22:06:06,670 DEBUG 2
2024-04-15 22:06:06,670 DEBUG --- stderr ---
2024-04-15 22:06:06,670 DEBUG
2024-04-15 22:06:06,670 INFO
2024-04-15 22:06:06,670 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:06:06,671 INFO
2024-04-15 22:06:06,671 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=idm --output jsonpath={.items[*].metadata.name}
2024-04-15 22:06:06,671 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:06:07,082 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:07,082 DEBUG --- stdout ---
2024-04-15 22:06:07,082 DEBUG idm-b85f9d978-j5pbm idm-b85f9d978-rl76f
2024-04-15 22:06:07,082 DEBUG --- stderr ---
2024-04-15 22:06:07,082 DEBUG
2024-04-15 22:06:07,082 INFO
2024-04-15 22:06:07,082 INFO ************************* Initializing component pods for END-USER-UI *************************
2024-04-15 22:06:07,083 INFO
2024-04-15 22:06:07,083 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:06:07,083 INFO
2024-04-15 22:06:07,083 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:06:07,083 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:06:07,417 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:07,417 DEBUG --- stdout ---
2024-04-15 22:06:07,417 DEBUG 1
2024-04-15 22:06:07,417 DEBUG --- stderr ---
2024-04-15 22:06:07,417 DEBUG
2024-04-15 22:06:07,418 INFO
2024-04-15 22:06:07,418 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:06:07,418 INFO
2024-04-15 22:06:07,418 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=end-user-ui --output jsonpath={.items[*].metadata.name}
2024-04-15 22:06:07,418 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:06:07,763 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:07,763 DEBUG --- stdout ---
2024-04-15 22:06:07,763 DEBUG end-user-ui-58878d7668-zg4h2
2024-04-15 22:06:07,763 DEBUG --- stderr ---
2024-04-15 22:06:07,763 DEBUG
2024-04-15 22:06:07,763 INFO
2024-04-15 22:06:07,763 INFO *************************** Initializing component pods for LOGIN-UI ***************************
2024-04-15 22:06:07,763 INFO
2024-04-15 22:06:07,763 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:06:07,764 INFO
2024-04-15 22:06:07,764 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:06:07,764 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:06:08,096 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:08,096 DEBUG --- stdout ---
2024-04-15 22:06:08,096 DEBUG 1
2024-04-15 22:06:08,096 DEBUG --- stderr ---
2024-04-15 22:06:08,096 DEBUG
2024-04-15 22:06:08,097 INFO
2024-04-15 22:06:08,097 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:06:08,097 INFO
2024-04-15 22:06:08,097 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=login-ui --output jsonpath={.items[*].metadata.name}
2024-04-15 22:06:08,097 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:06:08,408 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:08,408 DEBUG --- stdout ---
2024-04-15 22:06:08,408 DEBUG login-ui-5fdf49fc66-2wnx7
2024-04-15 22:06:08,408 DEBUG --- stderr ---
2024-04-15 22:06:08,408 DEBUG
2024-04-15 22:06:08,408 INFO
2024-04-15 22:06:08,408 INFO *************************** Initializing component pods for ADMIN-UI ***************************
2024-04-15 22:06:08,409 INFO
2024-04-15 22:06:08,409 INFO --------------------- Get expected number of pods ---------------------
2024-04-15 22:06:08,409 INFO
2024-04-15 22:06:08,409 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].spec.replicas}
2024-04-15 22:06:08,409 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-15 22:06:08,716 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:08,716 DEBUG --- stdout ---
2024-04-15 22:06:08,716 DEBUG 1
2024-04-15 22:06:08,716 DEBUG --- stderr ---
2024-04-15 22:06:08,716 DEBUG
2024-04-15 22:06:08,716 INFO
2024-04-15 22:06:08,716 INFO ---------------------------- Get pod list ----------------------------
2024-04-15 22:06:08,716 INFO
2024-04-15 22:06:08,716 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app.kubernetes.io/name=admin-ui --output jsonpath={.items[*].metadata.name}
2024-04-15 22:06:08,717 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-15 22:06:09,036 INFO [loop_until]: OK (rc = 0)
2024-04-15 22:06:09,036 DEBUG --- stdout ---
2024-04-15 22:06:09,036 DEBUG admin-ui-c4c7bf584-mbzlq
2024-04-15 22:06:09,037 DEBUG --- stderr ---
2024-04-15 22:06:09,037 DEBUG
2024-04-15 22:06:09,037 INFO
2024-04-15 22:06:09,037 INFO *********************************** Dumping components logs ***********************************
2024-04-15 22:06:09,037 INFO
2024-04-15 22:06:09,037 INFO ----------------------- Dumping logs for DS-CTS -----------------------
2024-04-15 22:06:10,258 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-cts-0.txt
2024-04-15 22:06:10,259 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-cts-0.txt
2024-04-15 22:06:10,259 INFO Check pod logs for errors
2024-04-15 22:06:11,605 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-cts-1.txt
2024-04-15 22:06:11,608 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-cts-1.txt
2024-04-15 22:06:11,611 INFO Check pod logs for errors
2024-04-15 22:06:13,067 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-cts-2.txt
2024-04-15 22:06:13,068 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-cts-2.txt
2024-04-15 22:06:13,071 INFO Check pod logs for errors
2024-04-15 22:06:13,074 INFO
2024-04-15 22:06:13,074 INFO --------------------- Dumping logs for DS-IDREPO ---------------------
2024-04-15 22:06:14,852 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-idrepo-0.txt
2024-04-15 22:06:14,852 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-idrepo-0.txt
2024-04-15 22:06:14,887 INFO Check pod logs for errors
2024-04-15 22:06:16,146 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-idrepo-1.txt
2024-04-15 22:06:16,147 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-idrepo-1.txt
2024-04-15 22:06:16,147 INFO Check pod logs for errors
2024-04-15 22:06:17,343 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-idrepo-2.txt
2024-04-15 22:06:17,345 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/ds-idrepo-2.txt
2024-04-15 22:06:17,345 INFO Check pod logs for errors
2024-04-15 22:06:17,346 INFO
2024-04-15 22:06:17,346 INFO ------------------------- Dumping logs for AM -------------------------
2024-04-15 22:06:19,078 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/am-7d597cf7d6-8ctft.txt
2024-04-15 22:06:19,078 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/am-7d597cf7d6-8ctft.txt
2024-04-15 22:06:19,102 INFO Check pod logs for errors
2024-04-15 22:06:20,878 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/am-7d597cf7d6-8vr8f.txt
2024-04-15 22:06:20,879 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/am-7d597cf7d6-8vr8f.txt
2024-04-15 22:06:20,911 INFO Check pod logs for errors
2024-04-15 22:06:22,576 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/am-7d597cf7d6-9wz9h.txt
2024-04-15 22:06:22,577 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/am-7d597cf7d6-9wz9h.txt
2024-04-15 22:06:22,602 INFO Check pod logs for errors
2024-04-15 22:06:22,619 INFO
2024-04-15 22:06:22,619 INFO ----------------------- Dumping logs for AMSTER -----------------------
2024-04-15 22:06:23,797 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/amster-qv96p.txt
2024-04-15 22:06:23,798 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/amster-qv96p.txt
2024-04-15 22:06:23,798 INFO Check pod logs for errors
2024-04-15 22:06:23,798 INFO
2024-04-15 22:06:23,798 INFO ------------------------ Dumping logs for IDM ------------------------
2024-04-15 22:06:25,506 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/idm-b85f9d978-j5pbm.txt
2024-04-15 22:06:25,506 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/idm-b85f9d978-j5pbm.txt
2024-04-15 22:06:25,534 INFO Check pod logs for errors
2024-04-15 22:06:27,233 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/idm-b85f9d978-rl76f.txt
2024-04-15 22:06:27,234 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/idm-b85f9d978-rl76f.txt
2024-04-15 22:06:27,262 INFO Check pod logs for errors
2024-04-15 22:06:27,279 INFO
2024-04-15 22:06:27,280 INFO -------------------- Dumping logs for END-USER-UI --------------------
2024-04-15 22:06:28,656 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/end-user-ui-58878d7668-zg4h2.txt
2024-04-15 22:06:28,657 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/end-user-ui-58878d7668-zg4h2.txt
2024-04-15 22:06:28,667 INFO Check pod logs for errors
2024-04-15 22:06:28,679 INFO
2024-04-15 22:06:28,679 INFO ---------------------- Dumping logs for LOGIN-UI ----------------------
2024-04-15 22:06:30,083 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/login-ui-5fdf49fc66-2wnx7.txt
2024-04-15 22:06:30,084 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/login-ui-5fdf49fc66-2wnx7.txt
2024-04-15 22:06:30,098 INFO Check pod logs for errors
2024-04-15 22:06:30,113 INFO
2024-04-15 22:06:30,113 INFO ---------------------- Dumping logs for ADMIN-UI ----------------------
2024-04-15 22:06:31,495 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/admin-ui-c4c7bf584-mbzlq.txt
2024-04-15 22:06:31,496 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240415_220602-after-deployment/admin-ui-c4c7bf584-mbzlq.txt
2024-04-15 22:06:31,508 INFO Check pod logs for errors
[2024-04-15 22:06:31] - INFO: Deployment ready
________________________________________________________________________________
[2024-04-15 22:06:31] Discover_all_components post : Post method
________________________________________________________________________________
Setting result to PASS
Task has been successfully stopped