--Task--
name: Deploy_or_discover_overseer
enabled: True
class_name: DeployOrDiscoverOverseerTask
source_name: controller
source_namespace: >default<
target_name: controller
target_namespace: >default<
start: 0.0
stop: None
timeout: no timeout
loop: False
interval: None
dependencies: ['Deploy_all_forgerock_components']
wait_for: []
preceding_task: None
options: {}
group_name: None
Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock
________________________________________________________________________________
[2024-04-11 01:25:50] Deploy_or_discover_overseer pre : Initialising task parameters
________________________________________________________________________________
task will be executed on controller (localhost)
________________________________________________________________________________
[2024-04-11 01:25:50] Deploy_or_discover_overseer step1 : Discover or deploy overseer
________________________________________________________________________________
2024-04-11 01:25:50,953 INFO
2024-04-11 01:25:50,954 INFO [loop_until]: kubectl --namespace=xlou get deployments -l "release=overseer" -o name
2024-04-11 01:25:50,954 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-11 01:25:51,247 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:25:51,247 DEBUG --- stdout ---
2024-04-11 01:25:51,247 DEBUG
2024-04-11 01:25:51,247 DEBUG --- stderr ---
2024-04-11 01:25:51,247 DEBUG
overseer-0 pod does not exit, deploy it
2024-04-11 01:25:51,495 INFO
2024-04-11 01:25:51,496 INFO ------------ Deploy the lodestarbox with overseer profile ------------
2024-04-11 01:25:51,496 INFO
2024-04-11 01:25:51,496 INFO [run_command]: skaffold deploy --profile overseer --config=/tmp/tmpgyefx5_z --status-check=true --namespace=xlou
2024-04-11 01:25:54,314 INFO Starting deploy...
2024-04-11 01:25:57,323 INFO - serviceaccount/k8s-svc-acct-0 created
2024-04-11 01:25:57,424 INFO - clusterrolebinding.rbac.authorization.k8s.io/k8s-svc-acct-crb-xlou-0 created
2024-04-11 01:25:57,725 INFO - configmap/overseer-config-0 created
2024-04-11 01:25:57,926 INFO - service/overseer-0 created
2024-04-11 01:25:58,227 INFO - persistentvolumeclaim/overseer-0 created
2024-04-11 01:25:58,428 INFO - deployment.apps/overseer-0 created
2024-04-11 01:25:58,729 INFO - ingress.networking.k8s.io/overseer-0 created
2024-04-11 01:25:58,830 INFO Waiting for deployments to stabilize...
2024-04-11 01:26:04,049 INFO - xlou:deployment/overseer-0: creating container overseer
2024-04-11 01:26:04,150 INFO - xlou:pod/overseer-0-78845f4f8c-v2r2q: creating container overseer
2024-04-11 01:26:10,468 INFO - xlou:deployment/overseer-0 is ready.
2024-04-11 01:26:10,569 INFO Deployments stabilized in 11.764 seconds
2024-04-11 01:26:10,670 ERROR There is a new version (2.11.0) of Skaffold available. Download it from:
2024-04-11 01:26:10,670 ERROR https://github.com/GoogleContainerTools/skaffold/releases/tag/v2.11.0
2024-04-11 01:26:10,670 ERROR
2024-04-11 01:26:10,670 ERROR Help improve Skaffold with our 2-minute anonymous survey: run 'skaffold survey'
2024-04-11 01:26:10,670 ERROR To help improve the quality of this product, we collect anonymized usage data for details on what is tracked and how we use this data visit . This data is handled in accordance with our privacy policy
2024-04-11 01:26:10,670 ERROR
2024-04-11 01:26:10,670 ERROR You may choose to opt out of this collection by running the following command:
2024-04-11 01:26:10,670 ERROR skaffold config set --global collect-metrics false
2024-04-11 01:26:10,671 INFO
2024-04-11 01:26:10,671 INFO --------------------- Get expected number of pods ---------------------
2024-04-11 01:26:10,671 INFO
2024-04-11 01:26:10,671 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-04-11 01:26:10,671 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-11 01:26:10,974 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:10,974 DEBUG --- stdout ---
2024-04-11 01:26:10,974 DEBUG 1
2024-04-11 01:26:10,974 DEBUG --- stderr ---
2024-04-11 01:26:10,974 DEBUG
2024-04-11 01:26:10,975 INFO
2024-04-11 01:26:10,975 INFO ---------------------------- Get pod list ----------------------------
2024-04-11 01:26:10,975 INFO
2024-04-11 01:26:10,975 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-11 01:26:10,975 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-11 01:26:11,294 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-11 01:26:11,295 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:11,295 DEBUG --- stdout ---
2024-04-11 01:26:11,295 DEBUG overseer-0-78845f4f8c-v2r2q
2024-04-11 01:26:11,295 DEBUG --- stderr ---
2024-04-11 01:26:11,295 DEBUG
2024-04-11 01:26:11,295 INFO
2024-04-11 01:26:11,295 INFO ---------- Check pod overseer-0-78845f4f8c-v2r2q is running ----------
2024-04-11 01:26:11,296 INFO
2024-04-11 01:26:11,296 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods overseer-0-78845f4f8c-v2r2q -o=jsonpath={.status.phase} | grep "Running"
2024-04-11 01:26:11,296 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-11 01:26:11,678 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-11 01:26:11,679 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:11,679 DEBUG --- stdout ---
2024-04-11 01:26:11,679 DEBUG Running
2024-04-11 01:26:11,679 DEBUG --- stderr ---
2024-04-11 01:26:11,679 DEBUG
2024-04-11 01:26:11,679 INFO
2024-04-11 01:26:11,679 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods overseer-0-78845f4f8c-v2r2q -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-04-11 01:26:11,679 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-04-11 01:26:12,052 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-11 01:26:12,053 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:12,053 DEBUG --- stdout ---
2024-04-11 01:26:12,053 DEBUG true
2024-04-11 01:26:12,053 DEBUG --- stderr ---
2024-04-11 01:26:12,053 DEBUG
2024-04-11 01:26:12,053 INFO
2024-04-11 01:26:12,053 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod overseer-0-78845f4f8c-v2r2q --output jsonpath={.status.startTime}
2024-04-11 01:26:12,053 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-11 01:26:12,370 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:12,370 DEBUG --- stdout ---
2024-04-11 01:26:12,370 DEBUG 2024-04-11T01:26:02Z
2024-04-11 01:26:12,370 DEBUG --- stderr ---
2024-04-11 01:26:12,370 DEBUG
2024-04-11 01:26:12,371 INFO
2024-04-11 01:26:12,371 INFO --- Check pod overseer-0-78845f4f8c-v2r2q filesystem is accessible ---
2024-04-11 01:26:13,163 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-11 01:26:13,163 INFO
2024-04-11 01:26:13,163 INFO --------- Check pod overseer-0-78845f4f8c-v2r2q restart count ---------
2024-04-11 01:26:13,164 INFO
2024-04-11 01:26:13,164 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod overseer-0-78845f4f8c-v2r2q --output jsonpath={.status.containerStatuses[*].restartCount}
2024-04-11 01:26:13,164 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-11 01:26:13,464 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:13,464 DEBUG --- stdout ---
2024-04-11 01:26:13,464 DEBUG 0
2024-04-11 01:26:13,464 DEBUG --- stderr ---
2024-04-11 01:26:13,465 DEBUG
2024-04-11 01:26:13,465 INFO Pod overseer-0-78845f4f8c-v2r2q has been restarted 0 times.
2024-04-11 01:26:13,465 INFO
2024-04-11 01:26:13,465 INFO --------------------- Get expected number of pods ---------------------
2024-04-11 01:26:13,465 INFO
2024-04-11 01:26:13,465 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-04-11 01:26:13,465 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-11 01:26:13,792 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:13,793 DEBUG --- stdout ---
2024-04-11 01:26:13,793 DEBUG 1
2024-04-11 01:26:13,793 DEBUG --- stderr ---
2024-04-11 01:26:13,793 DEBUG
2024-04-11 01:26:13,793 INFO
2024-04-11 01:26:13,793 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-11 01:26:13,793 INFO
2024-04-11 01:26:13,793 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments overseer-0 --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-11 01:26:13,793 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-11 01:26:14,096 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-11 01:26:14,097 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:14,097 DEBUG --- stdout ---
2024-04-11 01:26:14,097 DEBUG ready:1 replicas:1
2024-04-11 01:26:14,097 DEBUG --- stderr ---
2024-04-11 01:26:14,097 DEBUG
2024-04-11 01:26:14,097 INFO Livecheck to https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com
2024-04-11 01:26:14,098 INFO [http_cmd]: curl --insecure -L --request GET "https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com"
2024-04-11 01:26:14,364 INFO [http_cmd]: http status code OK
2024-04-11 01:26:14,364 DEBUG --- status code ---
2024-04-11 01:26:14,365 DEBUG http status code is 200 (expected 200)
2024-04-11 01:26:14,365 DEBUG --- http response ---
2024-04-11 01:26:14,365 DEBUG
Directory listing for /
Directory listing for /
2024-04-11 01:26:14,365 INFO Component overseer-0 is alive
2024-04-11 01:26:15,249 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240411_012614-after-overseer-0-deploy/overseer-0-78845f4f8c-v2r2q.txt
2024-04-11 01:26:15,250 INFO Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/pod-logs/stack/20240411_012614-after-overseer-0-deploy/overseer-0-78845f4f8c-v2r2q.txt
2024-04-11 01:26:15,250 INFO Check pod logs for errors
2024-04-11 01:26:15,251 INFO
2024-04-11 01:26:15,251 INFO --------------------- Get expected number of pods ---------------------
2024-04-11 01:26:15,251 INFO
2024-04-11 01:26:15,251 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-04-11 01:26:15,251 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-11 01:26:15,570 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:15,570 DEBUG --- stdout ---
2024-04-11 01:26:15,570 DEBUG 1
2024-04-11 01:26:15,570 DEBUG --- stderr ---
2024-04-11 01:26:15,570 DEBUG
2024-04-11 01:26:15,570 INFO
2024-04-11 01:26:15,570 INFO ---------------------------- Get pod list ----------------------------
2024-04-11 01:26:15,571 INFO
2024-04-11 01:26:15,571 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1
2024-04-11 01:26:15,571 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-04-11 01:26:15,889 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-04-11 01:26:15,890 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:15,890 DEBUG --- stdout ---
2024-04-11 01:26:15,890 DEBUG overseer-0-78845f4f8c-v2r2q
2024-04-11 01:26:15,890 DEBUG --- stderr ---
2024-04-11 01:26:15,890 DEBUG
2024-04-11 01:26:15,890 INFO
2024-04-11 01:26:15,890 INFO --------------------- Get expected number of pods ---------------------
2024-04-11 01:26:15,890 INFO
2024-04-11 01:26:15,890 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-04-11 01:26:15,890 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-04-11 01:26:16,222 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:16,222 DEBUG --- stdout ---
2024-04-11 01:26:16,222 DEBUG 1
2024-04-11 01:26:16,222 DEBUG --- stderr ---
2024-04-11 01:26:16,222 DEBUG
2024-04-11 01:26:16,223 INFO
2024-04-11 01:26:16,223 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-04-11 01:26:16,223 INFO
2024-04-11 01:26:16,223 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments overseer-0 --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-04-11 01:26:16,223 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-04-11 01:26:16,537 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-04-11 01:26:16,537 INFO [loop_until]: OK (rc = 0)
2024-04-11 01:26:16,537 DEBUG --- stdout ---
2024-04-11 01:26:16,537 DEBUG ready:1 replicas:1
2024-04-11 01:26:16,537 DEBUG --- stderr ---
2024-04-11 01:26:16,537 DEBUG
2024-04-11 01:26:16,538 INFO Livecheck to https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com
2024-04-11 01:26:16,539 INFO [http_cmd]: curl --insecure -L --request GET "https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com"
2024-04-11 01:26:16,745 INFO [http_cmd]: http status code OK
2024-04-11 01:26:16,745 DEBUG --- status code ---
2024-04-11 01:26:16,745 DEBUG http status code is 200 (expected 200)
2024-04-11 01:26:16,745 DEBUG --- http response ---
2024-04-11 01:26:16,746 DEBUG
Directory listing for /
Directory listing for /
________________________________________________________________________________
[2024-04-11 01:26:16] Deploy_or_discover_overseer post : Post method
________________________________________________________________________________
Setting result to PASS
Task has been successfully stopped