--Task--
name: Deploy_or_discover_overseer
enabled: True
class_name: DeployOrDiscoverOverseerTask
source_name: controller
source_namespace: >default<
target_name: controller
target_namespace: >default<
start: 0.0
stop: None
timeout: no timeout
loop: False
interval: None
dependencies: ['Discover_IDC_tenant']
wait_for: []
preceding_task: None
options: {}
group_name: None
Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock
________________________________________________________________________________
[2024-06-22 18:08:52] Deploy_or_discover_overseer pre : Initialising task parameters
________________________________________________________________________________
task will be executed on controller (localhost)
________________________________________________________________________________
[2024-06-22 18:08:52] Deploy_or_discover_overseer step1 : Discover or deploy overseer
________________________________________________________________________________
2024-06-22 18:08:53,155 INFO
2024-06-22 18:08:53,155 INFO [loop_until]: kubectl --namespace=xlou get deployments -l "release=overseer" -o name
2024-06-22 18:08:53,155 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-06-22 18:08:53,494 INFO [loop_until]: OK (rc = 0)
2024-06-22 18:08:53,494 DEBUG --- stdout ---
2024-06-22 18:08:53,494 DEBUG deployment.apps/overseer-0
2024-06-22 18:08:53,494 DEBUG --- stderr ---
2024-06-22 18:08:53,494 DEBUG
Discover overseer pod: overseer-0
2024-06-22 18:08:53,780 INFO
2024-06-22 18:08:53,780 INFO --------------------- Get expected number of pods ---------------------
2024-06-22 18:08:53,781 INFO
2024-06-22 18:08:53,781 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-06-22 18:08:53,781 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-06-22 18:08:54,101 INFO [loop_until]: OK (rc = 0)
2024-06-22 18:08:54,101 DEBUG --- stdout ---
2024-06-22 18:08:54,101 DEBUG 1
2024-06-22 18:08:54,101 DEBUG --- stderr ---
2024-06-22 18:08:54,102 DEBUG
2024-06-22 18:08:54,102 INFO
2024-06-22 18:08:54,102 INFO ------------- Reloading pod list for product "overseer-0" -------------
2024-06-22 18:08:54,102 INFO
2024-06-22 18:08:54,102 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1
2024-06-22 18:08:54,102 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0]
2024-06-22 18:08:54,437 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-06-22 18:08:54,437 INFO [loop_until]: OK (rc = 0)
2024-06-22 18:08:54,437 DEBUG --- stdout ---
2024-06-22 18:08:54,437 DEBUG overseer-0-5b6f9464fd-slbm8
2024-06-22 18:08:54,437 DEBUG --- stderr ---
2024-06-22 18:08:54,437 DEBUG
2024-06-22 18:08:54,437 INFO
2024-06-22 18:08:54,437 INFO ---------- Check pod overseer-0-5b6f9464fd-slbm8 is running ----------
2024-06-22 18:08:54,437 INFO
2024-06-22 18:08:54,437 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods overseer-0-5b6f9464fd-slbm8 -o=jsonpath={.status.phase} | grep "Running"
2024-06-22 18:08:54,437 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-06-22 18:08:54,760 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-06-22 18:08:54,761 INFO [loop_until]: OK (rc = 0)
2024-06-22 18:08:54,761 DEBUG --- stdout ---
2024-06-22 18:08:54,761 DEBUG Running
2024-06-22 18:08:54,761 DEBUG --- stderr ---
2024-06-22 18:08:54,761 DEBUG
2024-06-22 18:08:54,761 INFO
2024-06-22 18:08:54,761 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods overseer-0-5b6f9464fd-slbm8 -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-06-22 18:08:54,761 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-06-22 18:08:55,057 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-06-22 18:08:55,057 INFO [loop_until]: OK (rc = 0)
2024-06-22 18:08:55,057 DEBUG --- stdout ---
2024-06-22 18:08:55,057 DEBUG true
2024-06-22 18:08:55,057 DEBUG --- stderr ---
2024-06-22 18:08:55,057 DEBUG
2024-06-22 18:08:55,058 INFO
2024-06-22 18:08:55,058 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod overseer-0-5b6f9464fd-slbm8 --output jsonpath={.status.startTime}
2024-06-22 18:08:55,058 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-06-22 18:08:55,361 INFO [loop_until]: OK (rc = 0)
2024-06-22 18:08:55,362 DEBUG --- stdout ---
2024-06-22 18:08:55,362 DEBUG 2024-06-18T17:51:04Z
2024-06-22 18:08:55,362 DEBUG --- stderr ---
2024-06-22 18:08:55,362 DEBUG
2024-06-22 18:08:55,362 INFO
2024-06-22 18:08:55,362 INFO --- Check pod overseer-0-5b6f9464fd-slbm8 filesystem is accessible ---
2024-06-22 18:08:56,125 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-06-22 18:08:56,125 INFO
2024-06-22 18:08:56,125 INFO --------- Check pod overseer-0-5b6f9464fd-slbm8 restart count ---------
2024-06-22 18:08:56,125 INFO
2024-06-22 18:08:56,125 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod overseer-0-5b6f9464fd-slbm8 --output jsonpath={.status.containerStatuses[*].restartCount}
2024-06-22 18:08:56,125 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-06-22 18:08:56,428 INFO [loop_until]: OK (rc = 0)
2024-06-22 18:08:56,428 DEBUG --- stdout ---
2024-06-22 18:08:56,429 DEBUG 0
2024-06-22 18:08:56,429 DEBUG --- stderr ---
2024-06-22 18:08:56,429 DEBUG
2024-06-22 18:08:56,429 INFO Pod overseer-0-5b6f9464fd-slbm8 has been restarted 0 times.
2024-06-22 18:08:56,429 INFO
2024-06-22 18:08:56,429 INFO --------------------- Get expected number of pods ---------------------
2024-06-22 18:08:56,429 INFO
2024-06-22 18:08:56,429 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-06-22 18:08:56,429 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-06-22 18:08:56,733 INFO [loop_until]: OK (rc = 0)
2024-06-22 18:08:56,733 DEBUG --- stdout ---
2024-06-22 18:08:56,733 DEBUG 1
2024-06-22 18:08:56,733 DEBUG --- stderr ---
2024-06-22 18:08:56,733 DEBUG
2024-06-22 18:08:56,733 INFO
2024-06-22 18:08:56,733 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-06-22 18:08:56,734 INFO
2024-06-22 18:08:56,734 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments overseer-0 --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-06-22 18:08:56,734 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-06-22 18:08:57,056 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-06-22 18:08:57,056 INFO [loop_until]: OK (rc = 0)
2024-06-22 18:08:57,056 DEBUG --- stdout ---
2024-06-22 18:08:57,057 DEBUG ready:1 replicas:1
2024-06-22 18:08:57,057 DEBUG --- stderr ---
2024-06-22 18:08:57,057 DEBUG
2024-06-22 18:08:57,057 INFO Livecheck to https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com
2024-06-22 18:08:57,057 INFO [http_cmd]: curl --insecure -L --request GET "https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com"
2024-06-22 18:08:57,310 INFO [http_cmd]: http status code OK
2024-06-22 18:08:57,310 DEBUG --- status code ---
2024-06-22 18:08:57,310 DEBUG http status code is 200 (expected 200)
2024-06-22 18:08:57,310 DEBUG --- http response ---
2024-06-22 18:08:57,310 DEBUG
Directory listing for /
Directory listing for /
________________________________________________________________________________
[2024-06-22 18:08:57] Deploy_or_discover_overseer post : Post method
________________________________________________________________________________
Setting result to PASS
Task has been successfully stopped