--Task--
name: Deploy_or_discover_overseer
enabled: True
class_name: DeployOrDiscoverOverseerTask
source_name: controller
source_namespace: >default<
target_name: controller
target_namespace: >default<
start: 0.0
stop: None
timeout: no timeout
loop: False
interval: None
dependencies: ['Discover_Tenant']
wait_for: []
preceding_task: None
options: {}
group_name: None
Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock
task will be executed on controller (localhost)
2025-06-13 20:59:36 - INFO: Interval for this Task has changed to 2m (120 seconds)
2025-06-13 20:59:36 - INFO: interval was set to Task default because it was unset, based on self.real_duration value because task is NOT allowed to stop by itself
________________________________________________________________________________
[2025-06-13 20:59:36] Deploy_or_discover_overseer step1 : Discover or deploy overseer
________________________________________________________________________________
2025-06-13 20:59:36,677 INFO
2025-06-13 20:59:36,678 INFO [loop_until]: kubectl --namespace=xlou get deployments -l "release=overseer" -o name
2025-06-13 20:59:36,678 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2025-06-13 20:59:37,024 INFO [loop_until]: OK (rc = 0)
2025-06-13 20:59:37,024 DEBUG --- stdout ---
2025-06-13 20:59:37,025 DEBUG deployment.apps/overseer-0
2025-06-13 20:59:37,025 DEBUG --- stderr ---
2025-06-13 20:59:37,025 DEBUG
Discover overseer pod: overseer-0
2025-06-13 20:59:37,187 INFO
2025-06-13 20:59:37,187 INFO --------------------- Get expected number of pods ---------------------
2025-06-13 20:59:37,187 INFO
2025-06-13 20:59:37,187 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2025-06-13 20:59:37,187 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2025-06-13 20:59:37,467 INFO [loop_until]: OK (rc = 0)
2025-06-13 20:59:37,467 DEBUG --- stdout ---
2025-06-13 20:59:37,467 DEBUG 1
2025-06-13 20:59:37,467 DEBUG --- stderr ---
2025-06-13 20:59:37,467 DEBUG
2025-06-13 20:59:37,467 INFO
2025-06-13 20:59:37,467 INFO ------------- Reloading pod list for product "overseer-0" -------------
2025-06-13 20:59:37,467 INFO
2025-06-13 20:59:37,467 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1
2025-06-13 20:59:37,467 INFO [loop_until]: (max_time=360.0, interval=10, expected_rc=[0]
2025-06-13 20:59:37,774 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2025-06-13 20:59:37,774 INFO [loop_until]: OK (rc = 0)
2025-06-13 20:59:37,774 DEBUG --- stdout ---
2025-06-13 20:59:37,774 DEBUG overseer-0-898c87646-bjhnj
2025-06-13 20:59:37,774 DEBUG --- stderr ---
2025-06-13 20:59:37,774 DEBUG
2025-06-13 20:59:37,774 INFO
2025-06-13 20:59:37,774 INFO ----------- Check pod overseer-0-898c87646-bjhnj is running -----------
2025-06-13 20:59:37,774 INFO
2025-06-13 20:59:37,774 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods overseer-0-898c87646-bjhnj -o=jsonpath={.status.phase} | grep "Running"
2025-06-13 20:59:37,774 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2025-06-13 20:59:38,082 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2025-06-13 20:59:38,082 INFO [loop_until]: OK (rc = 0)
2025-06-13 20:59:38,082 DEBUG --- stdout ---
2025-06-13 20:59:38,083 DEBUG Running
2025-06-13 20:59:38,083 DEBUG --- stderr ---
2025-06-13 20:59:38,083 DEBUG
2025-06-13 20:59:38,084 INFO
2025-06-13 20:59:38,087 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods overseer-0-898c87646-bjhnj -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2025-06-13 20:59:38,087 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2025-06-13 20:59:38,385 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2025-06-13 20:59:38,385 INFO [loop_until]: OK (rc = 0)
2025-06-13 20:59:38,385 DEBUG --- stdout ---
2025-06-13 20:59:38,385 DEBUG true
2025-06-13 20:59:38,385 DEBUG --- stderr ---
2025-06-13 20:59:38,385 DEBUG
2025-06-13 20:59:38,386 INFO
2025-06-13 20:59:38,386 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod overseer-0-898c87646-bjhnj --output jsonpath={.status.startTime}
2025-06-13 20:59:38,386 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2025-06-13 20:59:38,688 INFO [loop_until]: OK (rc = 0)
2025-06-13 20:59:38,688 DEBUG --- stdout ---
2025-06-13 20:59:38,688 DEBUG 2025-06-07T11:04:27Z
2025-06-13 20:59:38,688 DEBUG --- stderr ---
2025-06-13 20:59:38,688 DEBUG
2025-06-13 20:59:38,688 INFO
2025-06-13 20:59:38,688 INFO ---- Check pod overseer-0-898c87646-bjhnj filesystem is accessible ----
2025-06-13 20:59:39,470 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2025-06-13 20:59:39,470 INFO
2025-06-13 20:59:39,471 INFO --------- Check pod overseer-0-898c87646-bjhnj restart count ---------
2025-06-13 20:59:39,471 INFO
2025-06-13 20:59:39,471 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod overseer-0-898c87646-bjhnj --output jsonpath={.status.containerStatuses[*].restartCount}
2025-06-13 20:59:39,471 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2025-06-13 20:59:39,751 INFO [loop_until]: OK (rc = 0)
2025-06-13 20:59:39,751 DEBUG --- stdout ---
2025-06-13 20:59:39,751 DEBUG 0
2025-06-13 20:59:39,751 DEBUG --- stderr ---
2025-06-13 20:59:39,751 DEBUG
2025-06-13 20:59:39,751 INFO Pod overseer-0-898c87646-bjhnj has been restarted 0 times.
2025-06-13 20:59:39,751 INFO
2025-06-13 20:59:39,751 INFO --------------------- Get expected number of pods ---------------------
2025-06-13 20:59:39,752 INFO
2025-06-13 20:59:39,752 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2025-06-13 20:59:39,752 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2025-06-13 20:59:40,048 INFO [loop_until]: OK (rc = 0)
2025-06-13 20:59:40,048 DEBUG --- stdout ---
2025-06-13 20:59:40,048 DEBUG 1
2025-06-13 20:59:40,048 DEBUG --- stderr ---
2025-06-13 20:59:40,048 DEBUG
2025-06-13 20:59:40,048 INFO
2025-06-13 20:59:40,048 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2025-06-13 20:59:40,049 INFO
2025-06-13 20:59:40,049 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments overseer-0 --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2025-06-13 20:59:40,049 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2025-06-13 20:59:40,362 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2025-06-13 20:59:40,363 INFO [loop_until]: OK (rc = 0)
2025-06-13 20:59:40,363 DEBUG --- stdout ---
2025-06-13 20:59:40,363 DEBUG ready:1 replicas:1
2025-06-13 20:59:40,363 DEBUG --- stderr ---
2025-06-13 20:59:40,363 DEBUG
2025-06-13 20:59:40,363 INFO Livecheck to https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com
2025-06-13 20:59:40,363 INFO [http_cmd]: curl --insecure -L --request GET "https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com"
2025-06-13 20:59:40,665 INFO [http_cmd]: http status code OK
2025-06-13 20:59:40,665 DEBUG --- status code ---
2025-06-13 20:59:40,666 DEBUG http status code is 200 (expected 200)
2025-06-13 20:59:40,666 DEBUG --- http response ---
2025-06-13 20:59:40,666 DEBUG
Directory listing for /
Directory listing for /
________________________________________________________________________________
[2025-06-13 20:59:40] Deploy_or_discover_overseer post : Post method
________________________________________________________________________________
Setting result to PASS
Task has been successfully stopped