--Task--
name: Deploy_or_discover_overseer
enabled: True
class_name: DeployOrDiscoverOverseerTask
source_name: controller
source_namespace: >default<
target_name: controller
target_namespace: >default<
start: 0.0
stop: None
timeout: no timeout
loop: False
interval: None
dependencies: ['Tenant_discovery_and_deployment']
wait_for: []
preceding_task: None
options: {}
group_name: None
Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock
________________________________________________________________________________
[2024-11-06 15:26:39] Deploy_or_discover_overseer pre : Initialising task parameters
________________________________________________________________________________
task will be executed on controller (localhost)
________________________________________________________________________________
[2024-11-06 15:26:39] Deploy_or_discover_overseer step1 : Discover or deploy overseer
________________________________________________________________________________
2024-11-06 15:26:39,893 INFO
2024-11-06 15:26:39,893 INFO [loop_until]: kubectl create namespace xlou
2024-11-06 15:26:39,893 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-11-06 15:26:40,181 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:26:40,181 DEBUG --- stdout ---
2024-11-06 15:26:40,181 DEBUG namespace/xlou created
2024-11-06 15:26:40,181 DEBUG --- stderr ---
2024-11-06 15:26:40,181 DEBUG
2024-11-06 15:26:40,181 INFO
2024-11-06 15:26:40,181 INFO [loop_until]: kubectl label namespace xlou self-service=false timeout=48
2024-11-06 15:26:40,181 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-11-06 15:26:40,540 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:26:40,540 DEBUG --- stdout ---
2024-11-06 15:26:40,540 DEBUG namespace/xlou labeled
2024-11-06 15:26:40,540 DEBUG --- stderr ---
2024-11-06 15:26:40,540 DEBUG
2024-11-06 15:26:40,540 INFO
2024-11-06 15:26:40,540 INFO [loop_until]: kubectl --namespace=xlou get deployments -l "release=overseer" -o name
2024-11-06 15:26:40,540 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-11-06 15:26:40,834 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:26:40,834 DEBUG --- stdout ---
2024-11-06 15:26:40,834 DEBUG
2024-11-06 15:26:40,834 DEBUG --- stderr ---
2024-11-06 15:26:40,834 DEBUG
overseer-0 pod does not exit, deploy it
2024-11-06 15:26:41,023 INFO
2024-11-06 15:26:41,023 INFO ------------ Deploy the lodestarbox with overseer profile ------------
2024-11-06 15:26:41,023 INFO
2024-11-06 15:26:41,023 INFO [run_command]: skaffold deploy --profile overseer --config=/tmp/tmp99umbkzu --status-check=true --namespace=xlou
2024-11-06 15:26:43,130 INFO Starting deploy...
2024-11-06 15:26:45,370 INFO - serviceaccount/k8s-svc-acct-0 created
2024-11-06 15:26:45,562 INFO - clusterrolebinding.rbac.authorization.k8s.io/k8s-svc-acct-crb-xlou-0 created
2024-11-06 15:26:45,795 INFO - configmap/overseer-config-0 created
2024-11-06 15:26:46,047 INFO - service/overseer-0 created
2024-11-06 15:26:46,280 INFO - persistentvolumeclaim/overseer-0 created
2024-11-06 15:26:46,523 INFO - deployment.apps/overseer-0 created
2024-11-06 15:26:46,907 INFO - Warning: annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
2024-11-06 15:26:46,917 INFO - ingress.networking.k8s.io/overseer-0 created
2024-11-06 15:26:46,928 INFO Waiting for deployments to stabilize...
2024-11-06 15:26:52,164 INFO - xlou:deployment/overseer-0: creating container overseer
2024-11-06 15:26:52,174 INFO - xlou:pod/overseer-0-6644bb6697-9gfkx: creating container overseer
2024-11-06 15:27:21,577 INFO - xlou:deployment/overseer-0 is ready.
2024-11-06 15:27:21,588 INFO Deployments stabilized in 34.67 seconds
2024-11-06 15:27:21,680 ERROR There is a new version (2.13.2) of Skaffold available. Download it from:
2024-11-06 15:27:21,680 ERROR https://github.com/GoogleContainerTools/skaffold/releases/tag/v2.13.2
2024-11-06 15:27:21,680 ERROR
2024-11-06 15:27:21,680 ERROR Help improve Skaffold with our 2-minute anonymous survey: run 'skaffold survey'
2024-11-06 15:27:21,680 ERROR To help improve the quality of this product, we collect anonymized usage data for details on what is tracked and how we use this data visit . This data is handled in accordance with our privacy policy
2024-11-06 15:27:21,680 ERROR
2024-11-06 15:27:21,680 ERROR You may choose to opt out of this collection by running the following command:
2024-11-06 15:27:21,680 ERROR skaffold config set --global collect-metrics false
2024-11-06 15:27:21,680 INFO
2024-11-06 15:27:21,681 INFO --------------------- Get expected number of pods ---------------------
2024-11-06 15:27:21,681 INFO
2024-11-06 15:27:21,681 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-11-06 15:27:21,681 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-11-06 15:27:21,955 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:21,955 DEBUG --- stdout ---
2024-11-06 15:27:21,955 DEBUG 1
2024-11-06 15:27:21,955 DEBUG --- stderr ---
2024-11-06 15:27:21,955 DEBUG
2024-11-06 15:27:21,955 INFO
2024-11-06 15:27:21,955 INFO ------------- Reloading pod list for product "overseer-0" -------------
2024-11-06 15:27:21,955 INFO
2024-11-06 15:27:21,955 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1
2024-11-06 15:27:21,955 INFO [loop_until]: (max_time=360.0, interval=10, expected_rc=[0]
2024-11-06 15:27:22,230 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-11-06 15:27:22,230 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:22,230 DEBUG --- stdout ---
2024-11-06 15:27:22,230 DEBUG overseer-0-6644bb6697-9gfkx
2024-11-06 15:27:22,230 DEBUG --- stderr ---
2024-11-06 15:27:22,230 DEBUG
2024-11-06 15:27:22,230 INFO
2024-11-06 15:27:22,230 INFO ---------- Check pod overseer-0-6644bb6697-9gfkx is running ----------
2024-11-06 15:27:22,230 INFO
2024-11-06 15:27:22,230 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods overseer-0-6644bb6697-9gfkx -o=jsonpath={.status.phase} | grep "Running"
2024-11-06 15:27:22,230 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-11-06 15:27:22,528 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-11-06 15:27:22,528 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:22,528 DEBUG --- stdout ---
2024-11-06 15:27:22,528 DEBUG Running
2024-11-06 15:27:22,528 DEBUG --- stderr ---
2024-11-06 15:27:22,528 DEBUG
2024-11-06 15:27:22,529 INFO
2024-11-06 15:27:22,529 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods overseer-0-6644bb6697-9gfkx -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
2024-11-06 15:27:22,529 INFO [loop_until]: (max_time=360, interval=5, expected_rc=[0]
2024-11-06 15:27:22,801 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-11-06 15:27:22,801 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:22,801 DEBUG --- stdout ---
2024-11-06 15:27:22,801 DEBUG true
2024-11-06 15:27:22,801 DEBUG --- stderr ---
2024-11-06 15:27:22,801 DEBUG
2024-11-06 15:27:22,801 INFO
2024-11-06 15:27:22,801 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod overseer-0-6644bb6697-9gfkx --output jsonpath={.status.startTime}
2024-11-06 15:27:22,801 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-11-06 15:27:23,079 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:23,079 DEBUG --- stdout ---
2024-11-06 15:27:23,079 DEBUG 2024-11-06T15:26:50Z
2024-11-06 15:27:23,079 DEBUG --- stderr ---
2024-11-06 15:27:23,079 DEBUG
2024-11-06 15:27:23,080 INFO
2024-11-06 15:27:23,080 INFO --- Check pod overseer-0-6644bb6697-9gfkx filesystem is accessible ---
2024-11-06 15:27:23,838 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-11-06 15:27:23,838 INFO
2024-11-06 15:27:23,838 INFO --------- Check pod overseer-0-6644bb6697-9gfkx restart count ---------
2024-11-06 15:27:23,839 INFO
2024-11-06 15:27:23,839 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod overseer-0-6644bb6697-9gfkx --output jsonpath={.status.containerStatuses[*].restartCount}
2024-11-06 15:27:23,839 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-11-06 15:27:24,112 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:24,112 DEBUG --- stdout ---
2024-11-06 15:27:24,112 DEBUG 0
2024-11-06 15:27:24,112 DEBUG --- stderr ---
2024-11-06 15:27:24,112 DEBUG
2024-11-06 15:27:24,112 INFO Pod overseer-0-6644bb6697-9gfkx has been restarted 0 times.
2024-11-06 15:27:24,112 INFO
2024-11-06 15:27:24,112 INFO --------------------- Get expected number of pods ---------------------
2024-11-06 15:27:24,113 INFO
2024-11-06 15:27:24,113 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-11-06 15:27:24,113 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-11-06 15:27:24,390 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:24,390 DEBUG --- stdout ---
2024-11-06 15:27:24,390 DEBUG 1
2024-11-06 15:27:24,390 DEBUG --- stderr ---
2024-11-06 15:27:24,390 DEBUG
2024-11-06 15:27:24,390 INFO
2024-11-06 15:27:24,390 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-11-06 15:27:24,391 INFO
2024-11-06 15:27:24,391 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments overseer-0 --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-11-06 15:27:24,391 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-11-06 15:27:24,663 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-11-06 15:27:24,663 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:24,663 DEBUG --- stdout ---
2024-11-06 15:27:24,663 DEBUG ready:1 replicas:1
2024-11-06 15:27:24,663 DEBUG --- stderr ---
2024-11-06 15:27:24,663 DEBUG
2024-11-06 15:27:24,663 INFO Livecheck to https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com
2024-11-06 15:27:24,663 INFO [http_cmd]: curl --insecure -L --request GET "https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com"
2024-11-06 15:27:25,003 INFO [http_cmd]: http status code OK
2024-11-06 15:27:25,003 DEBUG --- status code ---
2024-11-06 15:27:25,003 DEBUG http status code is 200 (expected 200)
2024-11-06 15:27:25,003 DEBUG --- http response ---
2024-11-06 15:27:25,003 DEBUG
Directory listing for /
Directory listing for /
2024-11-06 15:27:25,004 INFO Component overseer-0 is alive
2024-11-06 15:27:25,799 INFO Dumping pod description and logs to /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/pod-logs/20241106_152725-after-overseer-0-deploy/overseer-0-6644bb6697-9gfkx.txt
2024-11-06 15:27:25,800 INFO Dumping pod description and logs to /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/pod-logs/20241106_152725-after-overseer-0-deploy/overseer-0-6644bb6697-9gfkx.txt
2024-11-06 15:27:25,800 INFO Check pod logs for errors
2024-11-06 15:27:25,800 INFO
2024-11-06 15:27:25,800 INFO --------------------- Get expected number of pods ---------------------
2024-11-06 15:27:25,800 INFO
2024-11-06 15:27:25,800 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-11-06 15:27:25,800 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-11-06 15:27:26,073 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:26,073 DEBUG --- stdout ---
2024-11-06 15:27:26,074 DEBUG 1
2024-11-06 15:27:26,074 DEBUG --- stderr ---
2024-11-06 15:27:26,074 DEBUG
2024-11-06 15:27:26,074 INFO
2024-11-06 15:27:26,074 INFO ------------- Reloading pod list for product "overseer-0" -------------
2024-11-06 15:27:26,074 INFO
2024-11-06 15:27:26,074 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1
2024-11-06 15:27:26,074 INFO [loop_until]: (max_time=360.0, interval=10, expected_rc=[0]
2024-11-06 15:27:26,351 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
2024-11-06 15:27:26,351 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:26,351 DEBUG --- stdout ---
2024-11-06 15:27:26,351 DEBUG overseer-0-6644bb6697-9gfkx
2024-11-06 15:27:26,351 DEBUG --- stderr ---
2024-11-06 15:27:26,351 DEBUG
2024-11-06 15:27:26,352 INFO
2024-11-06 15:27:26,352 INFO --------------------- Get expected number of pods ---------------------
2024-11-06 15:27:26,352 INFO
2024-11-06 15:27:26,352 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=overseer-0 --output jsonpath={.items[*].spec.replicas}
2024-11-06 15:27:26,352 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2024-11-06 15:27:26,627 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:26,627 DEBUG --- stdout ---
2024-11-06 15:27:26,628 DEBUG 1
2024-11-06 15:27:26,628 DEBUG --- stderr ---
2024-11-06 15:27:26,628 DEBUG
2024-11-06 15:27:26,628 INFO
2024-11-06 15:27:26,628 INFO -------------- Waiting for 1 expected pod(s) to be ready --------------
2024-11-06 15:27:26,628 INFO
2024-11-06 15:27:26,628 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments overseer-0 --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
2024-11-06 15:27:26,628 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0]
2024-11-06 15:27:26,901 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
2024-11-06 15:27:26,901 INFO [loop_until]: OK (rc = 0)
2024-11-06 15:27:26,901 DEBUG --- stdout ---
2024-11-06 15:27:26,901 DEBUG ready:1 replicas:1
2024-11-06 15:27:26,901 DEBUG --- stderr ---
2024-11-06 15:27:26,901 DEBUG
2024-11-06 15:27:26,901 INFO Livecheck to https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com
2024-11-06 15:27:26,902 INFO [http_cmd]: curl --insecure -L --request GET "https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com"
2024-11-06 15:27:27,109 INFO [http_cmd]: http status code OK
2024-11-06 15:27:27,109 DEBUG --- status code ---
2024-11-06 15:27:27,109 DEBUG http status code is 200 (expected 200)
2024-11-06 15:27:27,109 DEBUG --- http response ---
2024-11-06 15:27:27,109 DEBUG
Directory listing for /
Directory listing for /
________________________________________________________________________________
[2024-11-06 15:27:27] Deploy_or_discover_overseer post : Post method
________________________________________________________________________________
Setting result to PASS
Task has been successfully stopped