--Task-- name: restore-user-feeder enabled: True class_name: RestoreFileTask source_name: overseer-0 source_namespace: >default< target_name: idm target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: [] wait_for: [] preceding_task: None options: {'_options_': '/results/idm/user_feeder.csv'} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [2024-04-26 00:27:47] restore-user-feeder pre : detect if workload identity is enabled where the pod runs and add the necessary binding to the Google Service account (pit-test) ________________________________________________________________________________ TODO - self.option was string "/results/idm/user_feeder.csv" - that has to be refactored in datainit.py! 2024-04-26 00:27:47,691 INFO 2024-04-26 00:27:47,691 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:27:47,691 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:27:48,029 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:27:48,029 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:27:48,029 DEBUG --- stdout --- 2024-04-26 00:27:48,029 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:27:48,029 DEBUG --- stderr --- 2024-04-26 00:27:48,029 DEBUG 2024-04-26 00:27:48,030 INFO 2024-04-26 00:27:48,030 INFO [loop_until]: kubectl --namespace=xlou get pod overseer-0-8645978784-rshk9 --output jsonpath={.spec.nodeName} 2024-04-26 00:27:48,030 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-26 00:27:48,378 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:27:48,378 DEBUG --- stdout --- 2024-04-26 00:27:48,379 DEBUG gke-xlou-cdm-frontend-a8771548-i82n 2024-04-26 00:27:48,379 DEBUG --- stderr --- 2024-04-26 00:27:48,379 DEBUG 2024-04-26 00:27:48,379 INFO 2024-04-26 00:27:48,379 INFO [loop_until]: kubectl get node gke-xlou-cdm-frontend-a8771548-i82n --output jsonpath={.metadata.labels} 2024-04-26 00:27:48,379 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-26 00:27:48,803 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:27:48,803 DEBUG --- stdout --- 2024-04-26 00:27:48,803 DEBUG {"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"n2-standard-16","beta.kubernetes.io/os":"linux","cloud.google.com/gke-boot-disk":"pd-ssd","cloud.google.com/gke-container-runtime":"containerd","cloud.google.com/gke-cpu-scaling-level":"16","cloud.google.com/gke-logging-variant":"DEFAULT","cloud.google.com/gke-max-pods-per-node":"64","cloud.google.com/gke-nodepool":"frontend","cloud.google.com/gke-os-distribution":"cos","cloud.google.com/machine-family":"n2","cloud.google.com/private-node":"false","failure-domain.beta.kubernetes.io/region":"us-east1","failure-domain.beta.kubernetes.io/zone":"us-east1-d","forgerock.io/role":"frontend","frontend":"true","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"gke-xlou-cdm-frontend-a8771548-i82n","kubernetes.io/os":"linux","node.kubernetes.io/instance-type":"n2-standard-16","topology.gke.io/zone":"us-east1-d","topology.kubernetes.io/region":"us-east1","topology.kubernetes.io/zone":"us-east1-d"} 2024-04-26 00:27:48,803 DEBUG --- stderr --- 2024-04-26 00:27:48,803 DEBUG [2024-04-26 00:27:48] - INFO: Timeout for this Task has changed to 8h (28800 seconds) [2024-04-26 00:27:48] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset Pod log file : /results/orders/restore-user-feeder/stdout.txt Pod log : /results/orders/restore-user-feeder ________________________________________________________________________________ [2024-04-26 00:27:48] restore-user-feeder print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ gcloud storage cp gs://performance-bucket-us-east1/perf-tests/7.4-10M-users/user_feeder.csv /results/idm/user_feeder.csv ________________________________________________________________________________ [2024-04-26 00:27:48] restore-user-feeder create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file: /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/order.json { "name": "restore-user-feeder", "command": "gcloud storage cp gs://performance-bucket-us-east1/perf-tests/7.4-10M-users/user_feeder.csv /results/idm/user_feeder.csv", "out-dir": "/results/orders/restore-user-feeder" } ________________________________________________________________________________ [2024-04-26 00:27:48] restore-user-feeder loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2024-04-26 00:27:48,804 INFO 2024-04-26 00:27:48,804 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2024-04-26 00:27:48,804 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2024-04-26 00:27:49,102 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-26 00:27:49,102 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:27:49,102 DEBUG --- stdout --- 2024-04-26 00:27:49,102 DEBUG WAITING 2024-04-26 00:27:49,102 DEBUG --- stderr --- 2024-04-26 00:27:49,102 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 25 0 --:--:-- --:--:-- --:--:-- 25 ________________________________________________________________________________ [2024-04-26 00:27:49] restore-user-feeder start_order : Send the order to the overseer pod ________________________________________________________________________________ 2024-04-26 00:27:49,103 INFO 2024-04-26 00:27:49,103 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:27:49,103 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:27:49,405 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:27:49,405 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:27:49,405 DEBUG --- stdout --- 2024-04-26 00:27:49,405 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:27:49,405 DEBUG --- stderr --- 2024-04-26 00:27:49,405 DEBUG 2024-04-26 00:27:49,406 INFO 2024-04-26 00:27:49,406 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/order.json overseer-0-8645978784-rshk9:/results/orders/order.json 2024-04-26 00:27:49,406 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-26 00:27:50,803 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:27:50,803 DEBUG --- stdout --- 2024-04-26 00:27:50,804 DEBUG 2024-04-26 00:27:50,804 DEBUG --- stderr --- 2024-04-26 00:27:50,804 DEBUG Order has been successfully forwarded to overseer ________________________________________________________________________________ [2024-04-26 00:27:50] restore-user-feeder wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 8h Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING PASS : Order stopped by itself after 2m and before timeout (8h) ________________________________________________________________________________ [2024-04-26 00:29:51] restore-user-feeder stop_order : remove the policy binding to the GSA for the source's KSA ________________________________________________________________________________ 2024-04-26 00:29:51,163 INFO 2024-04-26 00:29:51,163 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:51,163 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:51,487 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:51,487 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:51,487 DEBUG --- stdout --- 2024-04-26 00:29:51,487 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:51,487 DEBUG --- stderr --- 2024-04-26 00:29:51,487 DEBUG 2024-04-26 00:29:51,487 INFO 2024-04-26 00:29:51,488 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-8645978784-rshk9 -- rm -f /results/orders/order.json 2024-04-26 00:29:51,488 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-26 00:29:52,286 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:52,286 DEBUG --- stdout --- 2024-04-26 00:29:52,286 DEBUG 2024-04-26 00:29:52,286 DEBUG --- stderr --- 2024-04-26 00:29:52,286 DEBUG Order ran during 2m 01s ________________________________________________________________________________ [2024-04-26 00:29:52] restore-user-feeder download_logs : Download individual order files ________________________________________________________________________________ [2024-04-26 00:29:52] - PASS: file already downloaded /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/order.json 2024-04-26 00:29:52,287 INFO 2024-04-26 00:29:52,287 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:52,287 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:52,596 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:52,597 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:52,597 DEBUG --- stdout --- 2024-04-26 00:29:52,597 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:52,597 DEBUG --- stderr --- 2024-04-26 00:29:52,597 DEBUG Checking pod: overseer-0-8645978784-rshk9 for file exists, if not wait. 2024-04-26 00:29:52,597 INFO 2024-04-26 00:29:52,597 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:52,597 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:52,915 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:52,915 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:52,915 DEBUG --- stdout --- 2024-04-26 00:29:52,915 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:52,916 DEBUG --- stderr --- 2024-04-26 00:29:52,916 DEBUG 2024-04-26 00:29:52,916 INFO 2024-04-26 00:29:52,917 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-8645978784-rshk9 --stdin -- bash -c 'if [ -f "/results/orders/restore-user-feeder/rc.txt" ]; then echo "/results/orders/restore-user-feeder/rc.txt file found"; exit 0; else echo "/results/orders/restore-user-feeder/rc.txt file not found"; exit 1; fi' | grep "/results/orders/restore-user-feeder/rc.txt file found" 2024-04-26 00:29:52,917 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-26 00:29:53,793 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-26 00:29:53,793 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:53,793 DEBUG --- stdout --- 2024-04-26 00:29:53,793 DEBUG /results/orders/restore-user-feeder/rc.txt file found 2024-04-26 00:29:53,793 DEBUG --- stderr --- 2024-04-26 00:29:53,793 DEBUG 2024-04-26 00:29:53,794 INFO 2024-04-26 00:29:53,794 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:53,794 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:54,109 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:54,110 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:54,111 DEBUG --- stdout --- 2024-04-26 00:29:54,111 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:54,111 DEBUG --- stderr --- 2024-04-26 00:29:54,111 DEBUG 2024-04-26 00:29:54,111 INFO 2024-04-26 00:29:54,111 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-8645978784-rshk9 --stdin -- du -k /results/orders/restore-user-feeder/rc.txt 2024-04-26 00:29:54,941 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-26 00:29:54,941 DEBUG --- stdout --- 2024-04-26 00:29:54,941 DEBUG 4 /results/orders/restore-user-feeder/rc.txt 2024-04-26 00:29:54,941 DEBUG --- stderr --- 2024-04-26 00:29:54,941 DEBUG 2024-04-26 00:29:54,941 INFO - Download file (size 4k) 2024-04-26 00:29:54,942 INFO 2024-04-26 00:29:54,942 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:54,942 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:55,246 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:55,247 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:55,247 DEBUG --- stdout --- 2024-04-26 00:29:55,247 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:55,247 DEBUG --- stderr --- 2024-04-26 00:29:55,247 DEBUG source : /results/orders/restore-user-feeder/rc.txt @ overseer-0-8645978784-rshk9 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/rc.txt 2024-04-26 00:29:55,247 INFO 2024-04-26 00:29:55,247 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:55,247 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:55,549 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:55,549 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:55,550 DEBUG --- stdout --- 2024-04-26 00:29:55,550 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:55,550 DEBUG --- stderr --- 2024-04-26 00:29:55,550 DEBUG 2024-04-26 00:29:55,550 INFO 2024-04-26 00:29:55,550 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-8645978784-rshk9:/results/orders/restore-user-feeder/rc.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/rc.txt 2024-04-26 00:29:55,550 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-26 00:29:56,296 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:56,297 DEBUG --- stdout --- 2024-04-26 00:29:56,297 DEBUG tar: Removing leading `/' from member names 2024-04-26 00:29:56,297 DEBUG --- stderr --- 2024-04-26 00:29:56,297 DEBUG 2024-04-26 00:29:56,297 INFO 2024-04-26 00:29:56,297 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:56,297 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:56,602 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:56,603 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:56,603 DEBUG --- stdout --- 2024-04-26 00:29:56,603 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:56,603 DEBUG --- stderr --- 2024-04-26 00:29:56,603 DEBUG Checking pod: overseer-0-8645978784-rshk9 for file exists, if not wait. 2024-04-26 00:29:56,603 INFO 2024-04-26 00:29:56,603 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:56,603 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:56,910 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:56,911 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:56,911 DEBUG --- stdout --- 2024-04-26 00:29:56,911 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:56,911 DEBUG --- stderr --- 2024-04-26 00:29:56,911 DEBUG 2024-04-26 00:29:56,911 INFO 2024-04-26 00:29:56,912 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-8645978784-rshk9 --stdin -- bash -c 'if [ -f "/results/orders/restore-user-feeder/stderr.txt" ]; then echo "/results/orders/restore-user-feeder/stderr.txt file found"; exit 0; else echo "/results/orders/restore-user-feeder/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/restore-user-feeder/stderr.txt file found" 2024-04-26 00:29:56,912 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-26 00:29:57,732 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-26 00:29:57,732 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:57,732 DEBUG --- stdout --- 2024-04-26 00:29:57,732 DEBUG /results/orders/restore-user-feeder/stderr.txt file found 2024-04-26 00:29:57,732 DEBUG --- stderr --- 2024-04-26 00:29:57,732 DEBUG 2024-04-26 00:29:57,732 INFO 2024-04-26 00:29:57,732 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:57,732 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:58,048 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:58,049 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:58,049 DEBUG --- stdout --- 2024-04-26 00:29:58,049 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:58,049 DEBUG --- stderr --- 2024-04-26 00:29:58,049 DEBUG 2024-04-26 00:29:58,049 INFO 2024-04-26 00:29:58,049 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-8645978784-rshk9 --stdin -- du -k /results/orders/restore-user-feeder/stderr.txt 2024-04-26 00:29:58,884 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-26 00:29:58,884 DEBUG --- stdout --- 2024-04-26 00:29:58,884 DEBUG 4 /results/orders/restore-user-feeder/stderr.txt 2024-04-26 00:29:58,884 DEBUG --- stderr --- 2024-04-26 00:29:58,884 DEBUG 2024-04-26 00:29:58,884 INFO - Download file (size 4k) 2024-04-26 00:29:58,884 INFO 2024-04-26 00:29:58,885 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:58,885 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:59,188 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:59,188 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:59,189 DEBUG --- stdout --- 2024-04-26 00:29:59,189 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:59,190 DEBUG --- stderr --- 2024-04-26 00:29:59,190 DEBUG source : /results/orders/restore-user-feeder/stderr.txt @ overseer-0-8645978784-rshk9 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/stderr.txt 2024-04-26 00:29:59,190 INFO 2024-04-26 00:29:59,190 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:29:59,190 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:29:59,494 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:29:59,494 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:29:59,494 DEBUG --- stdout --- 2024-04-26 00:29:59,495 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:29:59,495 DEBUG --- stderr --- 2024-04-26 00:29:59,495 DEBUG 2024-04-26 00:29:59,495 INFO 2024-04-26 00:29:59,495 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-8645978784-rshk9:/results/orders/restore-user-feeder/stderr.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/stderr.txt 2024-04-26 00:29:59,496 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-26 00:30:00,273 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:00,274 DEBUG --- stdout --- 2024-04-26 00:30:00,274 DEBUG tar: Removing leading `/' from member names 2024-04-26 00:30:00,274 DEBUG --- stderr --- 2024-04-26 00:30:00,274 DEBUG 2024-04-26 00:30:00,274 INFO 2024-04-26 00:30:00,275 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:00,275 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:00,646 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:00,646 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:00,647 DEBUG --- stdout --- 2024-04-26 00:30:00,647 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:00,647 DEBUG --- stderr --- 2024-04-26 00:30:00,647 DEBUG Checking pod: overseer-0-8645978784-rshk9 for file exists, if not wait. 2024-04-26 00:30:00,648 INFO 2024-04-26 00:30:00,648 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:00,648 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:00,983 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:00,983 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:00,983 DEBUG --- stdout --- 2024-04-26 00:30:00,983 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:00,983 DEBUG --- stderr --- 2024-04-26 00:30:00,983 DEBUG 2024-04-26 00:30:00,984 INFO 2024-04-26 00:30:00,984 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-8645978784-rshk9 --stdin -- bash -c 'if [ -f "/results/orders/restore-user-feeder/stdout.txt" ]; then echo "/results/orders/restore-user-feeder/stdout.txt file found"; exit 0; else echo "/results/orders/restore-user-feeder/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/restore-user-feeder/stdout.txt file found" 2024-04-26 00:30:00,984 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-26 00:30:01,833 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-26 00:30:01,833 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:01,833 DEBUG --- stdout --- 2024-04-26 00:30:01,834 DEBUG /results/orders/restore-user-feeder/stdout.txt file found 2024-04-26 00:30:01,834 DEBUG --- stderr --- 2024-04-26 00:30:01,834 DEBUG 2024-04-26 00:30:01,834 INFO 2024-04-26 00:30:01,834 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:01,834 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:02,186 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:02,186 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:02,186 DEBUG --- stdout --- 2024-04-26 00:30:02,186 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:02,186 DEBUG --- stderr --- 2024-04-26 00:30:02,186 DEBUG 2024-04-26 00:30:02,187 INFO 2024-04-26 00:30:02,187 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-8645978784-rshk9 --stdin -- du -k /results/orders/restore-user-feeder/stdout.txt 2024-04-26 00:30:03,014 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-26 00:30:03,014 DEBUG --- stdout --- 2024-04-26 00:30:03,014 DEBUG 0 /results/orders/restore-user-feeder/stdout.txt 2024-04-26 00:30:03,014 DEBUG --- stderr --- 2024-04-26 00:30:03,014 DEBUG 2024-04-26 00:30:03,014 INFO - Download file (size 0k) 2024-04-26 00:30:03,014 INFO 2024-04-26 00:30:03,015 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:03,015 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:03,336 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:03,336 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:03,336 DEBUG --- stdout --- 2024-04-26 00:30:03,336 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:03,336 DEBUG --- stderr --- 2024-04-26 00:30:03,336 DEBUG source : /results/orders/restore-user-feeder/stdout.txt @ overseer-0-8645978784-rshk9 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/stdout.txt 2024-04-26 00:30:03,337 INFO 2024-04-26 00:30:03,337 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:03,337 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:03,668 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:03,669 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:03,669 DEBUG --- stdout --- 2024-04-26 00:30:03,669 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:03,669 DEBUG --- stderr --- 2024-04-26 00:30:03,669 DEBUG 2024-04-26 00:30:03,669 INFO 2024-04-26 00:30:03,669 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-8645978784-rshk9:/results/orders/restore-user-feeder/stdout.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/stdout.txt 2024-04-26 00:30:03,669 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-26 00:30:04,433 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:04,434 DEBUG --- stdout --- 2024-04-26 00:30:04,434 DEBUG tar: Removing leading `/' from member names 2024-04-26 00:30:04,434 DEBUG --- stderr --- 2024-04-26 00:30:04,434 DEBUG ________________________________________________________________________________ [2024-04-26 00:30:04] restore-user-feeder show_cmd : Display order command ________________________________________________________________________________ 2024-04-26 00:30:04,435 INFO 2024-04-26 00:30:04,435 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:04,435 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:04,746 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:04,746 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:04,747 DEBUG --- stdout --- 2024-04-26 00:30:04,747 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:04,747 DEBUG --- stderr --- 2024-04-26 00:30:04,747 DEBUG Checking pod: overseer-0-8645978784-rshk9 for file exists, if not wait. 2024-04-26 00:30:04,747 INFO 2024-04-26 00:30:04,747 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:04,747 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:05,052 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:05,053 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:05,053 DEBUG --- stdout --- 2024-04-26 00:30:05,053 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:05,053 DEBUG --- stderr --- 2024-04-26 00:30:05,053 DEBUG 2024-04-26 00:30:05,053 INFO 2024-04-26 00:30:05,053 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-8645978784-rshk9 --stdin -- bash -c 'if [ -f "/results/orders/restore-user-feeder/cmd.txt" ]; then echo "/results/orders/restore-user-feeder/cmd.txt file found"; exit 0; else echo "/results/orders/restore-user-feeder/cmd.txt file not found"; exit 1; fi' | grep "/results/orders/restore-user-feeder/cmd.txt file found" 2024-04-26 00:30:05,053 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-26 00:30:05,879 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-26 00:30:05,880 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:05,880 DEBUG --- stdout --- 2024-04-26 00:30:05,880 DEBUG /results/orders/restore-user-feeder/cmd.txt file found 2024-04-26 00:30:05,880 DEBUG --- stderr --- 2024-04-26 00:30:05,880 DEBUG 2024-04-26 00:30:05,880 INFO 2024-04-26 00:30:05,880 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:05,880 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:06,215 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:06,215 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:06,216 DEBUG --- stdout --- 2024-04-26 00:30:06,216 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:06,216 DEBUG --- stderr --- 2024-04-26 00:30:06,216 DEBUG 2024-04-26 00:30:06,216 INFO 2024-04-26 00:30:06,216 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-8645978784-rshk9 --stdin -- du -k /results/orders/restore-user-feeder/cmd.txt 2024-04-26 00:30:07,084 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-26 00:30:07,085 DEBUG --- stdout --- 2024-04-26 00:30:07,085 DEBUG 4 /results/orders/restore-user-feeder/cmd.txt 2024-04-26 00:30:07,085 DEBUG --- stderr --- 2024-04-26 00:30:07,085 DEBUG 2024-04-26 00:30:07,085 INFO - Download file (size 4k) 2024-04-26 00:30:07,085 INFO 2024-04-26 00:30:07,085 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:07,085 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:07,390 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:07,390 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:07,390 DEBUG --- stdout --- 2024-04-26 00:30:07,390 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:07,390 DEBUG --- stderr --- 2024-04-26 00:30:07,390 DEBUG source : /results/orders/restore-user-feeder/cmd.txt @ overseer-0-8645978784-rshk9 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/cmd.txt 2024-04-26 00:30:07,391 INFO 2024-04-26 00:30:07,391 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-26 00:30:07,391 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-26 00:30:07,695 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-26 00:30:07,696 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:07,696 DEBUG --- stdout --- 2024-04-26 00:30:07,696 DEBUG overseer-0-8645978784-rshk9 2024-04-26 00:30:07,696 DEBUG --- stderr --- 2024-04-26 00:30:07,696 DEBUG 2024-04-26 00:30:07,696 INFO 2024-04-26 00:30:07,696 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-8645978784-rshk9:/results/orders/restore-user-feeder/cmd.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/cmd.txt 2024-04-26 00:30:07,696 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-26 00:30:08,444 INFO [loop_until]: OK (rc = 0) 2024-04-26 00:30:08,444 DEBUG --- stdout --- 2024-04-26 00:30:08,444 DEBUG tar: Removing leading `/' from member names 2024-04-26 00:30:08,444 DEBUG --- stderr --- 2024-04-26 00:30:08,444 DEBUG 2024-04-26 00:30:08,444 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/cmd.txt is small (1 lines), printing all lines: 2024-04-26 00:30:08,444 DEBUG ----- output ----- 2024-04-26 00:30:08,444 DEBUG gcloud storage cp gs://performance-bucket-us-east1/perf-tests/7.4-10M-users/user_feeder.csv /results/idm/user_feeder.csv 2024-04-26 00:30:08,444 INFO ________________________________________________________________________________ [2024-04-26 00:30:08] restore-user-feeder show_rc : Display rc order ________________________________________________________________________________ 2024-04-26 00:30:08,445 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/rc.txt is small (1 lines), printing all lines: 2024-04-26 00:30:08,445 DEBUG ----- output ----- 2024-04-26 00:30:08,445 DEBUG CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) 2024-04-26 00:30:08,445 INFO ________________________________________________________________________________ [2024-04-26 00:30:08] restore-user-feeder show_stdout : Display stdout order ________________________________________________________________________________ [2024-04-26 00:30:08] - INFO: gcloud storage outputs progress / status info to stderr by design to keep progress/status separated from data, this is why stderr_file file is displayed instead of stdout_file 2024-04-26 00:30:08,445 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/stderr.txt is small (5 lines), printing all lines: 2024-04-26 00:30:08,445 DEBUG ----- output ----- 2024-04-26 00:30:08,445 DEBUG Copying gs://performance-bucket-us-east1/perf-tests/7.4-10M-users/user_feeder.csv to file:///results/idm/user_feeder.csv ......... Average throughput: 659.7MiB/s 2024-04-26 00:30:08,445 INFO ________________________________________________________________________________ [2024-04-26 00:30:08] restore-user-feeder show_stderr : Display stderr order ________________________________________________________________________________ [2024-04-26 00:30:08] - INFO: gcloud storage outputs progress / status info to stderr by design to keep progress/status separated from data, this is why stdout_file file is displayed instead of stderr_file 2024-04-26 00:30:08,446 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/01_restore-user-feeder/stdout.txt is small (0 lines), printing all lines: 2024-04-26 00:30:08,446 DEBUG ----- output ----- 2024-04-26 00:30:08,446 DEBUG 2024-04-26 00:30:08,446 INFO ________________________________________________________________________________ [2024-04-26 00:30:08] restore-user-feeder check_rc_and_logs : Check order's outputs ________________________________________________________________________________ PASS: no error found in rc log [2024-04-26 00:30:08] - INFO: gcloud storage outputs progress / status info to stderr by design to keep progress/status separated from data, this is why stderr_file file is displayed instead of stdout_file PASS: no error found in stdout log [2024-04-26 00:30:08] - INFO: gcloud storage outputs progress / status info to stderr by design to keep progress/status separated from data, this is why stdout_file file is displayed instead of stderr_file PASS: no error found in stderr log ________________________________________________________________________________ [2024-04-26 00:30:08] restore-user-feeder print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Task result is INIT no need to display logs. ________________________________________________________________________________ [2024-04-26 00:30:08] restore-user-feeder post : Check output ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped