--Task-- name: prepare-workload enabled: True class_name: PrepareWorkloadTask source_name: overseer-0 source_namespace: >default< target_name: overseer-0 target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates below loop: False interval: None dependencies: ['cleanup'] wait_for: ['Pre-check Num Users'] preceding_task: None options: {'_options_': '--preload_output /results/idm/preload.csv --benchmark_records 1500000 --benchmark_output /results/idm/benchmark_users.csv'} group_name: None Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock ________________________________________________________________________________ [2024-10-16 19:49:13] prepare-workload pre : Checking task config ________________________________________________________________________________ 2024-10-16 19:49:13 - INFO: Timeout for this Task has changed to 2h (7200 seconds) 2024-10-16 19:49:13 - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset Pod log file : /results/orders/prepare-workload/stdout.txt Pod log : /results/orders/prepare-workload 2024-10-16 19:49:13 - INFO: --preload_records param with its value was added automatically ________________________________________________________________________________ [2024-10-16 19:49:13] prepare-workload print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ 2024-10-16 19:49:13 - INFO: --preload_records param with its value was added automatically /lodestar/pyrock/shared/scripts/idm/prepare_workload.py --preload_records 1000000 --preload_output /results/idm/preload.csv --benchmark_records 1500000 --benchmark_output /results/idm/benchmark_users.csv ________________________________________________________________________________ [2024-10-16 19:49:13] prepare-workload create_order : Create order for overseer with the command to run ________________________________________________________________________________ 2024-10-16 19:49:13 - INFO: --preload_records param with its value was added automatically Order file: /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/order.json { "name": "prepare-workload", "command": "/lodestar/pyrock/shared/scripts/idm/prepare_workload.py --preload_records 1000000 --preload_output /results/idm/preload.csv --benchmark_records 1500000 --benchmark_output /results/idm/benchmark_users.csv", "out-dir": "/results/orders/prepare-workload" } ________________________________________________________________________________ [2024-10-16 19:49:13] prepare-workload loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2024-10-16 19:49:13,467 INFO 2024-10-16 19:49:13,467 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2024-10-16 19:49:13,467 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2024-10-16 19:49:13,685 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-10-16 19:49:13,685 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:13,685 DEBUG --- stdout --- 2024-10-16 19:49:13,685 DEBUG WAITING 2024-10-16 19:49:13,685 DEBUG --- stderr --- 2024-10-16 19:49:13,685 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 33 0 --:--:-- --:--:-- --:--:-- 33 ________________________________________________________________________________ [2024-10-16 19:49:13] prepare-workload start_order : Send the order to the overseer pod ________________________________________________________________________________ 2024-10-16 19:49:13,686 INFO 2024-10-16 19:49:13,686 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:13,686 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:13,956 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:13,956 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:13,956 DEBUG --- stdout --- 2024-10-16 19:49:13,956 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:13,956 DEBUG --- stderr --- 2024-10-16 19:49:13,956 DEBUG 2024-10-16 19:49:13,956 INFO 2024-10-16 19:49:13,956 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/order.json overseer-0-68b86489-ztj5j:/results/orders/order.json 2024-10-16 19:49:13,956 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-10-16 19:49:15,277 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:15,277 DEBUG --- stdout --- 2024-10-16 19:49:15,277 DEBUG 2024-10-16 19:49:15,277 DEBUG --- stderr --- 2024-10-16 19:49:15,277 DEBUG Order has been successfully forwarded to overseer ________________________________________________________________________________ [2024-10-16 19:49:15] prepare-workload wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 2h. Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING Order is running since 5s, retry Order is running since 10s, retry Order is running since 15s, retry Order is running since 20s, retry PASS : Order stopped by itself after 26s and before timeout (2h) ________________________________________________________________________________ [2024-10-16 19:49:41] prepare-workload stop_order : Stop the order on overseer pod ________________________________________________________________________________ 2024-10-16 19:49:41,321 INFO 2024-10-16 19:49:41,321 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:41,321 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:41,590 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:41,590 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:41,590 DEBUG --- stdout --- 2024-10-16 19:49:41,590 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:41,590 DEBUG --- stderr --- 2024-10-16 19:49:41,590 DEBUG 2024-10-16 19:49:41,590 INFO 2024-10-16 19:49:41,590 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-68b86489-ztj5j -- rm -f /results/orders/order.json 2024-10-16 19:49:41,590 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-10-16 19:49:42,301 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:42,301 DEBUG --- stdout --- 2024-10-16 19:49:42,301 DEBUG 2024-10-16 19:49:42,301 DEBUG --- stderr --- 2024-10-16 19:49:42,301 DEBUG Order ran during 27s ________________________________________________________________________________ [2024-10-16 19:49:42] prepare-workload download_logs : Download individual order files ________________________________________________________________________________ 2024-10-16 19:49:42 - PASS: file already downloaded /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/order.json 2024-10-16 19:49:42,302 INFO 2024-10-16 19:49:42,302 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:42,302 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:42,574 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:42,574 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:42,574 DEBUG --- stdout --- 2024-10-16 19:49:42,574 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:42,574 DEBUG --- stderr --- 2024-10-16 19:49:42,574 DEBUG Checking pod: overseer-0-68b86489-ztj5j for file exists, if not wait. 2024-10-16 19:49:42,574 INFO 2024-10-16 19:49:42,574 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:42,574 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:42,843 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:42,843 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:42,843 DEBUG --- stdout --- 2024-10-16 19:49:42,843 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:42,843 DEBUG --- stderr --- 2024-10-16 19:49:42,844 DEBUG 2024-10-16 19:49:42,844 INFO 2024-10-16 19:49:42,844 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-68b86489-ztj5j --stdin -- bash -c 'if [ -f "/results/orders/prepare-workload/rc.txt" ]; then echo "/results/orders/prepare-workload/rc.txt file found"; exit 0; else echo "/results/orders/prepare-workload/rc.txt file not found"; exit 1; fi' | grep "/results/orders/prepare-workload/rc.txt file found" 2024-10-16 19:49:42,844 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-10-16 19:49:43,628 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-10-16 19:49:43,628 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:43,628 DEBUG --- stdout --- 2024-10-16 19:49:43,628 DEBUG /results/orders/prepare-workload/rc.txt file found 2024-10-16 19:49:43,629 DEBUG --- stderr --- 2024-10-16 19:49:43,629 DEBUG 2024-10-16 19:49:43,629 INFO 2024-10-16 19:49:43,629 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:43,629 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:43,899 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:43,899 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:43,899 DEBUG --- stdout --- 2024-10-16 19:49:43,899 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:43,899 DEBUG --- stderr --- 2024-10-16 19:49:43,899 DEBUG 2024-10-16 19:49:43,900 INFO 2024-10-16 19:49:43,900 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-68b86489-ztj5j --stdin -- du -k /results/orders/prepare-workload/rc.txt 2024-10-16 19:49:44,674 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-10-16 19:49:44,674 DEBUG --- stdout --- 2024-10-16 19:49:44,674 DEBUG 4 /results/orders/prepare-workload/rc.txt 2024-10-16 19:49:44,674 DEBUG --- stderr --- 2024-10-16 19:49:44,674 DEBUG 2024-10-16 19:49:44,674 INFO - Download file (size 4k) 2024-10-16 19:49:44,675 INFO 2024-10-16 19:49:44,675 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:44,675 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:44,945 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:44,945 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:44,945 DEBUG --- stdout --- 2024-10-16 19:49:44,945 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:44,945 DEBUG --- stderr --- 2024-10-16 19:49:44,945 DEBUG source : /results/orders/prepare-workload/rc.txt @ overseer-0-68b86489-ztj5j destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/rc.txt 2024-10-16 19:49:44,945 INFO 2024-10-16 19:49:44,945 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:44,945 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:45,216 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:45,216 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:45,216 DEBUG --- stdout --- 2024-10-16 19:49:45,216 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:45,216 DEBUG --- stderr --- 2024-10-16 19:49:45,216 DEBUG 2024-10-16 19:49:45,217 INFO 2024-10-16 19:49:45,217 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-68b86489-ztj5j:/results/orders/prepare-workload/rc.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/rc.txt 2024-10-16 19:49:45,217 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-10-16 19:49:45,956 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:45,956 DEBUG --- stdout --- 2024-10-16 19:49:45,956 DEBUG tar: Removing leading `/' from member names 2024-10-16 19:49:45,956 DEBUG --- stderr --- 2024-10-16 19:49:45,956 DEBUG 2024-10-16 19:49:45,957 INFO 2024-10-16 19:49:45,957 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:45,957 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:46,228 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:46,228 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:46,228 DEBUG --- stdout --- 2024-10-16 19:49:46,228 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:46,228 DEBUG --- stderr --- 2024-10-16 19:49:46,228 DEBUG Checking pod: overseer-0-68b86489-ztj5j for file exists, if not wait. 2024-10-16 19:49:46,228 INFO 2024-10-16 19:49:46,228 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:46,228 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:46,499 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:46,499 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:46,499 DEBUG --- stdout --- 2024-10-16 19:49:46,499 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:46,499 DEBUG --- stderr --- 2024-10-16 19:49:46,499 DEBUG 2024-10-16 19:49:46,500 INFO 2024-10-16 19:49:46,500 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-68b86489-ztj5j --stdin -- bash -c 'if [ -f "/results/orders/prepare-workload/stderr.txt" ]; then echo "/results/orders/prepare-workload/stderr.txt file found"; exit 0; else echo "/results/orders/prepare-workload/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/prepare-workload/stderr.txt file found" 2024-10-16 19:49:46,500 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-10-16 19:49:47,280 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-10-16 19:49:47,280 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:47,280 DEBUG --- stdout --- 2024-10-16 19:49:47,280 DEBUG /results/orders/prepare-workload/stderr.txt file found 2024-10-16 19:49:47,280 DEBUG --- stderr --- 2024-10-16 19:49:47,280 DEBUG 2024-10-16 19:49:47,281 INFO 2024-10-16 19:49:47,281 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:47,281 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:47,550 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:47,550 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:47,551 DEBUG --- stdout --- 2024-10-16 19:49:47,551 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:47,551 DEBUG --- stderr --- 2024-10-16 19:49:47,551 DEBUG 2024-10-16 19:49:47,551 INFO 2024-10-16 19:49:47,551 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-68b86489-ztj5j --stdin -- du -k /results/orders/prepare-workload/stderr.txt 2024-10-16 19:49:48,328 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-10-16 19:49:48,328 DEBUG --- stdout --- 2024-10-16 19:49:48,328 DEBUG 0 /results/orders/prepare-workload/stderr.txt 2024-10-16 19:49:48,328 DEBUG --- stderr --- 2024-10-16 19:49:48,328 DEBUG 2024-10-16 19:49:48,328 INFO - Download file (size 0k) 2024-10-16 19:49:48,328 INFO 2024-10-16 19:49:48,328 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:48,328 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:48,598 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:48,598 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:48,598 DEBUG --- stdout --- 2024-10-16 19:49:48,598 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:48,598 DEBUG --- stderr --- 2024-10-16 19:49:48,598 DEBUG source : /results/orders/prepare-workload/stderr.txt @ overseer-0-68b86489-ztj5j destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/stderr.txt 2024-10-16 19:49:48,598 INFO 2024-10-16 19:49:48,598 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:48,598 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:48,868 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:48,868 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:48,868 DEBUG --- stdout --- 2024-10-16 19:49:48,868 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:48,868 DEBUG --- stderr --- 2024-10-16 19:49:48,868 DEBUG 2024-10-16 19:49:48,869 INFO 2024-10-16 19:49:48,869 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-68b86489-ztj5j:/results/orders/prepare-workload/stderr.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/stderr.txt 2024-10-16 19:49:48,869 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-10-16 19:49:49,663 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:49,663 DEBUG --- stdout --- 2024-10-16 19:49:49,663 DEBUG tar: Removing leading `/' from member names 2024-10-16 19:49:49,663 DEBUG --- stderr --- 2024-10-16 19:49:49,663 DEBUG 2024-10-16 19:49:49,663 INFO 2024-10-16 19:49:49,663 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:49,663 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:49,932 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:49,932 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:49,932 DEBUG --- stdout --- 2024-10-16 19:49:49,932 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:49,932 DEBUG --- stderr --- 2024-10-16 19:49:49,932 DEBUG Checking pod: overseer-0-68b86489-ztj5j for file exists, if not wait. 2024-10-16 19:49:49,933 INFO 2024-10-16 19:49:49,933 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:49,933 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:50,203 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:50,203 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:50,204 DEBUG --- stdout --- 2024-10-16 19:49:50,204 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:50,204 DEBUG --- stderr --- 2024-10-16 19:49:50,204 DEBUG 2024-10-16 19:49:50,204 INFO 2024-10-16 19:49:50,204 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-68b86489-ztj5j --stdin -- bash -c 'if [ -f "/results/orders/prepare-workload/stdout.txt" ]; then echo "/results/orders/prepare-workload/stdout.txt file found"; exit 0; else echo "/results/orders/prepare-workload/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/prepare-workload/stdout.txt file found" 2024-10-16 19:49:50,204 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-10-16 19:49:50,979 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-10-16 19:49:50,979 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:50,979 DEBUG --- stdout --- 2024-10-16 19:49:50,979 DEBUG /results/orders/prepare-workload/stdout.txt file found 2024-10-16 19:49:50,979 DEBUG --- stderr --- 2024-10-16 19:49:50,979 DEBUG 2024-10-16 19:49:50,979 INFO 2024-10-16 19:49:50,979 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:50,979 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:51,251 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:51,251 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:51,251 DEBUG --- stdout --- 2024-10-16 19:49:51,251 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:51,251 DEBUG --- stderr --- 2024-10-16 19:49:51,251 DEBUG 2024-10-16 19:49:51,251 INFO 2024-10-16 19:49:51,251 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-68b86489-ztj5j --stdin -- du -k /results/orders/prepare-workload/stdout.txt 2024-10-16 19:49:52,023 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-10-16 19:49:52,023 DEBUG --- stdout --- 2024-10-16 19:49:52,023 DEBUG 4 /results/orders/prepare-workload/stdout.txt 2024-10-16 19:49:52,023 DEBUG --- stderr --- 2024-10-16 19:49:52,023 DEBUG 2024-10-16 19:49:52,023 INFO - Download file (size 4k) 2024-10-16 19:49:52,024 INFO 2024-10-16 19:49:52,024 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:52,024 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:52,304 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:52,304 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:52,304 DEBUG --- stdout --- 2024-10-16 19:49:52,304 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:52,305 DEBUG --- stderr --- 2024-10-16 19:49:52,305 DEBUG source : /results/orders/prepare-workload/stdout.txt @ overseer-0-68b86489-ztj5j destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/stdout.txt 2024-10-16 19:49:52,305 INFO 2024-10-16 19:49:52,305 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:52,305 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:52,577 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:52,577 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:52,577 DEBUG --- stdout --- 2024-10-16 19:49:52,577 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:52,577 DEBUG --- stderr --- 2024-10-16 19:49:52,577 DEBUG 2024-10-16 19:49:52,577 INFO 2024-10-16 19:49:52,577 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-68b86489-ztj5j:/results/orders/prepare-workload/stdout.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/stdout.txt 2024-10-16 19:49:52,577 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-10-16 19:49:53,316 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:53,316 DEBUG --- stdout --- 2024-10-16 19:49:53,316 DEBUG tar: Removing leading `/' from member names 2024-10-16 19:49:53,316 DEBUG --- stderr --- 2024-10-16 19:49:53,316 DEBUG ________________________________________________________________________________ [2024-10-16 19:49:53] prepare-workload show_cmd : Display order command ________________________________________________________________________________ 2024-10-16 19:49:53,317 INFO 2024-10-16 19:49:53,317 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:53,317 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:53,588 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:53,588 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:53,588 DEBUG --- stdout --- 2024-10-16 19:49:53,588 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:53,588 DEBUG --- stderr --- 2024-10-16 19:49:53,588 DEBUG Checking pod: overseer-0-68b86489-ztj5j for file exists, if not wait. 2024-10-16 19:49:53,588 INFO 2024-10-16 19:49:53,588 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:53,588 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:53,858 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:53,858 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:53,858 DEBUG --- stdout --- 2024-10-16 19:49:53,858 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:53,858 DEBUG --- stderr --- 2024-10-16 19:49:53,858 DEBUG 2024-10-16 19:49:53,859 INFO 2024-10-16 19:49:53,859 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-68b86489-ztj5j --stdin -- bash -c 'if [ -f "/results/orders/prepare-workload/cmd.txt" ]; then echo "/results/orders/prepare-workload/cmd.txt file found"; exit 0; else echo "/results/orders/prepare-workload/cmd.txt file not found"; exit 1; fi' | grep "/results/orders/prepare-workload/cmd.txt file found" 2024-10-16 19:49:53,859 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-10-16 19:49:54,632 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-10-16 19:49:54,632 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:54,632 DEBUG --- stdout --- 2024-10-16 19:49:54,632 DEBUG /results/orders/prepare-workload/cmd.txt file found 2024-10-16 19:49:54,633 DEBUG --- stderr --- 2024-10-16 19:49:54,633 DEBUG 2024-10-16 19:49:54,633 INFO 2024-10-16 19:49:54,633 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:54,633 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:54,903 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:54,903 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:54,903 DEBUG --- stdout --- 2024-10-16 19:49:54,903 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:54,903 DEBUG --- stderr --- 2024-10-16 19:49:54,903 DEBUG 2024-10-16 19:49:54,903 INFO 2024-10-16 19:49:54,903 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-68b86489-ztj5j --stdin -- du -k /results/orders/prepare-workload/cmd.txt 2024-10-16 19:49:55,690 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-10-16 19:49:55,690 DEBUG --- stdout --- 2024-10-16 19:49:55,690 DEBUG 4 /results/orders/prepare-workload/cmd.txt 2024-10-16 19:49:55,690 DEBUG --- stderr --- 2024-10-16 19:49:55,690 DEBUG 2024-10-16 19:49:55,690 INFO - Download file (size 4k) 2024-10-16 19:49:55,690 INFO 2024-10-16 19:49:55,690 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:55,690 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:55,960 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:55,960 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:55,960 DEBUG --- stdout --- 2024-10-16 19:49:55,960 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:55,960 DEBUG --- stderr --- 2024-10-16 19:49:55,960 DEBUG source : /results/orders/prepare-workload/cmd.txt @ overseer-0-68b86489-ztj5j destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/cmd.txt 2024-10-16 19:49:55,961 INFO 2024-10-16 19:49:55,961 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-10-16 19:49:55,961 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-10-16 19:49:56,231 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-10-16 19:49:56,231 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:56,231 DEBUG --- stdout --- 2024-10-16 19:49:56,231 DEBUG overseer-0-68b86489-ztj5j 2024-10-16 19:49:56,231 DEBUG --- stderr --- 2024-10-16 19:49:56,231 DEBUG 2024-10-16 19:49:56,232 INFO 2024-10-16 19:49:56,232 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-68b86489-ztj5j:/results/orders/prepare-workload/cmd.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/cmd.txt 2024-10-16 19:49:56,232 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-10-16 19:49:56,971 INFO [loop_until]: OK (rc = 0) 2024-10-16 19:49:56,971 DEBUG --- stdout --- 2024-10-16 19:49:56,971 DEBUG tar: Removing leading `/' from member names 2024-10-16 19:49:56,971 DEBUG --- stderr --- 2024-10-16 19:49:56,971 DEBUG 2024-10-16 19:49:56,971 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/cmd.txt is small (1 lines), printing all lines: 2024-10-16 19:49:56,971 DEBUG ----- output ----- 2024-10-16 19:49:56,971 DEBUG /lodestar/pyrock/shared/scripts/idm/prepare_workload.py --preload_records 1000000 --preload_output /results/idm/preload.csv --benchmark_records 1500000 --benchmark_output /results/idm/benchmark_users.csv 2024-10-16 19:49:56,971 INFO ________________________________________________________________________________ [2024-10-16 19:49:56] prepare-workload show_rc : Display rc order ________________________________________________________________________________ 2024-10-16 19:49:56,971 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/rc.txt is small (1 lines), printing all lines: 2024-10-16 19:49:56,971 DEBUG ----- output ----- 2024-10-16 19:49:56,971 DEBUG CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) 2024-10-16 19:49:56,971 INFO ________________________________________________________________________________ [2024-10-16 19:49:56] prepare-workload show_stdout : Display stdout order ________________________________________________________________________________ 2024-10-16 19:49:56,971 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/stdout.txt is small (2 lines), printing all lines: 2024-10-16 19:49:56,971 DEBUG ----- output ----- 2024-10-16 19:49:56,971 DEBUG [pass] File created : /results/idm/preload.csv with 1000000 records [pass] File created : /results/idm/benchmark_users.csv with 1500000 records 2024-10-16 19:49:56,971 INFO ________________________________________________________________________________ [2024-10-16 19:49:56] prepare-workload show_stderr : Display stderr order ________________________________________________________________________________ 2024-10-16 19:49:56,971 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark_ch2023/05_Data_Init/03_prepare-workload/stderr.txt is small (0 lines), printing all lines: 2024-10-16 19:49:56,971 DEBUG ----- output ----- 2024-10-16 19:49:56,972 DEBUG 2024-10-16 19:49:56,972 INFO ________________________________________________________________________________ [2024-10-16 19:49:56] prepare-workload check_rc_and_logs : Check order's outputs ________________________________________________________________________________ PASS: no error found in rc log PASS: no error found in stdout log PASS: no error found in stderr log ________________________________________________________________________________ [2024-10-16 19:49:56] prepare-workload print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Task result is INIT no need to display logs. ________________________________________________________________________________ [2024-10-16 19:49:56] prepare-workload post : Check output ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped