--Task-- name: prepare-workload enabled: True class_name: PrepareWorkloadTask source_name: overseer-0 source_namespace: >default< target_name: overseer-0 target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: ['cleanup'] wait_for: ['Pre-check Num Users'] preceding_task: None options: {'_options_': '--preload_output /results/idm/preload.csv --benchmark_records 1500000 --benchmark_output /results/idm/benchmark_users.csv'} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [2024-05-24 18:09:01] prepare-workload pre : Checking task config ________________________________________________________________________________ [2024-05-24 18:09:01] - INFO: Timeout for this Task has changed to 16h (57600 seconds) [2024-05-24 18:09:01] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset Pod log file : /results/orders/prepare-workload/stdout.txt Pod log : /results/orders/prepare-workload [2024-05-24 18:09:01] - INFO: --preload_records param with its value was added automatically ________________________________________________________________________________ [2024-05-24 18:09:01] prepare-workload print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ [2024-05-24 18:09:01] - INFO: --preload_records param with its value was added automatically /lodestar/pyrock/shared/scripts/idm/prepare_workload.py --preload_records 1000000 --preload_output /results/idm/preload.csv --benchmark_records 1500000 --benchmark_output /results/idm/benchmark_users.csv ________________________________________________________________________________ [2024-05-24 18:09:01] prepare-workload create_order : Create order for overseer with the command to run ________________________________________________________________________________ [2024-05-24 18:09:01] - INFO: --preload_records param with its value was added automatically Order file: /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/order.json { "name": "prepare-workload", "command": "/lodestar/pyrock/shared/scripts/idm/prepare_workload.py --preload_records 1000000 --preload_output /results/idm/preload.csv --benchmark_records 1500000 --benchmark_output /results/idm/benchmark_users.csv", "out-dir": "/results/orders/prepare-workload" } ________________________________________________________________________________ [2024-05-24 18:09:01] prepare-workload loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2024-05-24 18:09:01,107 INFO 2024-05-24 18:09:01,107 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2024-05-24 18:09:01,107 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2024-05-24 18:09:01,361 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-05-24 18:09:01,361 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:09:01,361 DEBUG --- stdout --- 2024-05-24 18:09:01,361 DEBUG WAITING 2024-05-24 18:09:01,361 DEBUG --- stderr --- 2024-05-24 18:09:01,361 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 32 0 --:--:-- --:--:-- --:--:-- 32 ________________________________________________________________________________ [2024-05-24 18:09:01] prepare-workload start_order : Send the order to the overseer pod ________________________________________________________________________________ 2024-05-24 18:09:01,362 INFO 2024-05-24 18:09:01,362 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:09:01,362 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:09:01,685 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:09:01,685 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:09:01,685 DEBUG --- stdout --- 2024-05-24 18:09:01,685 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:09:01,685 DEBUG --- stderr --- 2024-05-24 18:09:01,686 DEBUG 2024-05-24 18:09:01,686 INFO 2024-05-24 18:09:01,686 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/order.json overseer-0-dfd5985cb-fzzf7:/results/orders/order.json 2024-05-24 18:09:01,686 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-05-24 18:09:03,042 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:09:03,042 DEBUG --- stdout --- 2024-05-24 18:09:03,042 DEBUG 2024-05-24 18:09:03,042 DEBUG --- stderr --- 2024-05-24 18:09:03,042 DEBUG Order has been successfully forwarded to overseer ________________________________________________________________________________ [2024-05-24 18:09:03] prepare-workload wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 16h Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING PASS : Order stopped by itself after 2m and before timeout (16h) ________________________________________________________________________________ [2024-05-24 18:11:03] prepare-workload stop_order : Stop the order on overseer pod ________________________________________________________________________________ 2024-05-24 18:11:03,403 INFO 2024-05-24 18:11:03,403 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:03,403 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:03,765 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:03,765 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:03,765 DEBUG --- stdout --- 2024-05-24 18:11:03,765 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:03,765 DEBUG --- stderr --- 2024-05-24 18:11:03,765 DEBUG 2024-05-24 18:11:03,765 INFO 2024-05-24 18:11:03,765 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-dfd5985cb-fzzf7 -- rm -f /results/orders/order.json 2024-05-24 18:11:03,765 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-05-24 18:11:04,571 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:04,571 DEBUG --- stdout --- 2024-05-24 18:11:04,571 DEBUG 2024-05-24 18:11:04,571 DEBUG --- stderr --- 2024-05-24 18:11:04,571 DEBUG Order ran during 2m 01s ________________________________________________________________________________ [2024-05-24 18:11:04] prepare-workload download_logs : Download individual order files ________________________________________________________________________________ [2024-05-24 18:11:04] - PASS: file already downloaded /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/order.json 2024-05-24 18:11:04,572 INFO 2024-05-24 18:11:04,572 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:04,572 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:04,884 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:04,884 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:04,884 DEBUG --- stdout --- 2024-05-24 18:11:04,884 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:04,884 DEBUG --- stderr --- 2024-05-24 18:11:04,884 DEBUG Checking pod: overseer-0-dfd5985cb-fzzf7 for file exists, if not wait. 2024-05-24 18:11:04,884 INFO 2024-05-24 18:11:04,884 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:04,884 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:05,194 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:05,195 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:05,195 DEBUG --- stdout --- 2024-05-24 18:11:05,195 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:05,195 DEBUG --- stderr --- 2024-05-24 18:11:05,195 DEBUG 2024-05-24 18:11:05,195 INFO 2024-05-24 18:11:05,195 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-dfd5985cb-fzzf7 --stdin -- bash -c 'if [ -f "/results/orders/prepare-workload/rc.txt" ]; then echo "/results/orders/prepare-workload/rc.txt file found"; exit 0; else echo "/results/orders/prepare-workload/rc.txt file not found"; exit 1; fi' | grep "/results/orders/prepare-workload/rc.txt file found" 2024-05-24 18:11:05,195 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-05-24 18:11:05,998 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-05-24 18:11:05,998 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:05,998 DEBUG --- stdout --- 2024-05-24 18:11:05,998 DEBUG /results/orders/prepare-workload/rc.txt file found 2024-05-24 18:11:05,998 DEBUG --- stderr --- 2024-05-24 18:11:05,998 DEBUG 2024-05-24 18:11:05,999 INFO 2024-05-24 18:11:05,999 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:05,999 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:06,309 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:06,309 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:06,309 DEBUG --- stdout --- 2024-05-24 18:11:06,309 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:06,309 DEBUG --- stderr --- 2024-05-24 18:11:06,309 DEBUG 2024-05-24 18:11:06,309 INFO 2024-05-24 18:11:06,309 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-dfd5985cb-fzzf7 --stdin -- du -k /results/orders/prepare-workload/rc.txt 2024-05-24 18:11:07,121 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-05-24 18:11:07,121 DEBUG --- stdout --- 2024-05-24 18:11:07,121 DEBUG 4 /results/orders/prepare-workload/rc.txt 2024-05-24 18:11:07,121 DEBUG --- stderr --- 2024-05-24 18:11:07,121 DEBUG 2024-05-24 18:11:07,121 INFO - Download file (size 4k) 2024-05-24 18:11:07,122 INFO 2024-05-24 18:11:07,122 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:07,122 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:07,454 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:07,454 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:07,454 DEBUG --- stdout --- 2024-05-24 18:11:07,454 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:07,454 DEBUG --- stderr --- 2024-05-24 18:11:07,454 DEBUG source : /results/orders/prepare-workload/rc.txt @ overseer-0-dfd5985cb-fzzf7 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/rc.txt 2024-05-24 18:11:07,455 INFO 2024-05-24 18:11:07,455 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:07,455 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:07,774 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:07,774 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:07,774 DEBUG --- stdout --- 2024-05-24 18:11:07,774 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:07,774 DEBUG --- stderr --- 2024-05-24 18:11:07,774 DEBUG 2024-05-24 18:11:07,775 INFO 2024-05-24 18:11:07,775 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-dfd5985cb-fzzf7:/results/orders/prepare-workload/rc.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/rc.txt 2024-05-24 18:11:07,775 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-05-24 18:11:08,566 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:08,566 DEBUG --- stdout --- 2024-05-24 18:11:08,566 DEBUG tar: Removing leading `/' from member names 2024-05-24 18:11:08,566 DEBUG --- stderr --- 2024-05-24 18:11:08,566 DEBUG 2024-05-24 18:11:08,567 INFO 2024-05-24 18:11:08,567 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:08,567 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:08,878 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:08,879 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:08,879 DEBUG --- stdout --- 2024-05-24 18:11:08,879 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:08,879 DEBUG --- stderr --- 2024-05-24 18:11:08,879 DEBUG Checking pod: overseer-0-dfd5985cb-fzzf7 for file exists, if not wait. 2024-05-24 18:11:08,879 INFO 2024-05-24 18:11:08,879 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:08,879 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:09,189 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:09,190 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:09,190 DEBUG --- stdout --- 2024-05-24 18:11:09,190 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:09,190 DEBUG --- stderr --- 2024-05-24 18:11:09,190 DEBUG 2024-05-24 18:11:09,190 INFO 2024-05-24 18:11:09,190 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-dfd5985cb-fzzf7 --stdin -- bash -c 'if [ -f "/results/orders/prepare-workload/stderr.txt" ]; then echo "/results/orders/prepare-workload/stderr.txt file found"; exit 0; else echo "/results/orders/prepare-workload/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/prepare-workload/stderr.txt file found" 2024-05-24 18:11:09,190 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-05-24 18:11:09,994 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-05-24 18:11:09,994 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:09,994 DEBUG --- stdout --- 2024-05-24 18:11:09,994 DEBUG /results/orders/prepare-workload/stderr.txt file found 2024-05-24 18:11:09,994 DEBUG --- stderr --- 2024-05-24 18:11:09,994 DEBUG 2024-05-24 18:11:09,995 INFO 2024-05-24 18:11:09,995 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:09,995 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:10,305 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:10,306 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:10,306 DEBUG --- stdout --- 2024-05-24 18:11:10,306 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:10,306 DEBUG --- stderr --- 2024-05-24 18:11:10,306 DEBUG 2024-05-24 18:11:10,306 INFO 2024-05-24 18:11:10,306 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-dfd5985cb-fzzf7 --stdin -- du -k /results/orders/prepare-workload/stderr.txt 2024-05-24 18:11:11,114 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-05-24 18:11:11,115 DEBUG --- stdout --- 2024-05-24 18:11:11,115 DEBUG 0 /results/orders/prepare-workload/stderr.txt 2024-05-24 18:11:11,115 DEBUG --- stderr --- 2024-05-24 18:11:11,115 DEBUG 2024-05-24 18:11:11,115 INFO - Download file (size 0k) 2024-05-24 18:11:11,115 INFO 2024-05-24 18:11:11,115 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:11,115 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:11,421 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:11,421 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:11,421 DEBUG --- stdout --- 2024-05-24 18:11:11,421 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:11,421 DEBUG --- stderr --- 2024-05-24 18:11:11,421 DEBUG source : /results/orders/prepare-workload/stderr.txt @ overseer-0-dfd5985cb-fzzf7 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/stderr.txt 2024-05-24 18:11:11,421 INFO 2024-05-24 18:11:11,422 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:11,422 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:11,730 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:11,730 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:11,730 DEBUG --- stdout --- 2024-05-24 18:11:11,730 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:11,731 DEBUG --- stderr --- 2024-05-24 18:11:11,731 DEBUG 2024-05-24 18:11:11,731 INFO 2024-05-24 18:11:11,731 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-dfd5985cb-fzzf7:/results/orders/prepare-workload/stderr.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/stderr.txt 2024-05-24 18:11:11,731 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-05-24 18:11:12,513 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:12,513 DEBUG --- stdout --- 2024-05-24 18:11:12,513 DEBUG tar: Removing leading `/' from member names 2024-05-24 18:11:12,513 DEBUG --- stderr --- 2024-05-24 18:11:12,513 DEBUG 2024-05-24 18:11:12,514 INFO 2024-05-24 18:11:12,514 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:12,514 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:12,820 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:12,821 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:12,821 DEBUG --- stdout --- 2024-05-24 18:11:12,821 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:12,821 DEBUG --- stderr --- 2024-05-24 18:11:12,821 DEBUG Checking pod: overseer-0-dfd5985cb-fzzf7 for file exists, if not wait. 2024-05-24 18:11:12,821 INFO 2024-05-24 18:11:12,821 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:12,821 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:13,125 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:13,126 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:13,126 DEBUG --- stdout --- 2024-05-24 18:11:13,126 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:13,126 DEBUG --- stderr --- 2024-05-24 18:11:13,126 DEBUG 2024-05-24 18:11:13,126 INFO 2024-05-24 18:11:13,126 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-dfd5985cb-fzzf7 --stdin -- bash -c 'if [ -f "/results/orders/prepare-workload/stdout.txt" ]; then echo "/results/orders/prepare-workload/stdout.txt file found"; exit 0; else echo "/results/orders/prepare-workload/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/prepare-workload/stdout.txt file found" 2024-05-24 18:11:13,126 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-05-24 18:11:13,921 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-05-24 18:11:13,923 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:13,923 DEBUG --- stdout --- 2024-05-24 18:11:13,923 DEBUG /results/orders/prepare-workload/stdout.txt file found 2024-05-24 18:11:13,923 DEBUG --- stderr --- 2024-05-24 18:11:13,923 DEBUG 2024-05-24 18:11:13,924 INFO 2024-05-24 18:11:13,924 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:13,924 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:14,251 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:14,251 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:14,251 DEBUG --- stdout --- 2024-05-24 18:11:14,252 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:14,252 DEBUG --- stderr --- 2024-05-24 18:11:14,252 DEBUG 2024-05-24 18:11:14,252 INFO 2024-05-24 18:11:14,252 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-dfd5985cb-fzzf7 --stdin -- du -k /results/orders/prepare-workload/stdout.txt 2024-05-24 18:11:15,053 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-05-24 18:11:15,053 DEBUG --- stdout --- 2024-05-24 18:11:15,053 DEBUG 4 /results/orders/prepare-workload/stdout.txt 2024-05-24 18:11:15,053 DEBUG --- stderr --- 2024-05-24 18:11:15,053 DEBUG 2024-05-24 18:11:15,053 INFO - Download file (size 4k) 2024-05-24 18:11:15,054 INFO 2024-05-24 18:11:15,054 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:15,054 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:15,366 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:15,367 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:15,367 DEBUG --- stdout --- 2024-05-24 18:11:15,367 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:15,367 DEBUG --- stderr --- 2024-05-24 18:11:15,367 DEBUG source : /results/orders/prepare-workload/stdout.txt @ overseer-0-dfd5985cb-fzzf7 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/stdout.txt 2024-05-24 18:11:15,367 INFO 2024-05-24 18:11:15,367 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:15,367 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:15,686 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:15,686 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:15,686 DEBUG --- stdout --- 2024-05-24 18:11:15,686 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:15,686 DEBUG --- stderr --- 2024-05-24 18:11:15,686 DEBUG 2024-05-24 18:11:15,687 INFO 2024-05-24 18:11:15,687 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-dfd5985cb-fzzf7:/results/orders/prepare-workload/stdout.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/stdout.txt 2024-05-24 18:11:15,687 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-05-24 18:11:16,416 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:16,416 DEBUG --- stdout --- 2024-05-24 18:11:16,416 DEBUG tar: Removing leading `/' from member names 2024-05-24 18:11:16,416 DEBUG --- stderr --- 2024-05-24 18:11:16,416 DEBUG ________________________________________________________________________________ [2024-05-24 18:11:16] prepare-workload show_cmd : Display order command ________________________________________________________________________________ 2024-05-24 18:11:16,417 INFO 2024-05-24 18:11:16,417 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:16,417 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:16,724 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:16,724 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:16,725 DEBUG --- stdout --- 2024-05-24 18:11:16,725 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:16,725 DEBUG --- stderr --- 2024-05-24 18:11:16,725 DEBUG Checking pod: overseer-0-dfd5985cb-fzzf7 for file exists, if not wait. 2024-05-24 18:11:16,725 INFO 2024-05-24 18:11:16,725 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:16,725 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:17,036 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:17,036 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:17,037 DEBUG --- stdout --- 2024-05-24 18:11:17,037 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:17,037 DEBUG --- stderr --- 2024-05-24 18:11:17,037 DEBUG 2024-05-24 18:11:17,037 INFO 2024-05-24 18:11:17,037 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-dfd5985cb-fzzf7 --stdin -- bash -c 'if [ -f "/results/orders/prepare-workload/cmd.txt" ]; then echo "/results/orders/prepare-workload/cmd.txt file found"; exit 0; else echo "/results/orders/prepare-workload/cmd.txt file not found"; exit 1; fi' | grep "/results/orders/prepare-workload/cmd.txt file found" 2024-05-24 18:11:17,037 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-05-24 18:11:17,849 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-05-24 18:11:17,849 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:17,849 DEBUG --- stdout --- 2024-05-24 18:11:17,849 DEBUG /results/orders/prepare-workload/cmd.txt file found 2024-05-24 18:11:17,849 DEBUG --- stderr --- 2024-05-24 18:11:17,849 DEBUG 2024-05-24 18:11:17,850 INFO 2024-05-24 18:11:17,850 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:17,850 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:18,169 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:18,169 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:18,169 DEBUG --- stdout --- 2024-05-24 18:11:18,169 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:18,169 DEBUG --- stderr --- 2024-05-24 18:11:18,169 DEBUG 2024-05-24 18:11:18,169 INFO 2024-05-24 18:11:18,170 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-dfd5985cb-fzzf7 --stdin -- du -k /results/orders/prepare-workload/cmd.txt 2024-05-24 18:11:18,984 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-05-24 18:11:18,986 DEBUG --- stdout --- 2024-05-24 18:11:18,986 DEBUG 4 /results/orders/prepare-workload/cmd.txt 2024-05-24 18:11:18,987 DEBUG --- stderr --- 2024-05-24 18:11:18,987 DEBUG 2024-05-24 18:11:18,987 INFO - Download file (size 4k) 2024-05-24 18:11:18,987 INFO 2024-05-24 18:11:18,987 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:18,987 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:19,290 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:19,290 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:19,290 DEBUG --- stdout --- 2024-05-24 18:11:19,290 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:19,290 DEBUG --- stderr --- 2024-05-24 18:11:19,290 DEBUG source : /results/orders/prepare-workload/cmd.txt @ overseer-0-dfd5985cb-fzzf7 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/cmd.txt 2024-05-24 18:11:19,291 INFO 2024-05-24 18:11:19,291 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-05-24 18:11:19,291 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-05-24 18:11:19,607 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-05-24 18:11:19,608 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:19,608 DEBUG --- stdout --- 2024-05-24 18:11:19,608 DEBUG overseer-0-dfd5985cb-fzzf7 2024-05-24 18:11:19,608 DEBUG --- stderr --- 2024-05-24 18:11:19,608 DEBUG 2024-05-24 18:11:19,608 INFO 2024-05-24 18:11:19,608 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-dfd5985cb-fzzf7:/results/orders/prepare-workload/cmd.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/cmd.txt 2024-05-24 18:11:19,608 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-05-24 18:11:20,389 INFO [loop_until]: OK (rc = 0) 2024-05-24 18:11:20,389 DEBUG --- stdout --- 2024-05-24 18:11:20,389 DEBUG tar: Removing leading `/' from member names 2024-05-24 18:11:20,389 DEBUG --- stderr --- 2024-05-24 18:11:20,389 DEBUG 2024-05-24 18:11:20,390 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/cmd.txt is small (1 lines), printing all lines: 2024-05-24 18:11:20,390 DEBUG ----- output ----- 2024-05-24 18:11:20,390 DEBUG /lodestar/pyrock/shared/scripts/idm/prepare_workload.py --preload_records 1000000 --preload_output /results/idm/preload.csv --benchmark_records 1500000 --benchmark_output /results/idm/benchmark_users.csv 2024-05-24 18:11:20,390 INFO ________________________________________________________________________________ [2024-05-24 18:11:20] prepare-workload show_rc : Display rc order ________________________________________________________________________________ 2024-05-24 18:11:20,390 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/rc.txt is small (1 lines), printing all lines: 2024-05-24 18:11:20,390 DEBUG ----- output ----- 2024-05-24 18:11:20,390 DEBUG CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) 2024-05-24 18:11:20,390 INFO ________________________________________________________________________________ [2024-05-24 18:11:20] prepare-workload show_stdout : Display stdout order ________________________________________________________________________________ 2024-05-24 18:11:20,391 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/stdout.txt is small (2 lines), printing all lines: 2024-05-24 18:11:20,391 DEBUG ----- output ----- 2024-05-24 18:11:20,391 DEBUG [pass] File created : /results/idm/preload.csv with 1000000 records [pass] File created : /results/idm/benchmark_users.csv with 1500000 records 2024-05-24 18:11:20,391 INFO ________________________________________________________________________________ [2024-05-24 18:11:20] prepare-workload show_stderr : Display stderr order ________________________________________________________________________________ 2024-05-24 18:11:20,391 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/03_prepare-workload/stderr.txt is small (0 lines), printing all lines: 2024-05-24 18:11:20,391 DEBUG ----- output ----- 2024-05-24 18:11:20,391 DEBUG 2024-05-24 18:11:20,391 INFO ________________________________________________________________________________ [2024-05-24 18:11:20] prepare-workload check_rc_and_logs : Check order's outputs ________________________________________________________________________________ PASS: no error found in rc log PASS: no error found in stdout log PASS: no error found in stderr log ________________________________________________________________________________ [2024-05-24 18:11:20] prepare-workload print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Task result is INIT no need to display logs. ________________________________________________________________________________ [2024-05-24 18:11:20] prepare-workload post : Check output ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped