--Task-- name: restore-benchmark-users enabled: True class_name: RestoreFileTask source_name: overseer-0 source_namespace: >default< target_name: idm target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: ['restore-user-feeder'] wait_for: [] preceding_task: None options: {'_options_': '/results/idm/benchmark_users.csv'} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [2024-04-24 05:23:54] restore-benchmark-users pre : detect if workload identity is enabled where the pod runs and add the necessary binding to the Google Service account (pit-test) ________________________________________________________________________________ TODO - self.option was string "/results/idm/benchmark_users.csv" - that has to be refactored in datainit.py! 2024-04-24 05:23:54,657 INFO 2024-04-24 05:23:54,657 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:23:54,657 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:23:54,969 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:23:54,970 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:23:54,970 DEBUG --- stdout --- 2024-04-24 05:23:54,970 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:23:54,970 DEBUG --- stderr --- 2024-04-24 05:23:54,970 DEBUG 2024-04-24 05:23:54,970 INFO 2024-04-24 05:23:54,970 INFO [loop_until]: kubectl --namespace=xlou get pod overseer-0-55d5975bd7-gp92m --output jsonpath={.spec.nodeName} 2024-04-24 05:23:54,970 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 05:23:55,279 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:23:55,279 DEBUG --- stdout --- 2024-04-24 05:23:55,279 DEBUG gke-xlou-cdm-frontend-a8771548-i82n 2024-04-24 05:23:55,279 DEBUG --- stderr --- 2024-04-24 05:23:55,279 DEBUG 2024-04-24 05:23:55,280 INFO 2024-04-24 05:23:55,280 INFO [loop_until]: kubectl get node gke-xlou-cdm-frontend-a8771548-i82n --output jsonpath={.metadata.labels} 2024-04-24 05:23:55,280 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 05:23:55,675 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:23:55,675 DEBUG --- stdout --- 2024-04-24 05:23:55,675 DEBUG {"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/instance-type":"n2-standard-16","beta.kubernetes.io/os":"linux","cloud.google.com/gke-boot-disk":"pd-ssd","cloud.google.com/gke-container-runtime":"containerd","cloud.google.com/gke-cpu-scaling-level":"16","cloud.google.com/gke-logging-variant":"DEFAULT","cloud.google.com/gke-max-pods-per-node":"64","cloud.google.com/gke-nodepool":"frontend","cloud.google.com/gke-os-distribution":"cos","cloud.google.com/machine-family":"n2","cloud.google.com/private-node":"false","failure-domain.beta.kubernetes.io/region":"us-east1","failure-domain.beta.kubernetes.io/zone":"us-east1-d","forgerock.io/role":"frontend","frontend":"true","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"gke-xlou-cdm-frontend-a8771548-i82n","kubernetes.io/os":"linux","node.kubernetes.io/instance-type":"n2-standard-16","topology.gke.io/zone":"us-east1-d","topology.kubernetes.io/region":"us-east1","topology.kubernetes.io/zone":"us-east1-d"} 2024-04-24 05:23:55,675 DEBUG --- stderr --- 2024-04-24 05:23:55,676 DEBUG [2024-04-24 05:23:55] - INFO: Timeout for this Task has changed to 8h (28800 seconds) [2024-04-24 05:23:55] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset Pod log file : /results/orders/restore-benchmark-users/stdout.txt Pod log : /results/orders/restore-benchmark-users ________________________________________________________________________________ [2024-04-24 05:23:55] restore-benchmark-users print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ gcloud storage cp gs://performance-bucket-us-east1/perf-tests/7.4-10M-users/benchmark_users.csv /results/idm/benchmark_users.csv ________________________________________________________________________________ [2024-04-24 05:23:55] restore-benchmark-users create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file: /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/order.json { "name": "restore-benchmark-users", "command": "gcloud storage cp gs://performance-bucket-us-east1/perf-tests/7.4-10M-users/benchmark_users.csv /results/idm/benchmark_users.csv", "out-dir": "/results/orders/restore-benchmark-users" } ________________________________________________________________________________ [2024-04-24 05:23:55] restore-benchmark-users loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2024-04-24 05:23:55,677 INFO 2024-04-24 05:23:55,677 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2024-04-24 05:23:55,677 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2024-04-24 05:23:55,913 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-24 05:23:55,913 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:23:55,913 DEBUG --- stdout --- 2024-04-24 05:23:55,913 DEBUG WAITING 2024-04-24 05:23:55,913 DEBUG --- stderr --- 2024-04-24 05:23:55,913 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 33 0 --:--:-- --:--:-- --:--:-- 33 ________________________________________________________________________________ [2024-04-24 05:23:55] restore-benchmark-users start_order : Send the order to the overseer pod ________________________________________________________________________________ 2024-04-24 05:23:55,914 INFO 2024-04-24 05:23:55,914 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:23:55,914 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:23:56,233 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:23:56,233 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:23:56,233 DEBUG --- stdout --- 2024-04-24 05:23:56,233 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:23:56,234 DEBUG --- stderr --- 2024-04-24 05:23:56,234 DEBUG 2024-04-24 05:23:56,234 INFO 2024-04-24 05:23:56,234 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/order.json overseer-0-55d5975bd7-gp92m:/results/orders/order.json 2024-04-24 05:23:56,234 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 05:23:57,628 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:23:57,628 DEBUG --- stdout --- 2024-04-24 05:23:57,628 DEBUG 2024-04-24 05:23:57,628 DEBUG --- stderr --- 2024-04-24 05:23:57,628 DEBUG Order has been successfully forwarded to overseer ________________________________________________________________________________ [2024-04-24 05:23:57] restore-benchmark-users wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 8h Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING PASS : Order stopped by itself after 2m and before timeout (8h) ________________________________________________________________________________ [2024-04-24 05:25:57] restore-benchmark-users stop_order : remove the policy binding to the GSA for the source's KSA ________________________________________________________________________________ 2024-04-24 05:25:57,987 INFO 2024-04-24 05:25:57,987 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:25:57,987 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:25:58,364 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:25:58,364 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:25:58,364 DEBUG --- stdout --- 2024-04-24 05:25:58,364 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:25:58,364 DEBUG --- stderr --- 2024-04-24 05:25:58,364 DEBUG 2024-04-24 05:25:58,365 INFO 2024-04-24 05:25:58,365 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-55d5975bd7-gp92m -- rm -f /results/orders/order.json 2024-04-24 05:25:58,365 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 05:25:59,192 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:25:59,193 DEBUG --- stdout --- 2024-04-24 05:25:59,193 DEBUG 2024-04-24 05:25:59,193 DEBUG --- stderr --- 2024-04-24 05:25:59,193 DEBUG Order ran during 2m 01s ________________________________________________________________________________ [2024-04-24 05:25:59] restore-benchmark-users download_logs : Download individual order files ________________________________________________________________________________ [2024-04-24 05:25:59] - PASS: file already downloaded /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/order.json 2024-04-24 05:25:59,194 INFO 2024-04-24 05:25:59,194 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:25:59,194 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:25:59,509 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:25:59,509 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:25:59,510 DEBUG --- stdout --- 2024-04-24 05:25:59,510 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:25:59,510 DEBUG --- stderr --- 2024-04-24 05:25:59,510 DEBUG Checking pod: overseer-0-55d5975bd7-gp92m for file exists, if not wait. 2024-04-24 05:25:59,510 INFO 2024-04-24 05:25:59,510 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:25:59,510 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:25:59,833 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:25:59,834 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:25:59,834 DEBUG --- stdout --- 2024-04-24 05:25:59,834 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:25:59,834 DEBUG --- stderr --- 2024-04-24 05:25:59,834 DEBUG 2024-04-24 05:25:59,835 INFO 2024-04-24 05:25:59,835 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-55d5975bd7-gp92m --stdin -- bash -c 'if [ -f "/results/orders/restore-benchmark-users/rc.txt" ]; then echo "/results/orders/restore-benchmark-users/rc.txt file found"; exit 0; else echo "/results/orders/restore-benchmark-users/rc.txt file not found"; exit 1; fi' | grep "/results/orders/restore-benchmark-users/rc.txt file found" 2024-04-24 05:25:59,835 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 05:26:00,672 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-24 05:26:00,672 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:00,672 DEBUG --- stdout --- 2024-04-24 05:26:00,672 DEBUG /results/orders/restore-benchmark-users/rc.txt file found 2024-04-24 05:26:00,672 DEBUG --- stderr --- 2024-04-24 05:26:00,672 DEBUG 2024-04-24 05:26:00,673 INFO 2024-04-24 05:26:00,673 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:00,673 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:01,025 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:01,026 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:01,026 DEBUG --- stdout --- 2024-04-24 05:26:01,026 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:01,026 DEBUG --- stderr --- 2024-04-24 05:26:01,026 DEBUG 2024-04-24 05:26:01,026 INFO 2024-04-24 05:26:01,026 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-55d5975bd7-gp92m --stdin -- du -k /results/orders/restore-benchmark-users/rc.txt 2024-04-24 05:26:01,902 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-24 05:26:01,902 DEBUG --- stdout --- 2024-04-24 05:26:01,902 DEBUG 4 /results/orders/restore-benchmark-users/rc.txt 2024-04-24 05:26:01,902 DEBUG --- stderr --- 2024-04-24 05:26:01,902 DEBUG 2024-04-24 05:26:01,902 INFO - Download file (size 4k) 2024-04-24 05:26:01,903 INFO 2024-04-24 05:26:01,903 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:01,903 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:02,230 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:02,230 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:02,230 DEBUG --- stdout --- 2024-04-24 05:26:02,230 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:02,230 DEBUG --- stderr --- 2024-04-24 05:26:02,230 DEBUG source : /results/orders/restore-benchmark-users/rc.txt @ overseer-0-55d5975bd7-gp92m destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/rc.txt 2024-04-24 05:26:02,231 INFO 2024-04-24 05:26:02,231 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:02,231 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:02,542 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:02,543 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:02,543 DEBUG --- stdout --- 2024-04-24 05:26:02,543 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:02,543 DEBUG --- stderr --- 2024-04-24 05:26:02,543 DEBUG 2024-04-24 05:26:02,543 INFO 2024-04-24 05:26:02,543 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-55d5975bd7-gp92m:/results/orders/restore-benchmark-users/rc.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/rc.txt 2024-04-24 05:26:02,543 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-24 05:26:03,297 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:03,298 DEBUG --- stdout --- 2024-04-24 05:26:03,298 DEBUG tar: Removing leading `/' from member names 2024-04-24 05:26:03,298 DEBUG --- stderr --- 2024-04-24 05:26:03,298 DEBUG 2024-04-24 05:26:03,298 INFO 2024-04-24 05:26:03,299 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:03,299 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:03,618 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:03,618 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:03,618 DEBUG --- stdout --- 2024-04-24 05:26:03,618 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:03,618 DEBUG --- stderr --- 2024-04-24 05:26:03,618 DEBUG Checking pod: overseer-0-55d5975bd7-gp92m for file exists, if not wait. 2024-04-24 05:26:03,619 INFO 2024-04-24 05:26:03,619 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:03,619 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:03,933 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:03,933 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:03,933 DEBUG --- stdout --- 2024-04-24 05:26:03,933 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:03,933 DEBUG --- stderr --- 2024-04-24 05:26:03,933 DEBUG 2024-04-24 05:26:03,934 INFO 2024-04-24 05:26:03,934 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-55d5975bd7-gp92m --stdin -- bash -c 'if [ -f "/results/orders/restore-benchmark-users/stderr.txt" ]; then echo "/results/orders/restore-benchmark-users/stderr.txt file found"; exit 0; else echo "/results/orders/restore-benchmark-users/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/restore-benchmark-users/stderr.txt file found" 2024-04-24 05:26:03,934 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 05:26:04,777 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-24 05:26:04,778 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:04,778 DEBUG --- stdout --- 2024-04-24 05:26:04,778 DEBUG /results/orders/restore-benchmark-users/stderr.txt file found 2024-04-24 05:26:04,778 DEBUG --- stderr --- 2024-04-24 05:26:04,778 DEBUG 2024-04-24 05:26:04,778 INFO 2024-04-24 05:26:04,778 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:04,778 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:05,112 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:05,112 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:05,112 DEBUG --- stdout --- 2024-04-24 05:26:05,112 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:05,112 DEBUG --- stderr --- 2024-04-24 05:26:05,112 DEBUG 2024-04-24 05:26:05,112 INFO 2024-04-24 05:26:05,113 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-55d5975bd7-gp92m --stdin -- du -k /results/orders/restore-benchmark-users/stderr.txt 2024-04-24 05:26:05,949 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-24 05:26:05,949 DEBUG --- stdout --- 2024-04-24 05:26:05,951 DEBUG 4 /results/orders/restore-benchmark-users/stderr.txt 2024-04-24 05:26:05,951 DEBUG --- stderr --- 2024-04-24 05:26:05,951 DEBUG 2024-04-24 05:26:05,951 INFO - Download file (size 4k) 2024-04-24 05:26:05,951 INFO 2024-04-24 05:26:05,951 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:05,952 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:06,301 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:06,301 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:06,301 DEBUG --- stdout --- 2024-04-24 05:26:06,301 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:06,301 DEBUG --- stderr --- 2024-04-24 05:26:06,302 DEBUG source : /results/orders/restore-benchmark-users/stderr.txt @ overseer-0-55d5975bd7-gp92m destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/stderr.txt 2024-04-24 05:26:06,302 INFO 2024-04-24 05:26:06,302 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:06,302 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:06,653 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:06,653 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:06,654 DEBUG --- stdout --- 2024-04-24 05:26:06,654 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:06,654 DEBUG --- stderr --- 2024-04-24 05:26:06,654 DEBUG 2024-04-24 05:26:06,654 INFO 2024-04-24 05:26:06,654 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-55d5975bd7-gp92m:/results/orders/restore-benchmark-users/stderr.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/stderr.txt 2024-04-24 05:26:06,654 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-24 05:26:07,441 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:07,441 DEBUG --- stdout --- 2024-04-24 05:26:07,441 DEBUG tar: Removing leading `/' from member names 2024-04-24 05:26:07,441 DEBUG --- stderr --- 2024-04-24 05:26:07,441 DEBUG 2024-04-24 05:26:07,442 INFO 2024-04-24 05:26:07,442 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:07,442 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:07,756 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:07,756 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:07,756 DEBUG --- stdout --- 2024-04-24 05:26:07,757 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:07,757 DEBUG --- stderr --- 2024-04-24 05:26:07,757 DEBUG Checking pod: overseer-0-55d5975bd7-gp92m for file exists, if not wait. 2024-04-24 05:26:07,757 INFO 2024-04-24 05:26:07,757 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:07,757 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:08,069 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:08,069 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:08,069 DEBUG --- stdout --- 2024-04-24 05:26:08,069 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:08,069 DEBUG --- stderr --- 2024-04-24 05:26:08,069 DEBUG 2024-04-24 05:26:08,070 INFO 2024-04-24 05:26:08,070 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-55d5975bd7-gp92m --stdin -- bash -c 'if [ -f "/results/orders/restore-benchmark-users/stdout.txt" ]; then echo "/results/orders/restore-benchmark-users/stdout.txt file found"; exit 0; else echo "/results/orders/restore-benchmark-users/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/restore-benchmark-users/stdout.txt file found" 2024-04-24 05:26:08,070 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 05:26:08,897 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-24 05:26:08,897 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:08,897 DEBUG --- stdout --- 2024-04-24 05:26:08,897 DEBUG /results/orders/restore-benchmark-users/stdout.txt file found 2024-04-24 05:26:08,897 DEBUG --- stderr --- 2024-04-24 05:26:08,897 DEBUG 2024-04-24 05:26:08,897 INFO 2024-04-24 05:26:08,897 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:08,897 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:09,213 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:09,213 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:09,213 DEBUG --- stdout --- 2024-04-24 05:26:09,214 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:09,214 DEBUG --- stderr --- 2024-04-24 05:26:09,214 DEBUG 2024-04-24 05:26:09,214 INFO 2024-04-24 05:26:09,214 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-55d5975bd7-gp92m --stdin -- du -k /results/orders/restore-benchmark-users/stdout.txt 2024-04-24 05:26:10,048 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-24 05:26:10,048 DEBUG --- stdout --- 2024-04-24 05:26:10,048 DEBUG 0 /results/orders/restore-benchmark-users/stdout.txt 2024-04-24 05:26:10,052 DEBUG --- stderr --- 2024-04-24 05:26:10,052 DEBUG 2024-04-24 05:26:10,053 INFO - Download file (size 0k) 2024-04-24 05:26:10,053 INFO 2024-04-24 05:26:10,053 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:10,053 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:10,442 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:10,442 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:10,442 DEBUG --- stdout --- 2024-04-24 05:26:10,442 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:10,443 DEBUG --- stderr --- 2024-04-24 05:26:10,443 DEBUG source : /results/orders/restore-benchmark-users/stdout.txt @ overseer-0-55d5975bd7-gp92m destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/stdout.txt 2024-04-24 05:26:10,443 INFO 2024-04-24 05:26:10,443 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:10,443 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:10,797 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:10,797 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:10,797 DEBUG --- stdout --- 2024-04-24 05:26:10,797 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:10,797 DEBUG --- stderr --- 2024-04-24 05:26:10,797 DEBUG 2024-04-24 05:26:10,798 INFO 2024-04-24 05:26:10,798 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-55d5975bd7-gp92m:/results/orders/restore-benchmark-users/stdout.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/stdout.txt 2024-04-24 05:26:10,798 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-24 05:26:11,554 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:11,555 DEBUG --- stdout --- 2024-04-24 05:26:11,555 DEBUG tar: Removing leading `/' from member names 2024-04-24 05:26:11,555 DEBUG --- stderr --- 2024-04-24 05:26:11,555 DEBUG ________________________________________________________________________________ [2024-04-24 05:26:11] restore-benchmark-users show_cmd : Display order command ________________________________________________________________________________ 2024-04-24 05:26:11,556 INFO 2024-04-24 05:26:11,556 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:11,556 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:11,878 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:11,878 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:11,878 DEBUG --- stdout --- 2024-04-24 05:26:11,878 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:11,878 DEBUG --- stderr --- 2024-04-24 05:26:11,878 DEBUG Checking pod: overseer-0-55d5975bd7-gp92m for file exists, if not wait. 2024-04-24 05:26:11,879 INFO 2024-04-24 05:26:11,879 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:11,879 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:12,196 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:12,196 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:12,196 DEBUG --- stdout --- 2024-04-24 05:26:12,196 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:12,196 DEBUG --- stderr --- 2024-04-24 05:26:12,196 DEBUG 2024-04-24 05:26:12,197 INFO 2024-04-24 05:26:12,197 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-55d5975bd7-gp92m --stdin -- bash -c 'if [ -f "/results/orders/restore-benchmark-users/cmd.txt" ]; then echo "/results/orders/restore-benchmark-users/cmd.txt file found"; exit 0; else echo "/results/orders/restore-benchmark-users/cmd.txt file not found"; exit 1; fi' | grep "/results/orders/restore-benchmark-users/cmd.txt file found" 2024-04-24 05:26:12,197 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 05:26:13,033 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-24 05:26:13,033 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:13,033 DEBUG --- stdout --- 2024-04-24 05:26:13,033 DEBUG /results/orders/restore-benchmark-users/cmd.txt file found 2024-04-24 05:26:13,033 DEBUG --- stderr --- 2024-04-24 05:26:13,033 DEBUG 2024-04-24 05:26:13,034 INFO 2024-04-24 05:26:13,034 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:13,034 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:13,359 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:13,360 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:13,360 DEBUG --- stdout --- 2024-04-24 05:26:13,360 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:13,360 DEBUG --- stderr --- 2024-04-24 05:26:13,360 DEBUG 2024-04-24 05:26:13,360 INFO 2024-04-24 05:26:13,360 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-55d5975bd7-gp92m --stdin -- du -k /results/orders/restore-benchmark-users/cmd.txt 2024-04-24 05:26:14,199 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-24 05:26:14,199 DEBUG --- stdout --- 2024-04-24 05:26:14,199 DEBUG 4 /results/orders/restore-benchmark-users/cmd.txt 2024-04-24 05:26:14,199 DEBUG --- stderr --- 2024-04-24 05:26:14,199 DEBUG 2024-04-24 05:26:14,199 INFO - Download file (size 4k) 2024-04-24 05:26:14,199 INFO 2024-04-24 05:26:14,200 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:14,200 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:14,516 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:14,516 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:14,516 DEBUG --- stdout --- 2024-04-24 05:26:14,516 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:14,516 DEBUG --- stderr --- 2024-04-24 05:26:14,516 DEBUG source : /results/orders/restore-benchmark-users/cmd.txt @ overseer-0-55d5975bd7-gp92m destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/cmd.txt 2024-04-24 05:26:14,517 INFO 2024-04-24 05:26:14,517 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-24 05:26:14,517 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 05:26:14,835 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-24 05:26:14,836 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:14,836 DEBUG --- stdout --- 2024-04-24 05:26:14,836 DEBUG overseer-0-55d5975bd7-gp92m 2024-04-24 05:26:14,836 DEBUG --- stderr --- 2024-04-24 05:26:14,836 DEBUG 2024-04-24 05:26:14,836 INFO 2024-04-24 05:26:14,836 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-55d5975bd7-gp92m:/results/orders/restore-benchmark-users/cmd.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/cmd.txt 2024-04-24 05:26:14,836 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-24 05:26:15,602 INFO [loop_until]: OK (rc = 0) 2024-04-24 05:26:15,602 DEBUG --- stdout --- 2024-04-24 05:26:15,602 DEBUG tar: Removing leading `/' from member names 2024-04-24 05:26:15,602 DEBUG --- stderr --- 2024-04-24 05:26:15,602 DEBUG 2024-04-24 05:26:15,602 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/cmd.txt is small (1 lines), printing all lines: 2024-04-24 05:26:15,602 DEBUG ----- output ----- 2024-04-24 05:26:15,602 DEBUG gcloud storage cp gs://performance-bucket-us-east1/perf-tests/7.4-10M-users/benchmark_users.csv /results/idm/benchmark_users.csv 2024-04-24 05:26:15,602 INFO ________________________________________________________________________________ [2024-04-24 05:26:15] restore-benchmark-users show_rc : Display rc order ________________________________________________________________________________ 2024-04-24 05:26:15,603 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/rc.txt is small (1 lines), printing all lines: 2024-04-24 05:26:15,603 DEBUG ----- output ----- 2024-04-24 05:26:15,603 DEBUG CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) 2024-04-24 05:26:15,603 INFO ________________________________________________________________________________ [2024-04-24 05:26:15] restore-benchmark-users show_stdout : Display stdout order ________________________________________________________________________________ [2024-04-24 05:26:15] - INFO: gcloud storage outputs progress / status info to stderr by design to keep progress/status separated from data, this is why stderr_file file is displayed instead of stdout_file 2024-04-24 05:26:15,603 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/stderr.txt is small (5 lines), printing all lines: 2024-04-24 05:26:15,603 DEBUG ----- output ----- 2024-04-24 05:26:15,603 DEBUG Copying gs://performance-bucket-us-east1/perf-tests/7.4-10M-users/benchmark_users.csv to file:///results/idm/benchmark_users.csv ............ Average throughput: 726.5MiB/s 2024-04-24 05:26:15,603 INFO ________________________________________________________________________________ [2024-04-24 05:26:15] restore-benchmark-users show_stderr : Display stderr order ________________________________________________________________________________ [2024-04-24 05:26:15] - INFO: gcloud storage outputs progress / status info to stderr by design to keep progress/status separated from data, this is why stdout_file file is displayed instead of stderr_file 2024-04-24 05:26:15,604 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_Init/02_restore-benchmark-users/stdout.txt is small (0 lines), printing all lines: 2024-04-24 05:26:15,604 DEBUG ----- output ----- 2024-04-24 05:26:15,604 DEBUG 2024-04-24 05:26:15,604 INFO ________________________________________________________________________________ [2024-04-24 05:26:15] restore-benchmark-users check_rc_and_logs : Check order's outputs ________________________________________________________________________________ PASS: no error found in rc log [2024-04-24 05:26:15] - INFO: gcloud storage outputs progress / status info to stderr by design to keep progress/status separated from data, this is why stderr_file file is displayed instead of stdout_file PASS: no error found in stdout log [2024-04-24 05:26:15] - INFO: gcloud storage outputs progress / status info to stderr by design to keep progress/status separated from data, this is why stdout_file file is displayed instead of stderr_file PASS: no error found in stderr log ________________________________________________________________________________ [2024-04-24 05:26:15] restore-benchmark-users print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Task result is INIT no need to display logs. ________________________________________________________________________________ [2024-04-24 05:26:15] restore-benchmark-users post : Check output ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped