--Task-- name: Logs-parse-benchmark-550 enabled: True class_name: GatlingParseStdOutTask source_name: overseer-0 source_namespace: >default< target_name: overseer-0 target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates below loop: False interval: None dependencies: [] wait_for: ['Logs-parse-benchmark-10', 'Logs-parse-benchmark-100', 'Logs-parse-benchmark-200', 'Logs-parse-benchmark-300', 'Logs-parse-benchmark-400', 'Logs-parse-benchmark-500'] preceding_task: None options: {'task_name': 'benchmark-550'} group_name: None Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock source (overseer-0) is a component 2025-06-14 04:08:57 - INFO: Timeout for this Task has changed to 10m (600 seconds) 2025-06-14 04:08:57 - INFO: timeout was set to GatlingParseStdOutTask default - 10 minutes 2025-06-14 04:08:57 - INFO: Interval for this Task has changed to 5s (5 seconds) 2025-06-14 04:08:57 - INFO: interval was set to Task default because it was unset, based on self.timeout value because task is allowed to stop by itself ________________________________________________________________________________ [2025-06-14 04:08:57] Logs-parse-benchmark-550 step1 : Write order for overseer to parse Gatling task stdout ________________________________________________________________________________ Order file: /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/08_Verification/08_Logs-parse-benchmark-550/order.json { "name": "Logs-parse-benchmark-550", "command": " /lodestar/pyrock/shared/scripts/logs_parser.py --non-interactive --input-file /results/orders/benchmark-550/stdout.txt", "out-dir": "/results/orders/Logs-parse-benchmark-550" } ________________________________________________________________________________ [2025-06-14 04:08:57] Logs-parse-benchmark-550 step2 : Make sure overseer is ready to accept order (not already busy) ________________________________________________________________________________ 2025-06-14 04:08:57,147 INFO 2025-06-14 04:08:57,147 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2025-06-14 04:08:57,147 INFO [loop_until]: (max_time=120, interval=5, expected_rc=[0] 2025-06-14 04:08:57,361 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-06-14 04:08:57,361 INFO [loop_until]: OK (rc = 0) 2025-06-14 04:08:57,361 DEBUG --- stdout --- 2025-06-14 04:08:57,361 DEBUG WAITING 2025-06-14 04:08:57,361 DEBUG --- stderr --- 2025-06-14 04:08:57,361 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 33 0 --:--:-- --:--:-- --:--:-- 33 ________________________________________________________________________________ [2025-06-14 04:08:57] Logs-parse-benchmark-550 step3 : Send order to overseer ________________________________________________________________________________ 2025-06-14 04:08:57,362 INFO 2025-06-14 04:08:57,362 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-06-14 04:08:57,362 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-06-14 04:08:57,640 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-06-14 04:08:57,641 INFO [loop_until]: OK (rc = 0) 2025-06-14 04:08:57,641 DEBUG --- stdout --- 2025-06-14 04:08:57,641 DEBUG overseer-0-898c87646-bjhnj 2025-06-14 04:08:57,641 DEBUG --- stderr --- 2025-06-14 04:08:57,641 DEBUG 2025-06-14 04:08:57,641 INFO 2025-06-14 04:08:57,641 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/08_Verification/08_Logs-parse-benchmark-550/order.json overseer-0-898c87646-bjhnj:/results/orders/order.json 2025-06-14 04:08:57,641 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-06-14 04:08:59,009 INFO [loop_until]: OK (rc = 0) 2025-06-14 04:08:59,009 DEBUG --- stdout --- 2025-06-14 04:08:59,009 DEBUG 2025-06-14 04:08:59,009 DEBUG --- stderr --- 2025-06-14 04:08:59,009 DEBUG Order has been successfully forwarded to overseer ________________________________________________________________________________ [2025-06-14 04:08:59] Logs-parse-benchmark-550 step4 : Wait order is done ________________________________________________________________________________ 2025-06-14 04:09:02,010 INFO 2025-06-14 04:09:02,010 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2025-06-14 04:09:02,010 INFO [loop_until]: (max_time=600, interval=5, expected_rc=[0] 2025-06-14 04:09:02,225 INFO [loop_until]: Function succeeded after 0s (rc=0) - failed to find expected output: WAITING - retry 2025-06-14 04:09:07,442 INFO [loop_until]: Function succeeded after 5s (rc=0) - failed to find expected output: WAITING - retry 2025-06-14 04:09:12,656 INFO [loop_until]: Function succeeded after 10s (rc=0) - failed to find expected output: WAITING - retry 2025-06-14 04:09:17,873 INFO [loop_until]: Function succeeded after 15s (rc=0) - failed to find expected output: WAITING - retry 2025-06-14 04:09:23,088 INFO [loop_until]: Function succeeded after 21s (rc=0) - failed to find expected output: WAITING - retry 2025-06-14 04:09:28,344 INFO [loop_until]: Function succeeded after 26s (rc=0) - failed to find expected output: WAITING - retry 2025-06-14 04:09:33,559 INFO [loop_until]: Function succeeded after 31s (rc=0) - expected pattern found 2025-06-14 04:09:33,559 INFO [loop_until]: OK (rc = 0) 2025-06-14 04:09:33,559 DEBUG --- stdout --- 2025-06-14 04:09:33,559 DEBUG WAITING 2025-06-14 04:09:33,559 DEBUG --- stderr --- 2025-06-14 04:09:33,559 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 33 0 --:--:-- --:--:-- --:--:-- 33 ________________________________________________________________________________ [2025-06-14 04:09:33] Logs-parse-benchmark-550 step5 : Get and log parsed logs ________________________________________________________________________________ Log parsing order did not succeeded - log-parsing order log: 2025-06-14 04:09:33,797 INFO 2025-06-14 04:09:33,797 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-06-14 04:09:33,797 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-06-14 04:09:34,074 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-06-14 04:09:34,074 INFO [loop_until]: OK (rc = 0) 2025-06-14 04:09:34,074 DEBUG --- stdout --- 2025-06-14 04:09:34,074 DEBUG overseer-0-898c87646-bjhnj 2025-06-14 04:09:34,074 DEBUG --- stderr --- 2025-06-14 04:09:34,074 DEBUG Checking if file exists on pod 'overseer-0-898c87646-bjhnj', wait max 3 minutes if not. 2025-06-14 04:09:34,075 INFO 2025-06-14 04:09:34,075 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-bjhnj --stdin -- bash -c 'if [ -f "/results/orders/Logs-parse-benchmark-550/stdout.txt" ]; then echo "/results/orders/Logs-parse-benchmark-550/stdout.txt file found"; exit 0; else echo "/results/orders/Logs-parse-benchmark-550/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/Logs-parse-benchmark-550/stdout.txt file found" 2025-06-14 04:09:34,075 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-06-14 04:09:34,891 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-06-14 04:09:34,891 INFO [loop_until]: OK (rc = 0) 2025-06-14 04:09:34,891 DEBUG --- stdout --- 2025-06-14 04:09:34,891 DEBUG /results/orders/Logs-parse-benchmark-550/stdout.txt file found 2025-06-14 04:09:34,891 DEBUG --- stderr --- 2025-06-14 04:09:34,891 DEBUG 2025-06-14 04:09:34,891 INFO 2025-06-14 04:09:34,891 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-bjhnj --stdin -- du -k /results/orders/Logs-parse-benchmark-550/stdout.txt | grep "^\d+ .+$" 2025-06-14 04:09:34,891 INFO [loop_until]: (max_time=240, interval=10, expected_rc=[0] 2025-06-14 04:09:35,705 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-06-14 04:09:35,706 INFO [loop_until]: OK (rc = 0) 2025-06-14 04:09:35,706 DEBUG --- stdout --- 2025-06-14 04:09:35,706 DEBUG 4 /results/orders/Logs-parse-benchmark-550/stdout.txt 2025-06-14 04:09:35,706 DEBUG --- stderr --- 2025-06-14 04:09:35,706 DEBUG - Download file (size 4kB) source : /results/orders/Logs-parse-benchmark-550/stdout.txt @ overseer-0-898c87646-bjhnj destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/08_Verification/08_Logs-parse-benchmark-550/stdout_logs-parsing.txt 2025-06-14 04:09:35,706 INFO 2025-06-14 04:09:35,706 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-898c87646-bjhnj:/results/orders/Logs-parse-benchmark-550/stdout.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/08_Verification/08_Logs-parse-benchmark-550/stdout_logs-parsing.txt 2025-06-14 04:09:35,706 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-06-14 04:09:36,440 INFO [loop_until]: OK (rc = 0) 2025-06-14 04:09:36,440 DEBUG --- stdout --- 2025-06-14 04:09:36,440 DEBUG tar: Removing leading `/' from member names 2025-06-14 04:09:36,440 DEBUG --- stderr --- 2025-06-14 04:09:36,440 DEBUG 2025-06-14 04:09:36,440 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/08_Verification/08_Logs-parse-benchmark-550/stdout_logs-parsing.txt is small (13 lines), printing all lines: 2025-06-14 04:09:36,440 INFO ----- output ----- Running in non-interactive mode. Output file name was not provided, /results/orders/benchmark-550/stdout_parsed.txt will be used. processed log items: 50k processed log items: 100k processed log items: 150k processed log items: 200k Log items processed. Formatting output... Traceback (most recent call last): File "/lodestar/pyrock/shared/scripts/logs_parser.py", line 411, in seconds = int(log_group_timestamp / 1000) ~~~~~~~~~~~~~~~~~~~~^~~~~~ TypeError: unsupported operand type(s) for /: 'NoneType' and 'int' ________________________________________________________________________________ [2025-06-14 04:09:36] Logs-parse-benchmark-550 step6 : N/A ________________________________________________________________________________ ________________________________________________________________________________ [2025-06-14 04:09:36] Logs-parse-benchmark-550 post : Check parent task state ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped