--Task-- name: Reporting-benchmark-500 enabled: True class_name: GatlingResultTask source_name: overseer-0 source_namespace: >default< target_name: overseer-0 target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates below loop: False interval: None dependencies: [] wait_for: ['Logs-parse-benchmark-10', 'Logs-parse-benchmark-100', 'Logs-parse-benchmark-200', 'Logs-parse-benchmark-300', 'Logs-parse-benchmark-400', 'Logs-parse-benchmark-500', 'Logs-parse-benchmark-550', 'Reporting-benchmark-10', 'Reporting-benchmark-100', 'Reporting-benchmark-200', 'Reporting-benchmark-300', 'Reporting-benchmark-400'] preceding_task: None options: {'task_name': 'benchmark-500'} group_name: None Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock source (overseer-0) is a component 2025-09-21 07:06:26 - INFO: Timeout for this Task has changed to 3h 30m (12600 seconds) 2025-09-21 07:06:26 - INFO: timeout was set to GatlingResultTask default (half of global_duration or 3600 seconds (whatever is higher)), because it was unset Pod stdout file : /results/orders/Reporting-benchmark-500/stdout.txt Pod stderr file : /results/orders/Reporting-benchmark-500/stderr.txt Pod log : /results/orders/Reporting-benchmark-500 2025-09-21 07:06:26 - INFO: Interval for this Task has changed to 2m (120 seconds) 2025-09-21 07:06:26 - INFO: interval was set to Task default because it was unset, based on self.timeout value because task is allowed to stop by itself ________________________________________________________________________________ [2025-09-21 07:06:26] Reporting-benchmark-500 step1 : Write order for overseer to generate the report ________________________________________________________________________________ Order file: /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/08_Verification/14_Reporting-benchmark-500/order.json { "name": "Reporting-benchmark-500", "command": " /lodestar/lodestar.py perf run-gatling --reports-only=\"/results/gatling/benchmark-500\" --simulation-jvm-args=\"-XX:+ExitOnOutOfMemoryError -Xmx4G\" --archive-name=\"benchmark-500.tar.gz\"", "out-dir": "/results/orders/Reporting-benchmark-500" } ________________________________________________________________________________ [2025-09-21 07:06:26] Reporting-benchmark-500 step2 : Make sure overseer is ready to accept order (not already busy) ________________________________________________________________________________ 2025-09-21 07:06:26,496 INFO 2025-09-21 07:06:26,496 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2025-09-21 07:06:26,496 INFO [loop_until]: (max_time=120, interval=5, expected_rc=[0] 2025-09-21 07:06:26,708 INFO [loop_until]: Function succeeded after 0s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:06:31,913 INFO [loop_until]: Function succeeded after 5s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:06:37,125 INFO [loop_until]: Function succeeded after 10s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:06:42,336 INFO [loop_until]: Function succeeded after 15s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:06:47,633 INFO [loop_until]: Function succeeded after 21s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:06:52,842 INFO [loop_until]: Function succeeded after 26s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:06:58,054 INFO [loop_until]: Function succeeded after 31s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:03,258 INFO [loop_until]: Function succeeded after 36s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:08,470 INFO [loop_until]: Function succeeded after 41s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:13,726 INFO [loop_until]: Function succeeded after 47s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:18,934 INFO [loop_until]: Function succeeded after 52s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:24,145 INFO [loop_until]: Function succeeded after 57s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:29,350 INFO [loop_until]: Function succeeded after 1m 02s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:34,556 INFO [loop_until]: Function succeeded after 1m 08s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:39,763 INFO [loop_until]: Function succeeded after 1m 13s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:44,974 INFO [loop_until]: Function succeeded after 1m 18s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:50,182 INFO [loop_until]: Function succeeded after 1m 23s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:07:55,391 INFO [loop_until]: Function succeeded after 1m 28s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:08:00,596 INFO [loop_until]: Function succeeded after 1m 34s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:08:05,803 INFO [loop_until]: Function succeeded after 1m 39s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:08:11,015 INFO [loop_until]: Function succeeded after 1m 44s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:08:16,258 INFO [loop_until]: Function succeeded after 1m 49s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:08:21,467 INFO [loop_until]: Function succeeded after 1m 54s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:08:26,676 INFO [loop_until]: Function succeeded after 2m (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 07:08:26,676 ERROR [loop_until]: Function does not return expected output (WAITING) 2025-09-21 07:08:26,677 DEBUG --- stdout --- 2025-09-21 07:08:26,677 DEBUG RUNNING 2025-09-21 07:08:26,677 DEBUG --- stderr --- 2025-09-21 07:08:26,677 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 34 0 --:--:-- --:--:-- --:--:-- 35 2025-09-21 07:08:26 - INFO: Traceback (most recent call last): File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/components/overseer.py", line 299, in loop_until_ready result = loop_until( ^^^^^^^^^^^ File "/mnt/disk1/xslou/workshop/lodestar-fork/shared/lib/utils/cmd/cmd.py", line 664, in loop_until raise FailException(message=exception_message) shared.lib.utils.exception.FailException: [loop_until]: Function does not return expected output (WAITING) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/Task.py", line 642, in run self._run() File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/StepTask.py", line 57, in _run self._run_method(step) File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/Task.py", line 683, in _run_method exec("self.%s()" % method) File "", line 1, in File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/tasks/scenario/gatling.py", line 638, in step2 self.source.loop_until_ready() File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/components/overseer.py", line 309, in loop_until_ready raise FailException shared.lib.utils.exception.FailException: None Setting result to FAIL Task has been successfully stopped