--Task-- name: benchmark-10 enabled: True class_name: GatlingTask source_name: overseer-0 source_namespace: >default< target_name: am target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates below loop: False interval: None dependencies: [] wait_for: [] preceding_task: None options: {'duration': 3600, 'java-opts': '-Drealm=alpha -Duser_password=Pa_ssw0rd -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html', 'testCase': 'idc.benchmark.Mix2024', 'testMode': 'open', 'throughput': 10} group_name: None Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock 2025-09-21 06:07:00,514 INFO 2025-09-21 06:07:00,514 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-09-21 06:07:00,514 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-09-21 06:07:00,790 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-09-21 06:07:00,790 INFO [loop_until]: OK (rc = 0) 2025-09-21 06:07:00,790 DEBUG --- stdout --- 2025-09-21 06:07:00,790 DEBUG overseer-0-57d79d468-khs5t 2025-09-21 06:07:00,790 DEBUG --- stderr --- 2025-09-21 06:07:00,790 DEBUG 2025-09-21 06:07:00,790 INFO Lodestar assumed that full stack is deployed so will set all products host URLs in Gatling system properties 2025-09-21 06:07:00 - INFO: Timeout for this Task has changed to 1h 12m (4320 seconds) 2025-09-21 06:07:00 - INFO: timeout was calculated based on simulation duration parameter 2025-09-21 06:07:00 - INFO: Interval for this Task has changed to 2m (120 seconds) 2025-09-21 06:07:00 - INFO: interval was set to Task default because it was unset, based on self.timeout value because task is allowed to stop by itself Pod log file : /results/orders/benchmark-10/stdout.txt Pod log : /results/orders/benchmark-10 2025-09-21 06:07:01 - INFO: Get overseer pod resources requests memory 2025-09-21 06:07:01,580 INFO 2025-09-21 06:07:01,580 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods overseer-0-57d79d468-khs5t --output jsonpath={.spec.containers[].resources.requests.memory} 2025-09-21 06:07:01,580 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-09-21 06:07:01,860 INFO [loop_until]: OK (rc = 0) 2025-09-21 06:07:01,860 DEBUG --- stdout --- 2025-09-21 06:07:01,860 DEBUG 4Gi 2025-09-21 06:07:01,860 DEBUG --- stderr --- 2025-09-21 06:07:01,860 DEBUG ________________________________________________________________________________ [2025-09-21 06:07:01] benchmark-10 print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class="com.forgerock.pyrock.Loadtest" --run-description="idc.benchmark.Mix2024@perf-haproxy" --results-folder="/results/gatling/benchmark-10" --no-reports --simulation-jvm-args="-XX:+ExitOnOutOfMemoryError -Xmx4G" --simulation-system-properties="-Dam_host=openam-perf-haproxy.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dduration=3600 -Didc=true -Didm_host=openam-perf-haproxy.forgeblocks.com -Dlogin_cookie=d0d18cac672abbc -Dnum_users=10000000 -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idc.benchmark.Mix2024 -DtestMode=open -Dthroughput=10 -Duser_password=Pa_ssw0rd" ________________________________________________________________________________ [2025-09-21 06:07:01] benchmark-10 create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file: /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/01_benchmark-10/order.json { "name": "benchmark-10", "command": " /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class=\"com.forgerock.pyrock.Loadtest\" --run-description=\"idc.benchmark.Mix2024@perf-haproxy\" --results-folder=\"/results/gatling/benchmark-10\" --no-reports --simulation-jvm-args=\"-XX:+ExitOnOutOfMemoryError -Xmx4G\" --simulation-system-properties=\"-Dam_host=openam-perf-haproxy.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dduration=3600 -Didc=true -Didm_host=openam-perf-haproxy.forgeblocks.com -Dlogin_cookie=d0d18cac672abbc -Dnum_users=10000000 -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idc.benchmark.Mix2024 -DtestMode=open -Dthroughput=10 -Duser_password=Pa_ssw0rd\"", "out-dir": "/results/orders/benchmark-10" } ________________________________________________________________________________ [2025-09-21 06:07:01] benchmark-10 loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2025-09-21 06:07:01,869 INFO 2025-09-21 06:07:01,869 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2025-09-21 06:07:01,869 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2025-09-21 06:07:02,106 INFO [loop_until]: Function succeeded after 0s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:07,318 INFO [loop_until]: Function succeeded after 5s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:12,529 INFO [loop_until]: Function succeeded after 10s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:17,735 INFO [loop_until]: Function succeeded after 15s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:22,942 INFO [loop_until]: Function succeeded after 21s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:28,147 INFO [loop_until]: Function succeeded after 26s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:33,421 INFO [loop_until]: Function succeeded after 31s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:38,626 INFO [loop_until]: Function succeeded after 36s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:43,834 INFO [loop_until]: Function succeeded after 41s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:49,045 INFO [loop_until]: Function succeeded after 47s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:54,250 INFO [loop_until]: Function succeeded after 52s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:07:59,454 INFO [loop_until]: Function succeeded after 57s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:04,690 INFO [loop_until]: Function succeeded after 1m 02s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:09,895 INFO [loop_until]: Function succeeded after 1m 08s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:15,100 INFO [loop_until]: Function succeeded after 1m 13s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:20,308 INFO [loop_until]: Function succeeded after 1m 18s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:25,512 INFO [loop_until]: Function succeeded after 1m 23s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:30,720 INFO [loop_until]: Function succeeded after 1m 28s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:35,956 INFO [loop_until]: Function succeeded after 1m 34s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:41,169 INFO [loop_until]: Function succeeded after 1m 39s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:46,380 INFO [loop_until]: Function succeeded after 1m 44s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:51,589 INFO [loop_until]: Function succeeded after 1m 49s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:08:56,798 INFO [loop_until]: Function succeeded after 1m 54s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:02,002 INFO [loop_until]: Function succeeded after 2m (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:07,260 INFO [loop_until]: Function succeeded after 2m 05s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:12,468 INFO [loop_until]: Function succeeded after 2m 10s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:17,680 INFO [loop_until]: Function succeeded after 2m 15s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:22,892 INFO [loop_until]: Function succeeded after 2m 21s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:28,100 INFO [loop_until]: Function succeeded after 2m 26s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:33,313 INFO [loop_until]: Function succeeded after 2m 31s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:38,553 INFO [loop_until]: Function succeeded after 2m 36s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:43,762 INFO [loop_until]: Function succeeded after 2m 41s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:48,967 INFO [loop_until]: Function succeeded after 2m 47s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:54,172 INFO [loop_until]: Function succeeded after 2m 52s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:09:59,379 INFO [loop_until]: Function succeeded after 2m 57s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:04,587 INFO [loop_until]: Function succeeded after 3m 02s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:09,803 INFO [loop_until]: Function succeeded after 3m 07s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:15,007 INFO [loop_until]: Function succeeded after 3m 13s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:20,212 INFO [loop_until]: Function succeeded after 3m 18s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:25,417 INFO [loop_until]: Function succeeded after 3m 23s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:30,622 INFO [loop_until]: Function succeeded after 3m 28s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:35,833 INFO [loop_until]: Function succeeded after 3m 33s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:41,040 INFO [loop_until]: Function succeeded after 3m 39s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:46,248 INFO [loop_until]: Function succeeded after 3m 44s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:51,456 INFO [loop_until]: Function succeeded after 3m 49s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:10:56,660 INFO [loop_until]: Function succeeded after 3m 54s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:01,865 INFO [loop_until]: Function succeeded after 3m 59s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:07,070 INFO [loop_until]: Function succeeded after 4m 05s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:12,306 INFO [loop_until]: Function succeeded after 4m 10s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:17,511 INFO [loop_until]: Function succeeded after 4m 15s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:22,722 INFO [loop_until]: Function succeeded after 4m 20s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:27,926 INFO [loop_until]: Function succeeded after 4m 26s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:33,134 INFO [loop_until]: Function succeeded after 4m 31s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:38,344 INFO [loop_until]: Function succeeded after 4m 36s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:43,592 INFO [loop_until]: Function succeeded after 4m 41s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:48,800 INFO [loop_until]: Function succeeded after 4m 46s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:54,005 INFO [loop_until]: Function succeeded after 4m 52s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:11:59,213 INFO [loop_until]: Function succeeded after 4m 57s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:12:04,421 INFO [loop_until]: Function succeeded after 5m 02s (rc=0) - failed to find expected output: WAITING - retry 2025-09-21 06:12:04,421 ERROR [loop_until]: Function does not return expected output (WAITING) 2025-09-21 06:12:04,421 DEBUG --- stdout --- 2025-09-21 06:12:04,422 DEBUG RUNNING 2025-09-21 06:12:04,422 DEBUG --- stderr --- 2025-09-21 06:12:04,422 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 35 0 --:--:-- --:--:-- --:--:-- 35 2025-09-21 06:12:04 - INFO: Traceback (most recent call last): File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/components/overseer.py", line 299, in loop_until_ready result = loop_until( ^^^^^^^^^^^ File "/mnt/disk1/xslou/workshop/lodestar-fork/shared/lib/utils/cmd/cmd.py", line 664, in loop_until raise FailException(message=exception_message) shared.lib.utils.exception.FailException: [loop_until]: Function does not return expected output (WAITING) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/Task.py", line 642, in run self._run() File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/OverseerTask.py", line 97, in _run self.loop_until_ready() File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/OverseerTask.py", line 208, in loop_until_ready self.source.loop_until_ready(max_time=max_time, interval=interval) File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/components/overseer.py", line 309, in loop_until_ready raise FailException shared.lib.utils.exception.FailException: None Setting result to FAIL Task has been successfully stopped