--Task-- name: benchmark-100 enabled: True class_name: GatlingTask source_name: overseer-0 source_namespace: >default< target_name: am target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates below loop: False interval: None dependencies: [] wait_for: ['benchmark-10'] preceding_task: benchmark-10 options: {'duration': 3600, 'java-opts': '-Drealm=alpha -Duser_password=Pa_ssw0rd -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html', 'testCase': 'idc.benchmark.Mix2024', 'testMode': 'open', 'throughput': 100} group_name: None Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock 2025-09-16 06:35:25,569 INFO 2025-09-16 06:35:25,569 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-09-16 06:35:25,569 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-09-16 06:35:25,851 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-09-16 06:35:25,851 INFO [loop_until]: OK (rc = 0) 2025-09-16 06:35:25,851 DEBUG --- stdout --- 2025-09-16 06:35:25,851 DEBUG overseer-0-57d79d468-khs5t 2025-09-16 06:35:25,851 DEBUG --- stderr --- 2025-09-16 06:35:25,851 DEBUG 2025-09-16 06:35:25,851 INFO Lodestar assumed that full stack is deployed so will set all products host URLs in Gatling system properties 2025-09-16 06:35:25 - INFO: Timeout for this Task has changed to 1h 12m (4320 seconds) 2025-09-16 06:35:25 - INFO: timeout was calculated based on simulation duration parameter 2025-09-16 06:35:25 - INFO: Interval for this Task has changed to 2m (120 seconds) 2025-09-16 06:35:25 - INFO: interval was set to Task default because it was unset, based on self.timeout value because task is allowed to stop by itself Pod log file : /results/orders/benchmark-100/stdout.txt Pod log : /results/orders/benchmark-100 ________________________________________________________________________________ [2025-09-16 06:35:25] benchmark-100 print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class="com.forgerock.pyrock.Loadtest" --run-description="idc.benchmark.Mix2024@perf-haproxy" --results-folder="/results/gatling/benchmark-100" --no-reports --simulation-jvm-args="-XX:+ExitOnOutOfMemoryError -Xmx4G" --simulation-system-properties="-Dam_host=openam-perf-haproxy.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dduration=3600 -Didc=true -Didm_host=openam-perf-haproxy.forgeblocks.com -Dlogin_cookie=d0d18cac672abbc -Dnum_users=10000000 -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idc.benchmark.Mix2024 -DtestMode=open -Dthroughput=100 -Duser_password=Pa_ssw0rd" ________________________________________________________________________________ [2025-09-16 06:35:25] benchmark-100 create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file: /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/order.json { "name": "benchmark-100", "command": " /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class=\"com.forgerock.pyrock.Loadtest\" --run-description=\"idc.benchmark.Mix2024@perf-haproxy\" --results-folder=\"/results/gatling/benchmark-100\" --no-reports --simulation-jvm-args=\"-XX:+ExitOnOutOfMemoryError -Xmx4G\" --simulation-system-properties=\"-Dam_host=openam-perf-haproxy.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dduration=3600 -Didc=true -Didm_host=openam-perf-haproxy.forgeblocks.com -Dlogin_cookie=d0d18cac672abbc -Dnum_users=10000000 -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idc.benchmark.Mix2024 -DtestMode=open -Dthroughput=100 -Duser_password=Pa_ssw0rd\"", "out-dir": "/results/orders/benchmark-100" } ________________________________________________________________________________ [2025-09-16 06:35:25] benchmark-100 loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2025-09-16 06:35:25,860 INFO 2025-09-16 06:35:25,860 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2025-09-16 06:35:25,860 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2025-09-16 06:35:26,064 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-09-16 06:35:26,064 INFO [loop_until]: OK (rc = 0) 2025-09-16 06:35:26,064 DEBUG --- stdout --- 2025-09-16 06:35:26,065 DEBUG WAITING 2025-09-16 06:35:26,065 DEBUG --- stderr --- 2025-09-16 06:35:26,065 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 35 0 --:--:-- --:--:-- --:--:-- 35 ________________________________________________________________________________ [2025-09-16 06:35:26] benchmark-100 start_order : Send the order to the overseer pod ________________________________________________________________________________ 2025-09-16 06:35:26,065 INFO 2025-09-16 06:35:26,065 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-09-16 06:35:26,065 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-09-16 06:35:26,341 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-09-16 06:35:26,342 INFO [loop_until]: OK (rc = 0) 2025-09-16 06:35:26,342 DEBUG --- stdout --- 2025-09-16 06:35:26,342 DEBUG overseer-0-57d79d468-khs5t 2025-09-16 06:35:26,342 DEBUG --- stderr --- 2025-09-16 06:35:26,342 DEBUG 2025-09-16 06:35:26,342 INFO 2025-09-16 06:35:26,342 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/order.json overseer-0-57d79d468-khs5t:/results/orders/order.json 2025-09-16 06:35:26,342 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-09-16 06:35:27,228 INFO [loop_until]: OK (rc = 0) 2025-09-16 06:35:27,228 DEBUG --- stdout --- 2025-09-16 06:35:27,228 DEBUG 2025-09-16 06:35:27,228 DEBUG --- stderr --- 2025-09-16 06:35:27,229 DEBUG Order has been successfully forwarded to overseer 2025-09-16 06:35:27 - INFO: Check if Gatling simulation started successfully 2025-09-16 06:35:27,229 INFO 2025-09-16 06:35:27,229 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- head --lines 5000 /results/orders/benchmark-100/stdout.txt | grep "Simulation com.forgerock.pyrock.Loadtest started..." 2025-09-16 06:35:27,229 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2025-09-16 06:35:27,767 INFO [loop_until]: Function succeeded after 0s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-09-16 06:35:33,298 INFO [loop_until]: Function succeeded after 6s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-09-16 06:35:38,839 INFO [loop_until]: Function succeeded after 11s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-09-16 06:35:44,394 INFO [loop_until]: Function succeeded after 17s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-09-16 06:35:49,938 INFO [loop_until]: Function succeeded after 22s (rc=0) - expected pattern found 2025-09-16 06:35:49,938 INFO [loop_until]: OK (rc = 0) 2025-09-16 06:35:49,938 DEBUG --- stdout --- 2025-09-16 06:35:49,938 DEBUG 2025-09-16 07:35:29,313 INFO pom.xml and Gatling Scala files copied from /home/jenkins/lodestar/pyrock/shared/clients/gatling to temporary folder /tmp/tmp3oyesrwn 2025-09-16 07:35:29,317 INFO Running Gatling simulation with command: mvn gatling:test --batch-mode --quiet -Dgatling.simulationClass="com.forgerock.pyrock.Loadtest" -Dgatling.runDescription="..." -Dgatling.resultsFolder="/results/gatling/benchmark-100" -Dgatling.noReports=true -Dgatling.jvmArgs="-XX:+ExitOnOutOfMemoryError,-Xmx4G" -Dgatling.propagateSystemProperties=true -Dam_host=openam-perf-haproxy.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dduration=3600 -Didc=true -Didm_host=openam-perf-haproxy.forgeblocks.com -Dlogin_cookie=d0d18cac672abbc -Dnum_users=10000000 -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idc.benchmark.Mix2024 -DtestMode=open -Dthroughput=100 -Duser_password=Pa_ssw0rd Gatling 3.13.5 is available! (you're using 3.12.0) System property think_time_multiplier not set. No think time emulation will be done in the simulation. Simulation com.forgerock.pyrock.Loadtest started... 07:35:48.815 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - Request 'authAsPolicyAdmin' failed for user 1: status.find.in([200, 209], 304), found 401 07:35:48.828 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: authAsPolicyAdmin: KO status.find.in([200, 209], 304), found 401 ========================= Session: Session(getToken,1,Map(gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@7f6a1633, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@5e2c04de, gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, e52b66c0-8062-47a8-92b4-f2076e0a6411 -> 0),KO,List(ExitOnCompleteLoopBlock(e52b66c0-8062-47a8-92b4-f2076e0a6411)),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@376f9a0d,io.netty.channel.epoll.EpollEventLoop@f79a760) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/realms/alpha/authenticate headers: X-OpenAM-Username: policyAdmin Content-Type: application/json Accept-API-Version: resource=2.0, protocol=1.0 X-OpenAM-Password: Pa_ssw0rd user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_5_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 accept: */* host: openam-perf-haproxy.forgeblocks.com content-length: 0 ========================= HTTP response: version: HTTP/1.1 status: 401 Unauthorized headers: x-frame-options: SAMEORIGIN content-security-policy-report-only: frame-ancestors 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline' x-content-type-options: nosniff cache-control: private content-api-version: resource=2.1 expires: 0 pragma: no-cache content-type: application/json Content-Length: 62 date: Tue, 16 Sep 2025 06:35:48 GMT x-forgerock-transactionid: 6c9253f7-6fc2-44bc-a845-4db5ccd85881 strict-transport-security: max-age=31536000; includeSubDomains; preload; x-robots-tag: none Via: 1.1 google Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 body: {"code":401,"reason":"Unauthorized","message":"Login failure"} <<<<<<<<<<<<<<<<<<<<<<<<< 07:35:48.831 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-8' crashed with 'j.u.NoSuchElementException: No attribute named 'policyEvalTokenId' is defined', forwarding to the next one 07:35:49.624 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:49.624 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:49.637 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:49.638 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:49.747 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:49.824 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:49.825 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 2025-09-16 06:35:49,938 DEBUG --- stderr --- 2025-09-16 06:35:49,938 DEBUG ________________________________________________________________________________ [2025-09-16 06:35:49] benchmark-100 wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 1h 12m. Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING Order is running since 2m, retry Order is running since 4m, retry Order is running since 6m, retry Order is running since 8m 01s, retry Order is running since 10m 01s, retry Order is running since 12m 01s, retry Order is running since 14m 01s, retry Order is running since 16m 02s, retry Order is running since 18m 02s, retry Order is running since 20m 02s, retry Order is running since 22m 02s, retry Order is running since 24m 03s, retry Order is running since 26m 03s, retry Order is running since 28m 03s, retry Order is running since 30m 04s, retry Order is running since 32m 04s, retry Order is running since 34m 04s, retry Order is running since 36m 04s, retry Order is running since 38m 05s, retry Order is running since 40m 05s, retry Order is running since 42m 05s, retry Order is running since 44m 05s, retry Order is running since 46m 06s, retry Order is running since 48m 06s, retry Order is running since 50m 06s, retry Order is running since 52m 07s, retry Order is running since 54m 07s, retry Order is running since 56m 07s, retry Order is running since 58m 07s, retry PASS : Order stopped by itself after 1h and before timeout (1h 12m) ________________________________________________________________________________ [2025-09-16 07:35:58] benchmark-100 stop_order : Stop the order on overseer pod ________________________________________________________________________________ 2025-09-16 07:35:58,013 INFO 2025-09-16 07:35:58,013 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-09-16 07:35:58,013 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-09-16 07:35:58,316 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-09-16 07:35:58,316 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:35:58,316 DEBUG --- stdout --- 2025-09-16 07:35:58,316 DEBUG overseer-0-57d79d468-khs5t 2025-09-16 07:35:58,316 DEBUG --- stderr --- 2025-09-16 07:35:58,316 DEBUG 2025-09-16 07:35:58,317 INFO 2025-09-16 07:35:58,317 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- rm -f /results/orders/order.json 2025-09-16 07:35:58,317 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-09-16 07:35:58,910 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:35:58,910 DEBUG --- stdout --- 2025-09-16 07:35:58,910 DEBUG 2025-09-16 07:35:58,910 DEBUG --- stderr --- 2025-09-16 07:35:58,910 DEBUG Order ran during 1h ________________________________________________________________________________ [2025-09-16 07:35:58] benchmark-100 download_logs : Download individual order files ________________________________________________________________________________ 2025-09-16 07:35:58 - PASS: file already downloaded /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/order.json 2025-09-16 07:35:58,911 INFO 2025-09-16 07:35:58,911 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-09-16 07:35:58,911 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-09-16 07:35:59,190 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-09-16 07:35:59,190 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:35:59,190 DEBUG --- stdout --- 2025-09-16 07:35:59,190 DEBUG overseer-0-57d79d468-khs5t 2025-09-16 07:35:59,190 DEBUG --- stderr --- 2025-09-16 07:35:59,190 DEBUG Checking if file exists on pod 'overseer-0-57d79d468-khs5t', wait max 3 minutes if not. 2025-09-16 07:35:59,191 INFO 2025-09-16 07:35:59,191 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- bash -c 'if [ -f "/results/orders/benchmark-100/rc.txt" ]; then echo "/results/orders/benchmark-100/rc.txt file found"; exit 0; else echo "/results/orders/benchmark-100/rc.txt file not found"; exit 1; fi' | grep "/results/orders/benchmark-100/rc.txt file found" 2025-09-16 07:35:59,191 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-09-16 07:35:59,726 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-09-16 07:35:59,726 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:35:59,726 DEBUG --- stdout --- 2025-09-16 07:35:59,726 DEBUG /results/orders/benchmark-100/rc.txt file found 2025-09-16 07:35:59,726 DEBUG --- stderr --- 2025-09-16 07:35:59,726 DEBUG 2025-09-16 07:35:59,726 INFO 2025-09-16 07:35:59,726 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- du -k /results/orders/benchmark-100/rc.txt | grep "^\d+ .+$" 2025-09-16 07:35:59,726 INFO [loop_until]: (max_time=240, interval=10, expected_rc=[0] 2025-09-16 07:36:00,273 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-09-16 07:36:00,273 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:00,273 DEBUG --- stdout --- 2025-09-16 07:36:00,273 DEBUG 4 /results/orders/benchmark-100/rc.txt 2025-09-16 07:36:00,273 DEBUG --- stderr --- 2025-09-16 07:36:00,273 DEBUG - Download file (size 4kB) source : /results/orders/benchmark-100/rc.txt @ overseer-0-57d79d468-khs5t destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/rc.txt 2025-09-16 07:36:00,274 INFO 2025-09-16 07:36:00,274 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-57d79d468-khs5t:/results/orders/benchmark-100/rc.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/rc.txt 2025-09-16 07:36:00,274 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-09-16 07:36:00,822 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:00,822 DEBUG --- stdout --- 2025-09-16 07:36:00,822 DEBUG 2025-09-16 07:36:00,822 DEBUG --- stderr --- 2025-09-16 07:36:00,822 DEBUG 2025-09-16 07:36:00,822 INFO 2025-09-16 07:36:00,822 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-09-16 07:36:00,822 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-09-16 07:36:01,102 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-09-16 07:36:01,102 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:01,102 DEBUG --- stdout --- 2025-09-16 07:36:01,102 DEBUG overseer-0-57d79d468-khs5t 2025-09-16 07:36:01,102 DEBUG --- stderr --- 2025-09-16 07:36:01,102 DEBUG Checking if file exists on pod 'overseer-0-57d79d468-khs5t', wait max 3 minutes if not. 2025-09-16 07:36:01,103 INFO 2025-09-16 07:36:01,103 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- bash -c 'if [ -f "/results/orders/benchmark-100/stderr.txt" ]; then echo "/results/orders/benchmark-100/stderr.txt file found"; exit 0; else echo "/results/orders/benchmark-100/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/benchmark-100/stderr.txt file found" 2025-09-16 07:36:01,103 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-09-16 07:36:01,640 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-09-16 07:36:01,641 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:01,641 DEBUG --- stdout --- 2025-09-16 07:36:01,641 DEBUG /results/orders/benchmark-100/stderr.txt file found 2025-09-16 07:36:01,641 DEBUG --- stderr --- 2025-09-16 07:36:01,641 DEBUG 2025-09-16 07:36:01,641 INFO 2025-09-16 07:36:01,641 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- du -k /results/orders/benchmark-100/stderr.txt | grep "^\d+ .+$" 2025-09-16 07:36:01,641 INFO [loop_until]: (max_time=240, interval=10, expected_rc=[0] 2025-09-16 07:36:02,182 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-09-16 07:36:02,182 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:02,182 DEBUG --- stdout --- 2025-09-16 07:36:02,182 DEBUG 0 /results/orders/benchmark-100/stderr.txt 2025-09-16 07:36:02,182 DEBUG --- stderr --- 2025-09-16 07:36:02,182 DEBUG - Download file (size 0B) source : /results/orders/benchmark-100/stderr.txt @ overseer-0-57d79d468-khs5t destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/stderr.txt 2025-09-16 07:36:02,183 INFO 2025-09-16 07:36:02,183 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-57d79d468-khs5t:/results/orders/benchmark-100/stderr.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/stderr.txt 2025-09-16 07:36:02,183 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-09-16 07:36:02,721 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:02,721 DEBUG --- stdout --- 2025-09-16 07:36:02,721 DEBUG 2025-09-16 07:36:02,721 DEBUG --- stderr --- 2025-09-16 07:36:02,721 DEBUG 2025-09-16 07:36:02,722 INFO 2025-09-16 07:36:02,722 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-09-16 07:36:02,722 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-09-16 07:36:03,008 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-09-16 07:36:03,008 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:03,008 DEBUG --- stdout --- 2025-09-16 07:36:03,008 DEBUG overseer-0-57d79d468-khs5t 2025-09-16 07:36:03,008 DEBUG --- stderr --- 2025-09-16 07:36:03,008 DEBUG Checking if file exists on pod 'overseer-0-57d79d468-khs5t', wait max 3 minutes if not. 2025-09-16 07:36:03,009 INFO 2025-09-16 07:36:03,009 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- bash -c 'if [ -f "/results/orders/benchmark-100/stdout.txt" ]; then echo "/results/orders/benchmark-100/stdout.txt file found"; exit 0; else echo "/results/orders/benchmark-100/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/benchmark-100/stdout.txt file found" 2025-09-16 07:36:03,009 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-09-16 07:36:03,547 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-09-16 07:36:03,547 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:03,547 DEBUG --- stdout --- 2025-09-16 07:36:03,548 DEBUG /results/orders/benchmark-100/stdout.txt file found 2025-09-16 07:36:03,548 DEBUG --- stderr --- 2025-09-16 07:36:03,548 DEBUG 2025-09-16 07:36:03,549 INFO 2025-09-16 07:36:03,549 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- du -k /results/orders/benchmark-100/stdout.txt | grep "^\d+ .+$" 2025-09-16 07:36:03,549 INFO [loop_until]: (max_time=240, interval=10, expected_rc=[0] 2025-09-16 07:36:04,090 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-09-16 07:36:04,090 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:04,090 DEBUG --- stdout --- 2025-09-16 07:36:04,091 DEBUG 141256 /results/orders/benchmark-100/stdout.txt 2025-09-16 07:36:04,091 DEBUG --- stderr --- 2025-09-16 07:36:04,091 DEBUG - Download file (size 141.256MB exceeds limit of 100MB, only first 1000 and last 1000 lines will be downloaded) source : /results/orders/benchmark-100/stdout.txt @ overseer-0-57d79d468-khs5t destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/stdout.txt 2025-09-16 07:36:04,091 INFO 2025-09-16 07:36:04,091 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- head -n 1000 /results/orders/benchmark-100/stdout.txt 2025-09-16 07:36:04,828 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2025-09-16 07:36:04,828 DEBUG --- stdout --- 2025-09-16 07:36:04,829 DEBUG [print_head_tail]: Print head (80 lines) and tail (80 lines) of input string (1000 lines): 2025-09-16 07:36:04,829 DEBUG ----- output ----- 2025-09-16 07:35:29,313 INFO pom.xml and Gatling Scala files copied from /home/jenkins/lodestar/pyrock/shared/clients/gatling to temporary folder /tmp/tmp3oyesrwn 2025-09-16 07:35:29,317 INFO Running Gatling simulation with command: mvn gatling:test --batch-mode --quiet -Dgatling.simulationClass="com.forgerock.pyrock.Loadtest" -Dgatling.runDescription="..." -Dgatling.resultsFolder="/results/gatling/benchmark-100" -Dgatling.noReports=true -Dgatling.jvmArgs="-XX:+ExitOnOutOfMemoryError,-Xmx4G" -Dgatling.propagateSystemProperties=true -Dam_host=openam-perf-haproxy.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dduration=3600 -Didc=true -Didm_host=openam-perf-haproxy.forgeblocks.com -Dlogin_cookie=d0d18cac672abbc -Dnum_users=10000000 -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idc.benchmark.Mix2024 -DtestMode=open -Dthroughput=100 -Duser_password=Pa_ssw0rd Gatling 3.13.5 is available! (you're using 3.12.0) System property think_time_multiplier not set. No think time emulation will be done in the simulation. Simulation com.forgerock.pyrock.Loadtest started... 07:35:48.815 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - Request 'authAsPolicyAdmin' failed for user 1: status.find.in([200, 209], 304), found 401 07:35:48.828 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: authAsPolicyAdmin: KO status.find.in([200, 209], 304), found 401 ========================= Session: Session(getToken,1,Map(gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@7f6a1633, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@5e2c04de, gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, e52b66c0-8062-47a8-92b4-f2076e0a6411 -> 0),KO,List(ExitOnCompleteLoopBlock(e52b66c0-8062-47a8-92b4-f2076e0a6411)),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@376f9a0d,io.netty.channel.epoll.EpollEventLoop@f79a760) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/realms/alpha/authenticate headers: X-OpenAM-Username: policyAdmin Content-Type: application/json Accept-API-Version: resource=2.0, protocol=1.0 X-OpenAM-Password: Pa_ssw0rd user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_5_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 accept: */* host: openam-perf-haproxy.forgeblocks.com content-length: 0 ========================= HTTP response: version: HTTP/1.1 status: 401 Unauthorized headers: x-frame-options: SAMEORIGIN content-security-policy-report-only: frame-ancestors 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline' x-content-type-options: nosniff cache-control: private content-api-version: resource=2.1 expires: 0 pragma: no-cache content-type: application/json Content-Length: 62 date: Tue, 16 Sep 2025 06:35:48 GMT x-forgerock-transactionid: 6c9253f7-6fc2-44bc-a845-4db5ccd85881 strict-transport-security: max-age=31536000; includeSubDomains; preload; x-robots-tag: none Via: 1.1 google Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 body: {"code":401,"reason":"Unauthorized","message":"Login failure"} <<<<<<<<<<<<<<<<<<<<<<<<< 07:35:48.831 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-8' crashed with 'j.u.NoSuchElementException: No attribute named 'policyEvalTokenId' is defined', forwarding to the next one 07:35:49.624 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:49.624 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:49.637 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:49.638 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:49.747 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:49.824 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:49.825 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.025 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.025 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.225 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.226 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.344 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:50.427 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.427 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.552 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:50.626 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.627 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.730 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.731 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.839 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:50.842 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.842 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.953 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.953 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.064 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.065 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.175 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.176 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one [...] 07:36:02.639 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.659 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.659 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.679 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.680 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.696 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:02.700 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.700 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.721 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.721 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.736 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:02.741 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.741 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.761 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.761 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.778 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:02.782 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.782 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.802 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.802 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.823 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.823 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.843 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.843 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.863 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.863 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.884 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.884 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.895 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:02.904 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.904 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.925 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.925 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.945 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.945 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.965 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.965 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.986 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:02.986 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:02.986 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:03.006 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.006 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.027 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.027 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.047 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.047 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.067 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.067 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.069 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:03.088 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.088 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.108 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.108 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.129 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.129 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.149 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.149 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.169 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:03.170 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.170 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.190 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.190 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.199 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:03.210 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:03.210 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.210 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.231 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.231 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.251 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.251 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.272 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.272 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.274 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:03.286 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:03.292 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.292 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.312 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:36:03.312 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:36:03.317 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:36:03.333 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 2025-09-16 07:36:04,829 DEBUG --- stderr --- 2025-09-16 07:36:04,829 DEBUG 2025-09-16 07:36:04,829 INFO 2025-09-16 07:36:04,829 INFO 2025-09-16 07:36:04,829 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- tail -n 1000 /results/orders/benchmark-100/stdout.txt 2025-09-16 07:36:05,564 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2025-09-16 07:36:05,564 DEBUG --- stdout --- 2025-09-16 07:36:05,565 DEBUG [print_head_tail]: Print head (80 lines) and tail (80 lines) of input string (1000 lines): 2025-09-16 07:36:05,565 DEBUG ----- output ----- 08:35:45.290 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.290 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.295 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.295 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.300 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.323 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.351 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.351 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.358 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.358 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.374 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.374 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.380 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.380 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.395 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.395 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.410 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.410 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.430 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.430 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.441 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.441 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.443 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.443 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.451 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.451 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.458 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.458 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.459 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.467 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.467 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.471 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.471 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.471 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.471 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.478 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.478 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.494 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.494 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.511 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.511 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.514 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.514 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.516 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.519 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.519 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.525 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.525 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.538 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.553 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.553 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.563 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.572 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.572 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.574 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.574 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.582 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.588 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.588 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.592 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.592 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.595 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.595 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.597 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.597 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.600 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.600 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.610 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.610 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.626 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.662 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.662 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.664 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.664 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:45.676 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.676 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.681 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.681 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 08:35:45.684 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 08:35:45.684 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one [...] content-length: 0 body:StringChunksRequestBody{charset=UTF-8, content=} ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 08:35:48.617 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: restLoginInitiate: KO j.i.IOException: Premature close ========================= Session: Session(idc.benchmark.Mix2024,358846,HashMap(gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, username -> user.1922617, gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@13d110fa, fake-user-agent -> Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3, password -> Pa_ssw0rd, fake-ip -> 42.42.42.42, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@655ba0da),KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@110df069,io.netty.channel.epoll.EpollEventLoop@4f2c9ba6) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/realms/alpha/authenticate?authIndexType=service&authIndexValue=Login headers: accept: application/json content-type: application/json user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3 X-Forwarded-For: 42.42.42.42 host: openam-perf-haproxy.forgeblocks.com content-length: 0 body:StringChunksRequestBody{charset=UTF-8, content=} ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 08:35:48.617 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: UI AM validateGoto: KO j.i.IOException: Premature close ========================= Session: Session(idc.benchmark.Mix2024,358826,HashMap(gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, username -> user.3356061, gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@54eb935a, gatling.http.referer -> https://openam-perf-haproxy.forgeblocks.com/am/, callbacks -> {"tokenId":"ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..*","successUrl":"/enduser/?realm=/alpha","realm":"/alpha"}, gatling.http.cookies -> CookieJar(Map(CookieKey(amlbcookie,openam-perf-haproxy.forgeblocks.com,/) -> StoredCookie(amlbcookie=01, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None,false,false,1758008148506), CookieKey(d0d18cac672abbc,openam-perf-haproxy.forgeblocks.com,/) -> StoredCookie(d0d18cac672abbc=ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None,false,false,1758008148506))), password -> Pa_ssw0rd, fake-ip -> 253.200.200.111, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@5b9aad3f, gatling.http.cache.contentCache -> io.gatling.core.util.cache.Cache@2e0acf6d, tokenId -> ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, fake-user-agent -> Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36),KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@110df069,io.netty.channel.epoll.EpollEventLoop@2bef51f2) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/realms/alpha/users?_action=validateGoto headers: user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_5_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Accept-API-Version: resource=2.1, protocol=1.0 referer: https://openam-perf-haproxy.forgeblocks.com/am/ accept: */* cookie: amlbcookie=01; d0d18cac672abbc=ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..* origin: https://openam-perf-haproxy.forgeblocks.com host: openam-perf-haproxy.forgeblocks.com content-length: 0 cookies: amlbcookie=01, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None d0d18cac672abbc=ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 08:35:48.617 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:48.617 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-113' crashed with 'j.u.NoSuchElementException: No attribute named 'callbacks' is defined', forwarding to the next one 08:35:48.617 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: UI AM validateGoto: KO j.i.IOException: Premature close ========================= Session: Session(idc.benchmark.Mix2024,358824,HashMap(gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, username -> user.6085278, gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@2c122f49, gatling.http.referer -> https://openam-perf-haproxy.forgeblocks.com/am/, callbacks -> {"tokenId":"PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..*","successUrl":"/enduser/?realm=/alpha","realm":"/alpha"}, gatling.http.cookies -> CookieJar(Map(CookieKey(amlbcookie,openam-perf-haproxy.forgeblocks.com,/) -> StoredCookie(amlbcookie=01, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None,false,false,1758008148506), CookieKey(d0d18cac672abbc,openam-perf-haproxy.forgeblocks.com,/) -> StoredCookie(d0d18cac672abbc=PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None,false,false,1758008148506))), password -> Pa_ssw0rd, fake-ip -> 10.94.105.31, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@1e81e7b2, gatling.http.cache.contentCache -> io.gatling.core.util.cache.Cache@3fd69f2e, tokenId -> PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, fake-user-agent -> Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0),KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@110df069,io.netty.channel.epoll.EpollEventLoop@652a7737) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/realms/alpha/users?_action=validateGoto headers: user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_5_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Accept-API-Version: resource=2.1, protocol=1.0 referer: https://openam-perf-haproxy.forgeblocks.com/am/ accept: */* cookie: amlbcookie=01; d0d18cac672abbc=PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..* origin: https://openam-perf-haproxy.forgeblocks.com host: openam-perf-haproxy.forgeblocks.com content-length: 0 cookies: amlbcookie=01, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None d0d18cac672abbc=PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 2025-09-16 08:35:48,693 INFO Gatling simulation run ended successfully 2025-09-16 08:35:48,693 DEBUG No archive name provided. No archive will be created 2025-09-16 07:36:05,565 DEBUG --- stderr --- 2025-09-16 07:36:05,565 DEBUG 2025-09-16 07:36:05,565 INFO ________________________________________________________________________________ [2025-09-16 07:36:05] benchmark-100 show_cmd : Display order command ________________________________________________________________________________ 2025-09-16 07:36:05,565 INFO 2025-09-16 07:36:05,565 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-09-16 07:36:05,565 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-09-16 07:36:05,863 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-09-16 07:36:05,863 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:05,863 DEBUG --- stdout --- 2025-09-16 07:36:05,863 DEBUG overseer-0-57d79d468-khs5t 2025-09-16 07:36:05,863 DEBUG --- stderr --- 2025-09-16 07:36:05,863 DEBUG Checking if file exists on pod 'overseer-0-57d79d468-khs5t', wait max 3 minutes if not. 2025-09-16 07:36:05,864 INFO 2025-09-16 07:36:05,864 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- bash -c 'if [ -f "/results/orders/benchmark-100/cmd.txt" ]; then echo "/results/orders/benchmark-100/cmd.txt file found"; exit 0; else echo "/results/orders/benchmark-100/cmd.txt file not found"; exit 1; fi' | grep "/results/orders/benchmark-100/cmd.txt file found" 2025-09-16 07:36:05,864 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-09-16 07:36:06,405 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-09-16 07:36:06,405 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:06,405 DEBUG --- stdout --- 2025-09-16 07:36:06,405 DEBUG /results/orders/benchmark-100/cmd.txt file found 2025-09-16 07:36:06,405 DEBUG --- stderr --- 2025-09-16 07:36:06,405 DEBUG 2025-09-16 07:36:06,406 INFO 2025-09-16 07:36:06,406 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-57d79d468-khs5t -- du -k /results/orders/benchmark-100/cmd.txt | grep "^\d+ .+$" 2025-09-16 07:36:06,406 INFO [loop_until]: (max_time=240, interval=10, expected_rc=[0] 2025-09-16 07:36:06,945 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-09-16 07:36:06,945 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:06,945 DEBUG --- stdout --- 2025-09-16 07:36:06,945 DEBUG 4 /results/orders/benchmark-100/cmd.txt 2025-09-16 07:36:06,945 DEBUG --- stderr --- 2025-09-16 07:36:06,945 DEBUG - Download file (size 4kB) source : /results/orders/benchmark-100/cmd.txt @ overseer-0-57d79d468-khs5t destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/cmd.txt 2025-09-16 07:36:06,946 INFO 2025-09-16 07:36:06,946 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-57d79d468-khs5t:/results/orders/benchmark-100/cmd.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/cmd.txt 2025-09-16 07:36:06,946 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-09-16 07:36:07,568 INFO [loop_until]: OK (rc = 0) 2025-09-16 07:36:07,569 DEBUG --- stdout --- 2025-09-16 07:36:07,569 DEBUG 2025-09-16 07:36:07,569 DEBUG --- stderr --- 2025-09-16 07:36:07,569 DEBUG 2025-09-16 07:36:07,569 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/cmd.txt is small (1 lines), printing all lines: 2025-09-16 07:36:07,569 INFO ----- output ----- /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class="com.forgerock.pyrock.Loadtest" --run-description="idc.benchmark.Mix2024@perf-haproxy" --results-folder="/results/gatling/benchmark-100" --no-reports --simulation-jvm-args="-XX:+ExitOnOutOfMemoryError -Xmx4G" --simulation-system-properties="-Dam_host=openam-perf-haproxy.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=M@ythe4th -DamadminUsername=xiaosong.lou@pingidentity.com -Dduration=3600 -Didc=true -Didm_host=openam-perf-haproxy.forgeblocks.com -Dlogin_cookie=d0d18cac672abbc -Dnum_users=10000000 -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idc.benchmark.Mix2024 -DtestMode=open -Dthroughput=100 -Duser_password=Pa_ssw0rd" ________________________________________________________________________________ [2025-09-16 07:36:07] benchmark-100 show_rc : Display rc order ________________________________________________________________________________ 2025-09-16 07:36:07,569 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/rc.txt is small (1 lines), printing all lines: 2025-09-16 07:36:07,569 INFO ----- output ----- CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) ________________________________________________________________________________ [2025-09-16 07:36:07] benchmark-100 show_stdout : Display stdout order ________________________________________________________________________________ 2025-09-16 07:36:07,569 DEBUG [print_head_tail]: Print head (100 lines) and tail (100 lines) of input file (/mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/stdout.txt is 2010 lines): 2025-09-16 07:36:07,570 INFO ----- output ----- NOTE: This file was too big so it was truncated. Only first 1000 and last 1000 lines are present. ------------------------------------------------------------ 2025-09-16 07:35:29,313 INFO pom.xml and Gatling Scala files copied from /home/jenkins/lodestar/pyrock/shared/clients/gatling to temporary folder /tmp/tmp3oyesrwn 2025-09-16 07:35:29,317 INFO Running Gatling simulation with command: mvn gatling:test --batch-mode --quiet -Dgatling.simulationClass="com.forgerock.pyrock.Loadtest" -Dgatling.runDescription="..." -Dgatling.resultsFolder="/results/gatling/benchmark-100" -Dgatling.noReports=true -Dgatling.jvmArgs="-XX:+ExitOnOutOfMemoryError,-Xmx4G" -Dgatling.propagateSystemProperties=true -Dam_host=openam-perf-haproxy.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dduration=3600 -Didc=true -Didm_host=openam-perf-haproxy.forgeblocks.com -Dlogin_cookie=d0d18cac672abbc -Dnum_users=10000000 -Doauth2_client_id=endUserUIClient -Doauth2_redirect_uri=/enduser/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idc.benchmark.Mix2024 -DtestMode=open -Dthroughput=100 -Duser_password=Pa_ssw0rd Gatling 3.13.5 is available! (you're using 3.12.0) System property think_time_multiplier not set. No think time emulation will be done in the simulation. Simulation com.forgerock.pyrock.Loadtest started... 07:35:48.815 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - Request 'authAsPolicyAdmin' failed for user 1: status.find.in([200, 209], 304), found 401 07:35:48.828 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: authAsPolicyAdmin: KO status.find.in([200, 209], 304), found 401 ========================= Session: Session(getToken,1,Map(gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@7f6a1633, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@5e2c04de, gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, e52b66c0-8062-47a8-92b4-f2076e0a6411 -> 0),KO,List(ExitOnCompleteLoopBlock(e52b66c0-8062-47a8-92b4-f2076e0a6411)),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@376f9a0d,io.netty.channel.epoll.EpollEventLoop@f79a760) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/realms/alpha/authenticate headers: X-OpenAM-Username: policyAdmin Content-Type: application/json Accept-API-Version: resource=2.0, protocol=1.0 X-OpenAM-Password: Pa_ssw0rd user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_5_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 accept: */* host: openam-perf-haproxy.forgeblocks.com content-length: 0 ========================= HTTP response: version: HTTP/1.1 status: 401 Unauthorized headers: x-frame-options: SAMEORIGIN content-security-policy-report-only: frame-ancestors 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline' x-content-type-options: nosniff cache-control: private content-api-version: resource=2.1 expires: 0 pragma: no-cache content-type: application/json Content-Length: 62 date: Tue, 16 Sep 2025 06:35:48 GMT x-forgerock-transactionid: 6c9253f7-6fc2-44bc-a845-4db5ccd85881 strict-transport-security: max-age=31536000; includeSubDomains; preload; x-robots-tag: none Via: 1.1 google Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 body: {"code":401,"reason":"Unauthorized","message":"Login failure"} <<<<<<<<<<<<<<<<<<<<<<<<< 07:35:48.831 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-8' crashed with 'j.u.NoSuchElementException: No attribute named 'policyEvalTokenId' is defined', forwarding to the next one 07:35:49.624 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:49.624 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:49.637 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:49.638 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:49.747 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:49.824 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:49.825 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.025 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.025 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.225 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.226 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.344 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:50.427 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.427 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.552 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:50.626 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.627 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.730 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.731 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.839 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:50.842 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.842 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:50.953 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:50.953 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.064 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.065 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.175 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.176 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.287 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.287 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.399 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.399 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.509 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.510 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.619 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.619 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.629 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 07:35:51.709 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.709 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.800 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.800 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.891 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one 07:35:51.891 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-199' crashed with 'j.u.NoSuchElementException: key not found: tokenIdCookieName', forwarding to the next one 07:35:51.982 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-200' crashed with 'j.u.NoSuchElementException: key not found: policyEvalTokenId', forwarding to the next one [...] content-length: 0 ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 08:35:48.617 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: restLoginInitiate: KO j.i.IOException: Premature close ========================= Session: Session(idc.benchmark.Mix2024,358845,HashMap(gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, username -> user.443505, gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@4ae9556e, fake-user-agent -> Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134, password -> Pa_ssw0rd, fake-ip -> 42.42.42.42, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@106ddc70),KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@110df069,io.netty.channel.epoll.EpollEventLoop@1e1d3956) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/authenticate headers: accept: application/json content-type: application/json user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134 X-Forwarded-For: 42.42.42.42 host: openam-perf-haproxy.forgeblocks.com content-length: 0 body:StringChunksRequestBody{charset=UTF-8, content=} ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 08:35:48.617 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: restLoginInitiate: KO j.i.IOException: Premature close ========================= Session: Session(idc.benchmark.Mix2024,358846,HashMap(gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, username -> user.1922617, gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@13d110fa, fake-user-agent -> Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3, password -> Pa_ssw0rd, fake-ip -> 42.42.42.42, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@655ba0da),KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@110df069,io.netty.channel.epoll.EpollEventLoop@4f2c9ba6) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/realms/alpha/authenticate?authIndexType=service&authIndexValue=Login headers: accept: application/json content-type: application/json user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3 X-Forwarded-For: 42.42.42.42 host: openam-perf-haproxy.forgeblocks.com content-length: 0 body:StringChunksRequestBody{charset=UTF-8, content=} ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 08:35:48.617 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: UI AM validateGoto: KO j.i.IOException: Premature close ========================= Session: Session(idc.benchmark.Mix2024,358826,HashMap(gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, username -> user.3356061, gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@54eb935a, gatling.http.referer -> https://openam-perf-haproxy.forgeblocks.com/am/, callbacks -> {"tokenId":"ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..*","successUrl":"/enduser/?realm=/alpha","realm":"/alpha"}, gatling.http.cookies -> CookieJar(Map(CookieKey(amlbcookie,openam-perf-haproxy.forgeblocks.com,/) -> StoredCookie(amlbcookie=01, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None,false,false,1758008148506), CookieKey(d0d18cac672abbc,openam-perf-haproxy.forgeblocks.com,/) -> StoredCookie(d0d18cac672abbc=ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None,false,false,1758008148506))), password -> Pa_ssw0rd, fake-ip -> 253.200.200.111, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@5b9aad3f, gatling.http.cache.contentCache -> io.gatling.core.util.cache.Cache@2e0acf6d, tokenId -> ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, fake-user-agent -> Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36),KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@110df069,io.netty.channel.epoll.EpollEventLoop@2bef51f2) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/realms/alpha/users?_action=validateGoto headers: user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_5_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Accept-API-Version: resource=2.1, protocol=1.0 referer: https://openam-perf-haproxy.forgeblocks.com/am/ accept: */* cookie: amlbcookie=01; d0d18cac672abbc=ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..* origin: https://openam-perf-haproxy.forgeblocks.com host: openam-perf-haproxy.forgeblocks.com content-length: 0 cookies: amlbcookie=01, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None d0d18cac672abbc=ME4pDZsOrTVFcjZy9tnPdgwYBts.*AAJTSQACMDIAAlNLABx0UGVVTENlUVc4b2J6RDN5VndRTWRTNUViZjQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 08:35:48.617 [ERROR] i.g.h.a.HttpRequestAction - Failed to build request EvalPolicy: No attribute named 'tokenIdCookieName' is defined 08:35:48.617 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-113' crashed with 'j.u.NoSuchElementException: No attribute named 'callbacks' is defined', forwarding to the next one 08:35:48.617 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: UI AM validateGoto: KO j.i.IOException: Premature close ========================= Session: Session(idc.benchmark.Mix2024,358824,HashMap(gatling.http.cache.baseUrl -> https://openam-perf-haproxy.forgeblocks.com:443, username -> user.6085278, gatling.http.ssl.sslContexts -> io.gatling.http.util.SslContexts@2c122f49, gatling.http.referer -> https://openam-perf-haproxy.forgeblocks.com/am/, callbacks -> {"tokenId":"PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..*","successUrl":"/enduser/?realm=/alpha","realm":"/alpha"}, gatling.http.cookies -> CookieJar(Map(CookieKey(amlbcookie,openam-perf-haproxy.forgeblocks.com,/) -> StoredCookie(amlbcookie=01, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None,false,false,1758008148506), CookieKey(d0d18cac672abbc,openam-perf-haproxy.forgeblocks.com,/) -> StoredCookie(d0d18cac672abbc=PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None,false,false,1758008148506))), password -> Pa_ssw0rd, fake-ip -> 10.94.105.31, gatling.http.cache.dns -> io.gatling.http.resolver.ShufflingNameResolver@1e81e7b2, gatling.http.cache.contentCache -> io.gatling.core.util.cache.Cache@3fd69f2e, tokenId -> PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, fake-user-agent -> Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0),KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$822/0x00007831286987a0@110df069,io.netty.channel.epoll.EpollEventLoop@652a7737) ========================= HTTP request: POST https://openam-perf-haproxy.forgeblocks.com/am/json/realms/root/realms/alpha/users?_action=validateGoto headers: user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_5_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Accept-API-Version: resource=2.1, protocol=1.0 referer: https://openam-perf-haproxy.forgeblocks.com/am/ accept: */* cookie: amlbcookie=01; d0d18cac672abbc=PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..* origin: https://openam-perf-haproxy.forgeblocks.com host: openam-perf-haproxy.forgeblocks.com content-length: 0 cookies: amlbcookie=01, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None d0d18cac672abbc=PRVeNc712f2iRS5i5w3wRwFoiUo.*AAJTSQACMDIAAlNLABxsVlJsTlFPbVBDZDJTRVp0M2YrSmdTR21vc1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..*, domain=openam-perf-haproxy.forgeblocks.com, path=/, secure, HTTPOnly, SameSite=None ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 2025-09-16 08:35:48,693 INFO Gatling simulation run ended successfully 2025-09-16 08:35:48,693 DEBUG No archive name provided. No archive will be created ________________________________________________________________________________ [2025-09-16 07:36:07] benchmark-100 show_stderr : Display stderr order ________________________________________________________________________________ 2025-09-16 07:36:07,570 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/07_Scenario_benchmark/02_benchmark-100/stderr.txt is small (0 lines), printing all lines: 2025-09-16 07:36:07,570 INFO ----- output ----- ________________________________________________________________________________ [2025-09-16 07:36:07] benchmark-100 check_rc_and_logs : Check order's outputs ________________________________________________________________________________ CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) PASS: no error found in rc log PASS: no error found in stdout log PASS: no error found in stderr log ________________________________________________________________________________ [2025-09-16 07:36:07] benchmark-100 print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Task result is INIT no need to display logs. ________________________________________________________________________________ [2025-09-16 07:36:07] benchmark-100 post : Post method ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped