--Task-- name: mixedquery-10000 enabled: True class_name: GatlingTask source_name: overseer-0 source_namespace: >default< target_name: idm target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates below loop: False interval: None dependencies: [] wait_for: ['mixedquery-8000'] preceding_task: mixedquery-8000 options: {'testMode': 'open', 'testCase': 'idm.MixedQuery', 'duration': 1800, 'throughput': 10000, 'java-opts': '-Dcsvfile=/results/idm/user_feeder.csv -Drealm=alpha'} group_name: None Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock 2025-03-28 00:53:29,478 INFO 2025-03-28 00:53:29,478 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-03-28 00:53:29,478 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-03-28 00:53:29,757 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-03-28 00:53:29,758 INFO [loop_until]: OK (rc = 0) 2025-03-28 00:53:29,758 DEBUG --- stdout --- 2025-03-28 00:53:29,758 DEBUG overseer-0-59c948f569-rgrkv 2025-03-28 00:53:29,758 DEBUG --- stderr --- 2025-03-28 00:53:29,758 DEBUG 2025-03-28 00:53:29,758 INFO Lodestar assumed that full stack is deployed so will set all products host URLs in Gatling system properties 2025-03-28 00:53:29 - INFO: Timeout for this Task has changed to 36m (2160 seconds) 2025-03-28 00:53:29 - INFO: timeout was calculated based on simulation duration parameter Pod log file : /results/orders/mixedquery-10000/stdout.txt Pod log : /results/orders/mixedquery-10000 ________________________________________________________________________________ [2025-03-28 00:53:29] mixedquery-10000 print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ /lodestar/lodestar.py perf run-gatling --simulation-class="com.forgerock.pyrock.Loadtest" --run-description="idm.MixedQuery@perf-200m" --results-folder="/results/gatling/mixedquery-10000" --no-reports --simulation-jvm-args="-XX:+ExitOnOutOfMemoryError -Xmx8G" --simulation-system-properties="-Dam_host=openam-perf-200m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcsvfile=/results/idm/user_feeder.csv -Dduration=1800 -Didc=true -Didm_host=openam-perf-200m.forgeblocks.com -Dlogin_cookie=15892ffd844ef80 -Dnum_users=50000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idm.MixedQuery -DtestMode=open -Dthroughput=10000 -Duser_password=**** ________________________________________________________________________________ [2025-03-28 00:53:29] mixedquery-10000 create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file: /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/order.json { "name": "mixedquery-10000", "command": " /lodestar/lodestar.py perf run-gatling --simulation-class=\"com.forgerock.pyrock.Loadtest\" --run-description=\"idm.MixedQuery@perf-200m\" --results-folder=\"/results/gatling/mixedquery-10000\" --no-reports --simulation-jvm-args=\"-XX:+ExitOnOutOfMemoryError -Xmx8G\" --simulation-system-properties=\"-Dam_host=openam-perf-200m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcsvfile=/results/idm/user_feeder.csv -Dduration=1800 -Didc=true -Didm_host=openam-perf-200m.forgeblocks.com -Dlogin_cookie=15892ffd844ef80 -Dnum_users=50000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idm.MixedQuery -DtestMode=open -Dthroughput=10000 -Duser_password=T35!tr0c-k123\"", "out-dir": "/results/orders/mixedquery-10000" } ________________________________________________________________________________ [2025-03-28 00:53:29] mixedquery-10000 loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2025-03-28 00:53:29,765 INFO 2025-03-28 00:53:29,765 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2025-03-28 00:53:29,765 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2025-03-28 00:53:29,980 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-03-28 00:53:29,980 INFO [loop_until]: OK (rc = 0) 2025-03-28 00:53:29,980 DEBUG --- stdout --- 2025-03-28 00:53:29,980 DEBUG WAITING 2025-03-28 00:53:29,980 DEBUG --- stderr --- 2025-03-28 00:53:29,980 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 33 0 --:--:-- --:--:-- --:--:-- 33 ________________________________________________________________________________ [2025-03-28 00:53:29] mixedquery-10000 start_order : Send the order to the overseer pod ________________________________________________________________________________ 2025-03-28 00:53:29,980 INFO 2025-03-28 00:53:29,980 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-03-28 00:53:29,980 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-03-28 00:53:30,261 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-03-28 00:53:30,261 INFO [loop_until]: OK (rc = 0) 2025-03-28 00:53:30,261 DEBUG --- stdout --- 2025-03-28 00:53:30,261 DEBUG overseer-0-59c948f569-rgrkv 2025-03-28 00:53:30,261 DEBUG --- stderr --- 2025-03-28 00:53:30,261 DEBUG 2025-03-28 00:53:30,262 INFO 2025-03-28 00:53:30,262 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/order.json overseer-0-59c948f569-rgrkv:/results/orders/order.json 2025-03-28 00:53:30,262 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-03-28 00:53:31,634 INFO [loop_until]: OK (rc = 0) 2025-03-28 00:53:31,634 DEBUG --- stdout --- 2025-03-28 00:53:31,634 DEBUG 2025-03-28 00:53:31,634 DEBUG --- stderr --- 2025-03-28 00:53:31,634 DEBUG Order has been successfully forwarded to overseer 2025-03-28 00:53:31 - INFO: Check if Gatling simulation started successfully 2025-03-28 00:53:31,634 INFO 2025-03-28 00:53:31,634 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv -- head --lines 5000 /results/orders/mixedquery-10000/stdout.txt | grep "Simulation com.forgerock.pyrock.Loadtest started..." 2025-03-28 00:53:31,634 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2025-03-28 00:53:32,369 INFO [loop_until]: Function succeeded after 0s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-03-28 00:53:38,105 INFO [loop_until]: Function succeeded after 6s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-03-28 00:53:43,864 INFO [loop_until]: Function succeeded after 12s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-03-28 00:53:49,602 INFO [loop_until]: Function succeeded after 17s (rc=0) - expected pattern found 2025-03-28 00:53:49,602 INFO [loop_until]: OK (rc = 0) 2025-03-28 00:53:49,602 DEBUG --- stdout --- 2025-03-28 00:53:49,602 DEBUG 2025-03-28 00:53:32,325 DEBUG Creating results folder /results/gatling/mixedquery-10000 2025-03-28 00:53:32,333 INFO pom.xml and Gatling Scala files copied from /home/jenkins/lodestar/pyrock/shared/clients/gatling to temporary folder /tmp/tmpa4p8w_t3 2025-03-28 00:53:32,336 INFO Running Gatling simulation with command: mvn gatling:test --batch-mode -Dgatling.simulationClass="com.forgerock.pyrock.Loadtest" -Dgatling.runDescription="..." -Dgatling.resultsFolder="/results/gatling/mixedquery-10000" -Dgatling.noReports=true -Dgatling.jvmArgs="-XX:+ExitOnOutOfMemoryError,-Xmx8G" -Dgatling.propagateSystemProperties=true -Dam_host=openam-perf-200m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcsvfile=/results/idm/user_feeder.csv -Dduration=1800 -Didc=true -Didm_host=openam-perf-200m.forgeblocks.com -Dlogin_cookie=15892ffd844ef80 -Dnum_users=50000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idm.MixedQuery -DtestMode=open -Dthroughput=10000 -Duser_password=**** [INFO] Scanning for projects... [INFO] [INFO] -----------< com.forgerock.pyrock:pyrock-gatling-simulation >----------- [INFO] Building pyrock-gatling-simulation 3.12.0 [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] >>> gatling:4.9.6:test (default-cli) > test-compile @ pyrock-gatling-simulation >>> [INFO] [INFO] --- resources:3.3.1:resources (default-resources) @ pyrock-gatling-simulation --- [INFO] skip non existing resourceDirectory /tmp/tmpa4p8w_t3/src/main/resources [INFO] [INFO] --- compiler:3.13.0:compile (default-compile) @ pyrock-gatling-simulation --- [INFO] No sources to compile [INFO] [INFO] --- resources:3.3.1:testResources (default-testResources) @ pyrock-gatling-simulation --- [INFO] Copying 2 resources from src/test/resources to target/test-classes [INFO] [INFO] --- compiler:3.13.0:testCompile (default-testCompile) @ pyrock-gatling-simulation --- [INFO] Nothing to compile - all classes are up to date. [INFO] [INFO] --- scala:4.9.2:testCompile (default) @ pyrock-gatling-simulation --- [INFO] compiling 38 Scala sources to /tmp/tmpa4p8w_t3/target/test-classes ... [INFO] compile in 8.3 s [INFO] [INFO] <<< gatling:4.9.6:test (default-cli) < test-compile @ pyrock-gatling-simulation <<< [INFO] [INFO] [INFO] --- gatling:4.9.6:test (default-cli) @ pyrock-gatling-simulation --- [INFO] Running simulation com.forgerock.pyrock.Loadtest. Gatling 3.13.5 is available! (you're using 3.12.0) System property think_time_multiplier not set. No think time emulation will be done in the simulation. Set idVarName to id Set idVarName to id Simulation com.forgerock.pyrock.Loadtest started... 2025-03-28 00:53:49,602 DEBUG --- stderr --- 2025-03-28 00:53:49,602 DEBUG ________________________________________________________________________________ [2025-03-28 00:53:49] mixedquery-10000 wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 36m. Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING Order is running since 5s, retry Order is running since 10s, retry Order is running since 15s, retry Order is running since 20s, retry Order is running since 26s, retry Order is running since 31s, retry Order is running since 36s, retry Order is running since 42s, retry Order is running since 47s, retry Order is running since 52s, retry Order is running since 57s, retry Order is running since 1m 02s, retry Order is running since 1m 08s, retry Order is running since 1m 13s, retry Order is running since 1m 18s, retry Order is running since 1m 23s, retry Order is running since 1m 29s, retry Order is running since 1m 34s, retry Order is running since 1m 39s, retry Order is running since 1m 45s, retry Order is running since 1m 50s, retry Order is running since 1m 55s, retry Order is running since 2m, retry Order is running since 2m 06s, retry Order is running since 2m 11s, retry Order is running since 2m 16s, retry Order is running since 2m 21s, retry Order is running since 2m 26s, retry Order is running since 2m 32s, retry Order is running since 2m 37s, retry Order is running since 2m 42s, retry Order is running since 2m 48s, retry Order is running since 2m 53s, retry Order is running since 2m 58s, retry Order is running since 3m 03s, retry Order is running since 3m 09s, retry Order is running since 3m 14s, retry Order is running since 3m 19s, retry Order is running since 3m 24s, retry Order is running since 3m 29s, retry Order is running since 3m 35s, retry Order is running since 3m 40s, retry Order is running since 3m 45s, retry Order is running since 3m 51s, retry Order is running since 3m 56s, retry Order is running since 4m 01s, retry Order is running since 4m 06s, retry Order is running since 4m 11s, retry Order is running since 4m 17s, retry Order is running since 4m 22s, retry Order is running since 4m 27s, retry Order is running since 4m 32s, retry Order is running since 4m 38s, retry Order is running since 4m 43s, retry Order is running since 4m 48s, retry Order is running since 4m 53s, retry Order is running since 4m 59s, retry Order is running since 5m 04s, retry Order is running since 5m 09s, retry Order is running since 5m 14s, retry Order is running since 5m 20s, retry Order is running since 5m 25s, retry Order is running since 5m 30s, retry Order is running since 5m 35s, retry Order is running since 5m 41s, retry Order is running since 5m 46s, retry Order is running since 5m 51s, retry Order is running since 5m 56s, retry Order is running since 6m 02s, retry Order is running since 6m 07s, retry Order is running since 6m 12s, retry Order is running since 6m 17s, retry Order is running since 6m 23s, retry Order is running since 6m 28s, retry Order is running since 6m 33s, retry Order is running since 6m 38s, retry Order is running since 6m 44s, retry Order is running since 6m 49s, retry Order is running since 6m 54s, retry Order is running since 6m 59s, retry Order is running since 7m 05s, retry Order is running since 7m 10s, retry Order is running since 7m 15s, retry Order is running since 7m 20s, retry Order is running since 7m 26s, retry Order is running since 7m 31s, retry Order is running since 7m 36s, retry Order is running since 7m 41s, retry Order is running since 7m 47s, retry Order is running since 7m 52s, retry Order is running since 7m 57s, retry Order is running since 8m 02s, retry Order is running since 8m 08s, retry Order is running since 8m 13s, retry Order is running since 8m 18s, retry Order is running since 8m 23s, retry Order is running since 8m 29s, retry Order is running since 8m 34s, retry Order is running since 8m 39s, retry Order is running since 8m 44s, retry Order is running since 8m 50s, retry Order is running since 8m 55s, retry Order is running since 9m, retry Order is running since 9m 05s, retry Order is running since 9m 11s, retry Order is running since 9m 16s, retry Order is running since 9m 21s, retry Order is running since 9m 26s, retry Order is running since 9m 31s, retry Order is running since 9m 37s, retry Order is running since 9m 42s, retry Order is running since 9m 47s, retry Order is running since 9m 52s, retry Order is running since 9m 58s, retry Order is running since 10m 03s, retry Order is running since 10m 08s, retry Order is running since 10m 13s, retry Order is running since 10m 19s, retry Order is running since 10m 24s, retry Order is running since 10m 29s, retry Order is running since 10m 34s, retry Order is running since 10m 40s, retry Order is running since 10m 45s, retry Order is running since 10m 50s, retry Order is running since 10m 55s, retry Order is running since 11m 01s, retry Order is running since 11m 06s, retry Order is running since 11m 11s, retry Order is running since 11m 16s, retry Order is running since 11m 22s, retry Order is running since 11m 27s, retry Order is running since 11m 32s, retry Order is running since 11m 37s, retry Order is running since 11m 43s, retry Order is running since 11m 48s, retry Order is running since 11m 53s, retry Order is running since 11m 58s, retry Order is running since 12m 04s, retry Order is running since 12m 09s, retry Order is running since 12m 14s, retry Order is running since 12m 19s, retry Order is running since 12m 25s, retry Order is running since 12m 30s, retry Order is running since 12m 35s, retry Order is running since 12m 40s, retry Order is running since 12m 46s, retry Order is running since 12m 51s, retry Order is running since 12m 56s, retry Order is running since 13m 01s, retry Order is running since 13m 07s, retry Order is running since 13m 12s, retry Order is running since 13m 17s, retry Order is running since 13m 22s, retry Order is running since 13m 27s, retry Order is running since 13m 33s, retry Order is running since 13m 38s, retry Order is running since 13m 43s, retry Order is running since 13m 48s, retry Order is running since 13m 54s, retry Order is running since 13m 59s, retry Order is running since 14m 04s, retry Order is running since 14m 09s, retry Order is running since 14m 15s, retry Order is running since 14m 20s, retry Order is running since 14m 25s, retry Order is running since 14m 30s, retry Order is running since 14m 36s, retry Order is running since 14m 41s, retry Order is running since 14m 46s, retry Order is running since 14m 51s, retry Order is running since 14m 57s, retry Order is running since 15m 02s, retry Order is running since 15m 07s, retry Order is running since 15m 12s, retry Order is running since 15m 18s, retry Order is running since 15m 23s, retry Order is running since 15m 28s, retry Order is running since 15m 33s, retry Order is running since 15m 39s, retry Order is running since 15m 44s, retry Order is running since 15m 49s, retry Order is running since 15m 54s, retry Order is running since 16m, retry Order is running since 16m 05s, retry Order is running since 16m 10s, retry Order is running since 16m 15s, retry Order is running since 16m 20s, retry Order is running since 16m 26s, retry Order is running since 16m 31s, retry Order is running since 16m 36s, retry Order is running since 16m 42s, retry Order is running since 16m 47s, retry Order is running since 16m 52s, retry Order is running since 16m 57s, retry Order is running since 17m 03s, retry Order is running since 17m 08s, retry Order is running since 17m 13s, retry Order is running since 17m 18s, retry Order is running since 17m 23s, retry Order is running since 17m 29s, retry Order is running since 17m 34s, retry Order is running since 17m 39s, retry Order is running since 17m 44s, retry Order is running since 17m 50s, retry Order is running since 17m 55s, retry Order is running since 18m, retry Order is running since 18m 05s, retry Order is running since 18m 11s, retry Order is running since 18m 16s, retry Order is running since 18m 21s, retry Order is running since 18m 26s, retry Order is running since 18m 32s, retry Order is running since 18m 37s, retry Order is running since 18m 42s, retry Order is running since 18m 47s, retry Order is running since 18m 53s, retry Order is running since 18m 58s, retry Order is running since 19m 03s, retry Order is running since 19m 08s, retry Order is running since 19m 14s, retry Order is running since 19m 19s, retry Order is running since 19m 24s, retry Order is running since 19m 29s, retry Order is running since 19m 35s, retry Order is running since 19m 40s, retry Order is running since 19m 45s, retry Order is running since 19m 50s, retry Order is running since 19m 56s, retry Order is running since 20m 01s, retry Order is running since 20m 06s, retry Order is running since 20m 11s, retry Order is running since 20m 17s, retry Order is running since 20m 22s, retry Order is running since 20m 27s, retry Order is running since 20m 32s, retry Order is running since 20m 38s, retry Order is running since 20m 43s, retry Order is running since 20m 48s, retry Order is running since 20m 53s, retry Order is running since 20m 59s, retry Order is running since 21m 04s, retry Order is running since 21m 09s, retry Order is running since 21m 14s, retry Order is running since 21m 20s, retry Order is running since 21m 25s, retry Order is running since 21m 30s, retry Order is running since 21m 35s, retry Order is running since 21m 41s, retry Order is running since 21m 46s, retry Order is running since 21m 51s, retry Order is running since 21m 56s, retry Order is running since 22m 02s, retry Order is running since 22m 07s, retry Order is running since 22m 12s, retry Order is running since 22m 17s, retry Order is running since 22m 22s, retry Order is running since 22m 28s, retry Order is running since 22m 33s, retry Order is running since 22m 38s, retry Order is running since 22m 43s, retry Order is running since 22m 49s, retry Order is running since 22m 54s, retry Order is running since 22m 59s, retry Order is running since 23m 04s, retry Order is running since 23m 10s, retry Order is running since 23m 15s, retry Order is running since 23m 20s, retry Order is running since 23m 25s, retry Order is running since 23m 31s, retry Order is running since 23m 36s, retry Order is running since 23m 41s, retry Order is running since 23m 46s, retry Order is running since 23m 52s, retry Order is running since 23m 57s, retry Order is running since 24m 02s, retry Order is running since 24m 07s, retry Order is running since 24m 13s, retry Order is running since 24m 18s, retry Order is running since 24m 23s, retry Order is running since 24m 28s, retry Order is running since 24m 34s, retry Order is running since 24m 39s, retry Order is running since 24m 44s, retry Order is running since 24m 49s, retry Order is running since 24m 55s, retry Order is running since 25m, retry Order is running since 25m 05s, retry Order is running since 25m 10s, retry Order is running since 25m 16s, retry Order is running since 25m 21s, retry Order is running since 25m 26s, retry Order is running since 25m 31s, retry Order is running since 25m 37s, retry Order is running since 25m 42s, retry Order is running since 25m 47s, retry Order is running since 25m 52s, retry Order is running since 25m 58s, retry Order is running since 26m 03s, retry Order is running since 26m 08s, retry Order is running since 26m 13s, retry Order is running since 26m 19s, retry Order is running since 26m 24s, retry Order is running since 26m 29s, retry Order is running since 26m 34s, retry Order is running since 26m 40s, retry Order is running since 26m 45s, retry Order is running since 26m 50s, retry Order is running since 26m 55s, retry Order is running since 27m 01s, retry Order is running since 27m 06s, retry Order is running since 27m 11s, retry Order is running since 27m 16s, retry Order is running since 27m 22s, retry Order is running since 27m 27s, retry Order is running since 27m 32s, retry Order is running since 27m 37s, retry Order is running since 27m 42s, retry Order is running since 27m 48s, retry Order is running since 27m 53s, retry Order is running since 27m 58s, retry Order is running since 28m 03s, retry Order is running since 28m 09s, retry Order is running since 28m 14s, retry Order is running since 28m 19s, retry Order is running since 28m 24s, retry Order is running since 28m 30s, retry Order is running since 28m 35s, retry Order is running since 28m 40s, retry Order is running since 28m 45s, retry Order is running since 28m 51s, retry Order is running since 28m 56s, retry Order is running since 29m 01s, retry Order is running since 29m 06s, retry Order is running since 29m 12s, retry Order is running since 29m 17s, retry Order is running since 29m 22s, retry Order is running since 29m 27s, retry Order is running since 29m 33s, retry Order is running since 29m 38s, retry Order is running since 29m 43s, retry Order is running since 29m 48s, retry Order is running since 29m 54s, retry PASS : Order stopped by itself after 29m 59s and before timeout (36m) ________________________________________________________________________________ [2025-03-28 01:23:48] mixedquery-10000 stop_order : Stop the order on overseer pod ________________________________________________________________________________ 2025-03-28 01:23:48,903 INFO 2025-03-28 01:23:48,903 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-03-28 01:23:48,903 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-03-28 01:23:49,239 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-03-28 01:23:49,239 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:49,239 DEBUG --- stdout --- 2025-03-28 01:23:49,239 DEBUG overseer-0-59c948f569-rgrkv 2025-03-28 01:23:49,239 DEBUG --- stderr --- 2025-03-28 01:23:49,239 DEBUG 2025-03-28 01:23:49,240 INFO 2025-03-28 01:23:49,240 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv -- rm -f /results/orders/order.json 2025-03-28 01:23:49,240 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-03-28 01:23:50,054 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:50,054 DEBUG --- stdout --- 2025-03-28 01:23:50,054 DEBUG 2025-03-28 01:23:50,054 DEBUG --- stderr --- 2025-03-28 01:23:50,054 DEBUG Order ran during 30m 18s ________________________________________________________________________________ [2025-03-28 01:23:50] mixedquery-10000 download_logs : Download individual order files ________________________________________________________________________________ 2025-03-28 01:23:50 - PASS: file already downloaded /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/order.json 2025-03-28 01:23:50,055 INFO 2025-03-28 01:23:50,055 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-03-28 01:23:50,055 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-03-28 01:23:50,334 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-03-28 01:23:50,334 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:50,334 DEBUG --- stdout --- 2025-03-28 01:23:50,334 DEBUG overseer-0-59c948f569-rgrkv 2025-03-28 01:23:50,334 DEBUG --- stderr --- 2025-03-28 01:23:50,334 DEBUG Checking pod: overseer-0-59c948f569-rgrkv for file exists, if not wait. 2025-03-28 01:23:50,335 INFO 2025-03-28 01:23:50,335 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv --stdin -- bash -c 'if [ -f "/results/orders/mixedquery-10000/rc.txt" ]; then echo "/results/orders/mixedquery-10000/rc.txt file found"; exit 0; else echo "/results/orders/mixedquery-10000/rc.txt file not found"; exit 1; fi' | grep "/results/orders/mixedquery-10000/rc.txt file found" 2025-03-28 01:23:50,335 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-03-28 01:23:51,137 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-03-28 01:23:51,137 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:51,137 DEBUG --- stdout --- 2025-03-28 01:23:51,137 DEBUG /results/orders/mixedquery-10000/rc.txt file found 2025-03-28 01:23:51,137 DEBUG --- stderr --- 2025-03-28 01:23:51,137 DEBUG 2025-03-28 01:23:51,137 INFO 2025-03-28 01:23:51,138 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv --stdin -- du -k /results/orders/mixedquery-10000/rc.txt 2025-03-28 01:23:51,938 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2025-03-28 01:23:51,938 DEBUG --- stdout --- 2025-03-28 01:23:51,938 DEBUG 4 /results/orders/mixedquery-10000/rc.txt 2025-03-28 01:23:51,938 DEBUG --- stderr --- 2025-03-28 01:23:51,938 DEBUG 2025-03-28 01:23:51,938 INFO - Download file (size 4k) source : /results/orders/mixedquery-10000/rc.txt @ overseer-0-59c948f569-rgrkv destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/rc.txt 2025-03-28 01:23:51,939 INFO 2025-03-28 01:23:51,939 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-59c948f569-rgrkv:/results/orders/mixedquery-10000/rc.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/rc.txt 2025-03-28 01:23:51,939 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-03-28 01:23:52,666 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:52,666 DEBUG --- stdout --- 2025-03-28 01:23:52,666 DEBUG tar: Removing leading `/' from member names 2025-03-28 01:23:52,666 DEBUG --- stderr --- 2025-03-28 01:23:52,666 DEBUG 2025-03-28 01:23:52,667 INFO 2025-03-28 01:23:52,667 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-03-28 01:23:52,667 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-03-28 01:23:52,946 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-03-28 01:23:52,946 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:52,946 DEBUG --- stdout --- 2025-03-28 01:23:52,946 DEBUG overseer-0-59c948f569-rgrkv 2025-03-28 01:23:52,946 DEBUG --- stderr --- 2025-03-28 01:23:52,946 DEBUG Checking pod: overseer-0-59c948f569-rgrkv for file exists, if not wait. 2025-03-28 01:23:52,947 INFO 2025-03-28 01:23:52,947 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv --stdin -- bash -c 'if [ -f "/results/orders/mixedquery-10000/stderr.txt" ]; then echo "/results/orders/mixedquery-10000/stderr.txt file found"; exit 0; else echo "/results/orders/mixedquery-10000/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/mixedquery-10000/stderr.txt file found" 2025-03-28 01:23:52,947 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-03-28 01:23:53,749 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-03-28 01:23:53,749 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:53,749 DEBUG --- stdout --- 2025-03-28 01:23:53,749 DEBUG /results/orders/mixedquery-10000/stderr.txt file found 2025-03-28 01:23:53,749 DEBUG --- stderr --- 2025-03-28 01:23:53,749 DEBUG 2025-03-28 01:23:53,749 INFO 2025-03-28 01:23:53,750 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv --stdin -- du -k /results/orders/mixedquery-10000/stderr.txt 2025-03-28 01:23:54,554 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2025-03-28 01:23:54,554 DEBUG --- stdout --- 2025-03-28 01:23:54,554 DEBUG 4 /results/orders/mixedquery-10000/stderr.txt 2025-03-28 01:23:54,555 DEBUG --- stderr --- 2025-03-28 01:23:54,555 DEBUG 2025-03-28 01:23:54,555 INFO - Download file (size 4k) source : /results/orders/mixedquery-10000/stderr.txt @ overseer-0-59c948f569-rgrkv destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/stderr.txt 2025-03-28 01:23:54,555 INFO 2025-03-28 01:23:54,555 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-59c948f569-rgrkv:/results/orders/mixedquery-10000/stderr.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/stderr.txt 2025-03-28 01:23:54,555 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-03-28 01:23:55,358 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:55,358 DEBUG --- stdout --- 2025-03-28 01:23:55,358 DEBUG tar: Removing leading `/' from member names 2025-03-28 01:23:55,358 DEBUG --- stderr --- 2025-03-28 01:23:55,358 DEBUG 2025-03-28 01:23:55,359 INFO 2025-03-28 01:23:55,359 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-03-28 01:23:55,359 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-03-28 01:23:55,660 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-03-28 01:23:55,660 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:55,660 DEBUG --- stdout --- 2025-03-28 01:23:55,660 DEBUG overseer-0-59c948f569-rgrkv 2025-03-28 01:23:55,660 DEBUG --- stderr --- 2025-03-28 01:23:55,660 DEBUG Checking pod: overseer-0-59c948f569-rgrkv for file exists, if not wait. 2025-03-28 01:23:55,661 INFO 2025-03-28 01:23:55,661 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv --stdin -- bash -c 'if [ -f "/results/orders/mixedquery-10000/stdout.txt" ]; then echo "/results/orders/mixedquery-10000/stdout.txt file found"; exit 0; else echo "/results/orders/mixedquery-10000/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/mixedquery-10000/stdout.txt file found" 2025-03-28 01:23:55,661 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-03-28 01:23:56,475 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-03-28 01:23:56,475 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:56,475 DEBUG --- stdout --- 2025-03-28 01:23:56,475 DEBUG /results/orders/mixedquery-10000/stdout.txt file found 2025-03-28 01:23:56,475 DEBUG --- stderr --- 2025-03-28 01:23:56,476 DEBUG 2025-03-28 01:23:56,476 INFO 2025-03-28 01:23:56,476 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv --stdin -- du -k /results/orders/mixedquery-10000/stdout.txt 2025-03-28 01:23:57,286 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2025-03-28 01:23:57,286 DEBUG --- stdout --- 2025-03-28 01:23:57,286 DEBUG 2636 /results/orders/mixedquery-10000/stdout.txt 2025-03-28 01:23:57,286 DEBUG --- stderr --- 2025-03-28 01:23:57,286 DEBUG 2025-03-28 01:23:57,286 INFO - Download file (size 2636k) source : /results/orders/mixedquery-10000/stdout.txt @ overseer-0-59c948f569-rgrkv destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/stdout.txt 2025-03-28 01:23:57,287 INFO 2025-03-28 01:23:57,287 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-59c948f569-rgrkv:/results/orders/mixedquery-10000/stdout.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/stdout.txt 2025-03-28 01:23:57,287 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-03-28 01:23:58,500 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:58,500 DEBUG --- stdout --- 2025-03-28 01:23:58,500 DEBUG tar: Removing leading `/' from member names 2025-03-28 01:23:58,500 DEBUG --- stderr --- 2025-03-28 01:23:58,500 DEBUG ________________________________________________________________________________ [2025-03-28 01:23:58] mixedquery-10000 show_cmd : Display order command ________________________________________________________________________________ 2025-03-28 01:23:58,500 INFO 2025-03-28 01:23:58,500 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-03-28 01:23:58,500 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-03-28 01:23:58,779 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-03-28 01:23:58,779 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:58,779 DEBUG --- stdout --- 2025-03-28 01:23:58,779 DEBUG overseer-0-59c948f569-rgrkv 2025-03-28 01:23:58,779 DEBUG --- stderr --- 2025-03-28 01:23:58,779 DEBUG Checking pod: overseer-0-59c948f569-rgrkv for file exists, if not wait. 2025-03-28 01:23:58,780 INFO 2025-03-28 01:23:58,780 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv --stdin -- bash -c 'if [ -f "/results/orders/mixedquery-10000/cmd.txt" ]; then echo "/results/orders/mixedquery-10000/cmd.txt file found"; exit 0; else echo "/results/orders/mixedquery-10000/cmd.txt file not found"; exit 1; fi' | grep "/results/orders/mixedquery-10000/cmd.txt file found" 2025-03-28 01:23:58,780 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-03-28 01:23:59,611 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-03-28 01:23:59,611 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:23:59,611 DEBUG --- stdout --- 2025-03-28 01:23:59,611 DEBUG /results/orders/mixedquery-10000/cmd.txt file found 2025-03-28 01:23:59,611 DEBUG --- stderr --- 2025-03-28 01:23:59,611 DEBUG 2025-03-28 01:23:59,612 INFO 2025-03-28 01:23:59,612 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-59c948f569-rgrkv --stdin -- du -k /results/orders/mixedquery-10000/cmd.txt 2025-03-28 01:24:00,407 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2025-03-28 01:24:00,407 DEBUG --- stdout --- 2025-03-28 01:24:00,407 DEBUG 4 /results/orders/mixedquery-10000/cmd.txt 2025-03-28 01:24:00,407 DEBUG --- stderr --- 2025-03-28 01:24:00,407 DEBUG 2025-03-28 01:24:00,407 INFO - Download file (size 4k) source : /results/orders/mixedquery-10000/cmd.txt @ overseer-0-59c948f569-rgrkv destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/cmd.txt 2025-03-28 01:24:00,408 INFO 2025-03-28 01:24:00,408 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-59c948f569-rgrkv:/results/orders/mixedquery-10000/cmd.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/cmd.txt 2025-03-28 01:24:00,408 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-03-28 01:24:01,142 INFO [loop_until]: OK (rc = 0) 2025-03-28 01:24:01,142 DEBUG --- stdout --- 2025-03-28 01:24:01,142 DEBUG tar: Removing leading `/' from member names 2025-03-28 01:24:01,142 DEBUG --- stderr --- 2025-03-28 01:24:01,142 DEBUG 2025-03-28 01:24:01,142 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/cmd.txt is small (1 lines), printing all lines: 2025-03-28 01:24:01,142 INFO ----- output ----- /lodestar/lodestar.py perf run-gatling --simulation-class="com.forgerock.pyrock.Loadtest" --run-description="idm.MixedQuery@perf-200m" --results-folder="/results/gatling/mixedquery-10000" --no-reports --simulation-jvm-args="-XX:+ExitOnOutOfMemoryError -Xmx8G" --simulation-system-properties="-Dam_host=openam-perf-200m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=M@ythe4th -DamadminUsername=xiaosong.lou@pingidentity.com -Dcsvfile=/results/idm/user_feeder.csv -Dduration=1800 -Didc=true -Didm_host=openam-perf-200m.forgeblocks.com -Dlogin_cookie=15892ffd844ef80 -Dnum_users=50000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idm.MixedQuery -DtestMode=open -Dthroughput=10000 -Duser_password=T35!tr0c-k123" ________________________________________________________________________________ [2025-03-28 01:24:01] mixedquery-10000 show_rc : Display rc order ________________________________________________________________________________ 2025-03-28 01:24:01,142 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/rc.txt is small (1 lines), printing all lines: 2025-03-28 01:24:01,142 INFO ----- output ----- CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) ________________________________________________________________________________ [2025-03-28 01:24:01] mixedquery-10000 show_stdout : Display stdout order ________________________________________________________________________________ 2025-03-28 01:24:01,145 DEBUG [print_head_tail]: Print head (100 lines) and tail (100 lines) of input file (/mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/stdout.txt is 19090 lines): 2025-03-28 01:24:01,149 INFO ----- output ----- 2025-03-28 00:53:32,325 DEBUG Creating results folder /results/gatling/mixedquery-10000 2025-03-28 00:53:32,333 INFO pom.xml and Gatling Scala files copied from /home/jenkins/lodestar/pyrock/shared/clients/gatling to temporary folder /tmp/tmpa4p8w_t3 2025-03-28 00:53:32,336 INFO Running Gatling simulation with command: mvn gatling:test --batch-mode -Dgatling.simulationClass="com.forgerock.pyrock.Loadtest" -Dgatling.runDescription="..." -Dgatling.resultsFolder="/results/gatling/mixedquery-10000" -Dgatling.noReports=true -Dgatling.jvmArgs="-XX:+ExitOnOutOfMemoryError,-Xmx8G" -Dgatling.propagateSystemProperties=true -Dam_host=openam-perf-200m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcsvfile=/results/idm/user_feeder.csv -Dduration=1800 -Didc=true -Didm_host=openam-perf-200m.forgeblocks.com -Dlogin_cookie=15892ffd844ef80 -Dnum_users=50000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idm.MixedQuery -DtestMode=open -Dthroughput=10000 -Duser_password=**** [INFO] Scanning for projects... [INFO] [INFO] -----------< com.forgerock.pyrock:pyrock-gatling-simulation >----------- [INFO] Building pyrock-gatling-simulation 3.12.0 [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] >>> gatling:4.9.6:test (default-cli) > test-compile @ pyrock-gatling-simulation >>> [INFO] [INFO] --- resources:3.3.1:resources (default-resources) @ pyrock-gatling-simulation --- [INFO] skip non existing resourceDirectory /tmp/tmpa4p8w_t3/src/main/resources [INFO] [INFO] --- compiler:3.13.0:compile (default-compile) @ pyrock-gatling-simulation --- [INFO] No sources to compile [INFO] [INFO] --- resources:3.3.1:testResources (default-testResources) @ pyrock-gatling-simulation --- [INFO] Copying 2 resources from src/test/resources to target/test-classes [INFO] [INFO] --- compiler:3.13.0:testCompile (default-testCompile) @ pyrock-gatling-simulation --- [INFO] Nothing to compile - all classes are up to date. [INFO] [INFO] --- scala:4.9.2:testCompile (default) @ pyrock-gatling-simulation --- [INFO] compiling 38 Scala sources to /tmp/tmpa4p8w_t3/target/test-classes ... [INFO] compile in 8.3 s [INFO] [INFO] <<< gatling:4.9.6:test (default-cli) < test-compile @ pyrock-gatling-simulation <<< [INFO] [INFO] [INFO] --- gatling:4.9.6:test (default-cli) @ pyrock-gatling-simulation --- [INFO] Running simulation com.forgerock.pyrock.Loadtest. Gatling 3.13.5 is available! (you're using 3.12.0) System property think_time_multiplier not set. No think time emulation will be done in the simulation. Set idVarName to id Set idVarName to id Simulation com.forgerock.pyrock.Loadtest started... ================================================================================ 2025-03-28 00:53:51 GMT 5s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=4092 KO=0 ) > restLoginInitiate (OK=1 KO=0 ) > restLoginUsernamePasswordCallback (OK=1 KO=0 ) > Skip 2FA (OK=1 KO=0 ) > User Authorize PKCE stage (OK=1 KO=0 ) > User AccessToken PKCE stage (OK=1 KO=0 ) > queryFull (OK=2009 KO=0 ) > ReadManagedUser (OK=2078 KO=0 ) ---- idm.MixedQuery ------------------------------------------------------------ [- ] 0% waiting: 18143350 / active: 70 / done: 4087 ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ================================================================================ ================================================================================ 2025-03-28 00:53:56 GMT 10s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=16504 KO=0 ) > restLoginInitiate (OK=1 KO=0 ) > restLoginUsernamePasswordCallback (OK=1 KO=0 ) > Skip 2FA (OK=1 KO=0 ) > User Authorize PKCE stage (OK=1 KO=0 ) > User AccessToken PKCE stage (OK=1 KO=0 ) > queryFull (OK=8120 KO=0 ) > ReadManagedUser (OK=8379 KO=0 ) ---- idm.MixedQuery ------------------------------------------------------------ [- ] 0% waiting: 18130859 / active: 151 / done: 16497 ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ================================================================================ ================================================================================ 2025-03-28 00:54:01 GMT 15s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=37251 KO=0 ) > restLoginInitiate (OK=1 KO=0 ) > restLoginUsernamePasswordCallback (OK=1 KO=0 ) > Skip 2FA (OK=1 KO=0 ) > User Authorize PKCE stage (OK=1 KO=0 ) > User AccessToken PKCE stage (OK=1 KO=0 ) > queryFull (OK=18508 KO=0 ) > ReadManagedUser (OK=18738 KO=0 ) ---- idm.MixedQuery ------------------------------------------------------------ [- ] 0% waiting: 18110036 / active: 225 / done: 37246 ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ================================================================================ [...] host: openam-perf-200m.forgeblocks.com ========================= HTTP response: <<<<<<<<<<<<<<<<<<<<<<<<< 01:23:46.840 [ERROR] i.g.c.a.AtomicRunnableActorRef - Actor csv(user_feeder.csv) crashed java.util.concurrent.RejectedExecutionException: Task io.gatling.core.actor.AtomicRunnableActorRef@2b3b80ea rejected from java.util.concurrent.ThreadPoolExecutor@49a5ee78[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 84855750] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21) at io.gatling.core.actor.AtomicRunnableActorRef.async(ActorSystem.scala:95) at io.gatling.core.actor.AtomicRunnableActorRef.run(ActorSystem.scala:127) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 01:23:46.839 [ERROR] i.g.c.a.AtomicRunnableActorRef - Actor controller crashed java.util.concurrent.RejectedExecutionException: Task io.gatling.core.actor.AtomicRunnableActorRef@6ee8d33f rejected from java.util.concurrent.ThreadPoolExecutor@49a5ee78[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 84855750] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21) at io.gatling.core.actor.AtomicRunnableActorRef.async(ActorSystem.scala:95) at io.gatling.core.actor.AtomicRunnableActorRef.$bang(ActorSystem.scala:70) at io.gatling.core.action.FeedActor.io$gatling$core$action$FeedActor$$$anonfun$init$1(FeedActor.scala:109) at io.gatling.core.action.FeedActor$$anonfun$init$2.apply(FeedActor.scala:92) at io.gatling.core.action.FeedActor$$anonfun$init$2.apply(FeedActor.scala:92) at io.gatling.core.actor.AtomicRunnableActorRef.$anonfun$run$1(ActorSystem.scala:115) at io.netty.util.internal.shaded.org.jctools.queues.MessagePassingQueueUtil.drain(MessagePassingQueueUtil.java:39) at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueue.drain(BaseMpscLinkedArrayQueue.java:612) at io.netty.util.internal.shaded.org.jctools.queues.MpscUnboundedArrayQueue.drain(MpscUnboundedArrayQueue.java:23) at io.gatling.core.actor.AtomicRunnableActorRef.run(ActorSystem.scala:120) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 01:23:46.839 [ERROR] i.g.c.a.AtomicRunnableActorRef - Actor injector crashed java.util.concurrent.RejectedExecutionException: Task io.gatling.core.actor.AtomicRunnableActorRef@1bdf0b5e rejected from java.util.concurrent.ThreadPoolExecutor@49a5ee78[Shutting down, pool size = 7, active threads = 2, queued tasks = 0, completed tasks = 84855750] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21) at io.gatling.core.actor.AtomicRunnableActorRef.async(ActorSystem.scala:95) at io.gatling.core.actor.AtomicRunnableActorRef.$bang(ActorSystem.scala:70) at io.gatling.core.action.Exit.execute(Exit.scala:29) at io.gatling.core.action.Action.$bang(Action.scala:42) at io.gatling.core.action.Action.$bang$(Action.scala:39) at io.gatling.core.action.Exit.$bang(Exit.scala:23) at io.gatling.core.action.If.$anonfun$execute$1(If.scala:44) at io.gatling.core.action.If.$anonfun$execute$1$adapted(If.scala:42) at io.gatling.commons.validation.Success.map(Validation.scala:41) at io.gatling.commons.validation.Validation.map$mcZ$sp(Validation.scala:28) at io.gatling.commons.validation.Validation.map$mcZ$sp$(Validation.scala:28) at io.gatling.commons.validation.Success.map$mcZ$sp(Validation.scala:40) at io.gatling.core.action.If.execute(If.scala:42) at io.gatling.core.action.Action.$bang(Action.scala:42) at io.gatling.core.action.Action.$bang$(Action.scala:39) at io.gatling.core.action.If.io$gatling$core$action$ChainableAction$$super$$bang(If.scala:24) at io.gatling.core.action.ChainableAction.$bang(Action.scala:75) at io.gatling.core.action.ChainableAction.$bang$(Action.scala:73) at io.gatling.core.action.If.io$gatling$core$action$ExitableAction$$super$$bang(If.scala:24) at io.gatling.core.action.ExitableAction.$bang(Action.scala:111) at io.gatling.core.action.ExitableAction.$bang$(Action.scala:108) at io.gatling.core.action.If.$bang(If.scala:24) at io.gatling.http.engine.response.RootNextExecutor.executeNextOnCrash(NextExecutor.scala:46) at io.gatling.http.engine.response.DefaultResponseProcessor.onComplete(ResponseProcessor.scala:160) at io.gatling.http.engine.GatlingHttpListener.onThrowable(GatlingHttpListener.scala:237) at io.gatling.http.client.impl.DefaultHttpClient.lambda$installSslHandler$13(DefaultHttpClient.java:907) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:2005) at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1165) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1352) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:850) at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:811) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:840) 01:23:46.841 [ERROR] i.g.c.action.If - 'if-40' crashed with 'j.u.c.RejectedExecutionException: Task io.gatling.core.actor.AtomicRunnableActorRef@1bdf0b5e rejected from java.util.concurrent.ThreadPoolExecutor@49a5ee78[Shutting down, pool size = 7, active threads = 2, queued tasks = 0, completed tasks = 84855750]', forwarding to the next one [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 30:13 min [INFO] Finished at: 2025-03-28T01:23:46Z [INFO] ------------------------------------------------------------------------ 2025-03-28 01:23:46,950 INFO Gatling simulation run ended successfully 2025-03-28 01:23:46,950 DEBUG No archive name provided. No archive will be created ________________________________________________________________________________ [2025-03-28 01:24:01] mixedquery-10000 show_stderr : Display stderr order ________________________________________________________________________________ 2025-03-28 01:24:01,150 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/stderr.txt is small (20 lines), printing all lines: 2025-03-28 01:24:01,150 INFO ----- output ----- Exception in thread "pool-1-thread-2" java.util.concurrent.RejectedExecutionException: Task io.gatling.core.actor.AtomicRunnableActorRef@2b3b80ea rejected from java.util.concurrent.ThreadPoolExecutor@49a5ee78[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 84855750] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21) at io.gatling.core.actor.AtomicRunnableActorRef.async(ActorSystem.scala:95) at io.gatling.core.actor.AtomicRunnableActorRef.run(ActorSystem.scala:127) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Exception in thread "pool-1-thread-5" java.util.concurrent.RejectedExecutionException: Task io.gatling.core.actor.AtomicRunnableActorRef@577dae8a rejected from java.util.concurrent.ThreadPoolExecutor@49a5ee78[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 84855751] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21) at io.gatling.core.actor.AtomicRunnableActorRef.async(ActorSystem.scala:95) at io.gatling.core.actor.AtomicRunnableActorRef.run(ActorSystem.scala:127) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) ________________________________________________________________________________ [2025-03-28 01:24:01] mixedquery-10000 check_rc_and_logs : Check order's outputs ________________________________________________________________________________ CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) PASS: no error found in rc log PASS: no error found in stdout log [check_stderr_log] stderr file is not empty : 2025-03-28 01:24:01,460 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_simple_managed_users/07_Scenario_scenario/06_mixedquery-10000/stderr.txt is small (20 lines), printing all lines: 2025-03-28 01:24:01,460 INFO ----- output ----- Exception in thread "pool-1-thread-2" java.util.concurrent.RejectedExecutionException: Task io.gatling.core.actor.AtomicRunnableActorRef@2b3b80ea rejected from java.util.concurrent.ThreadPoolExecutor@49a5ee78[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 84855750] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21) at io.gatling.core.actor.AtomicRunnableActorRef.async(ActorSystem.scala:95) at io.gatling.core.actor.AtomicRunnableActorRef.run(ActorSystem.scala:127) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Exception in thread "pool-1-thread-5" java.util.concurrent.RejectedExecutionException: Task io.gatling.core.actor.AtomicRunnableActorRef@577dae8a rejected from java.util.concurrent.ThreadPoolExecutor@49a5ee78[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 84855751] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21) at io.gatling.core.actor.AtomicRunnableActorRef.async(ActorSystem.scala:95) at io.gatling.core.actor.AtomicRunnableActorRef.run(ActorSystem.scala:127) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 2025-03-28 01:24:01 - INFO: Traceback (most recent call last): File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/Task.py", line 633, in run self._run() File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/OverseerTask.py", line 153, in _run self.check_rc_and_logs() File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/OverseerTask.py", line 327, in check_rc_and_logs self.source.check_stderr_log( File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/components/overseer.py", line 405, in check_stderr_log raise FailException(message) shared.lib.utils.exception.FailException: [check_stderr_log] stderr file is not empty Setting result to FAIL Task has been successfully stopped