--Task-- name: preload-with-gatling enabled: True class_name: GatlingTask source_name: overseer-0 source_namespace: >default< target_name: idm target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates below loop: False interval: None dependencies: ['prepare-workload'] wait_for: ['Pre-check Num Users'] preceding_task: None options: {'check-before-preload': True, 'testCase': 'idm.ManagedUsersPreload', 'concurrency': 20, 'java-opts': '-Dcsvfile=/results/idm/preload.csv -Doutput=/results/idm/user_feeder.csv -Drealm=alpha'} group_name: None Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock 2025-05-15 23:19:43,156 INFO 2025-05-15 23:19:43,156 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-05-15 23:19:43,156 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-05-15 23:19:43,431 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-05-15 23:19:43,431 INFO [loop_until]: OK (rc = 0) 2025-05-15 23:19:43,431 DEBUG --- stdout --- 2025-05-15 23:19:43,431 DEBUG overseer-0-898c87646-8747d 2025-05-15 23:19:43,431 DEBUG --- stderr --- 2025-05-15 23:19:43,431 DEBUG 2025-05-15 23:19:43,431 INFO Lodestar assumed that full stack is deployed so will set all products host URLs in Gatling system properties 2025-05-15 23:19:43 - INFO: Timeout for this Task has changed to 2h (7200 seconds) 2025-05-15 23:19:43 - INFO: timeout was set to GatlingTask default, because it was unset 2025-05-15 23:19:43 - INFO: Interval for this Task has changed to 2m (120 seconds) 2025-05-15 23:19:43 - INFO: interval was set to Task default because it was unset, based on self.timeout value because task is allowed to stop by itself Pod log file : /results/orders/preload-with-gatling/stdout.txt Pod log : /results/orders/preload-with-gatling ________________________________________________________________________________ [2025-05-15 23:19:43] preload-with-gatling print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class="com.forgerock.pyrock.Loadtest" --run-description="idm.ManagedUsersPreload@perf-10m" --results-folder="/results/gatling/preload-with-gatling" --no-reports --simulation-jvm-args="-XX:+ExitOnOutOfMemoryError -Xmx4G" --simulation-system-properties="-Dam_host=openam-perf-10m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcheck_before_preload=True -Dconcurrency=20 -Dcsvfile=/results/idm/preload.csv -Didc=true -Didm_host=openam-perf-10m.forgeblocks.com -Dlogin_cookie=3398f6a1737d73a -Dnum_users=10000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Doutput=/results/idm/user_feeder.csv -Drealm=alpha -DtestCase=idm.ManagedUsersPreload -Duser_password=**** ________________________________________________________________________________ [2025-05-15 23:19:43] preload-with-gatling create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file: /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/order.json { "name": "preload-with-gatling", "command": " /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class=\"com.forgerock.pyrock.Loadtest\" --run-description=\"idm.ManagedUsersPreload@perf-10m\" --results-folder=\"/results/gatling/preload-with-gatling\" --no-reports --simulation-jvm-args=\"-XX:+ExitOnOutOfMemoryError -Xmx4G\" --simulation-system-properties=\"-Dam_host=openam-perf-10m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcheck_before_preload=True -Dconcurrency=20 -Dcsvfile=/results/idm/preload.csv -Didc=true -Didm_host=openam-perf-10m.forgeblocks.com -Dlogin_cookie=3398f6a1737d73a -Dnum_users=10000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Doutput=/results/idm/user_feeder.csv -Drealm=alpha -DtestCase=idm.ManagedUsersPreload -Duser_password=T35!tr0c-k123\"", "out-dir": "/results/orders/preload-with-gatling" } ________________________________________________________________________________ [2025-05-15 23:19:43] preload-with-gatling loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2025-05-15 23:19:43,440 INFO 2025-05-15 23:19:43,441 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2025-05-15 23:19:43,441 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2025-05-15 23:19:43,652 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-05-15 23:19:43,652 INFO [loop_until]: OK (rc = 0) 2025-05-15 23:19:43,652 DEBUG --- stdout --- 2025-05-15 23:19:43,652 DEBUG WAITING 2025-05-15 23:19:43,652 DEBUG --- stderr --- 2025-05-15 23:19:43,652 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 34 0 --:--:-- --:--:-- --:--:-- 34 ________________________________________________________________________________ [2025-05-15 23:19:43] preload-with-gatling start_order : Send the order to the overseer pod ________________________________________________________________________________ 2025-05-15 23:19:43,653 INFO 2025-05-15 23:19:43,653 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-05-15 23:19:43,653 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-05-15 23:19:43,938 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-05-15 23:19:43,938 INFO [loop_until]: OK (rc = 0) 2025-05-15 23:19:43,938 DEBUG --- stdout --- 2025-05-15 23:19:43,938 DEBUG overseer-0-898c87646-8747d 2025-05-15 23:19:43,938 DEBUG --- stderr --- 2025-05-15 23:19:43,938 DEBUG 2025-05-15 23:19:43,939 INFO 2025-05-15 23:19:43,939 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/order.json overseer-0-898c87646-8747d:/results/orders/order.json 2025-05-15 23:19:43,939 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-05-15 23:19:45,285 INFO [loop_until]: OK (rc = 0) 2025-05-15 23:19:45,285 DEBUG --- stdout --- 2025-05-15 23:19:45,285 DEBUG 2025-05-15 23:19:45,285 DEBUG --- stderr --- 2025-05-15 23:19:45,285 DEBUG Order has been successfully forwarded to overseer 2025-05-15 23:19:45 - INFO: Check if Gatling simulation started successfully 2025-05-15 23:19:45,285 INFO 2025-05-15 23:19:45,285 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d -- head --lines 5000 /results/orders/preload-with-gatling/stdout.txt | grep "Simulation com.forgerock.pyrock.Loadtest started..." 2025-05-15 23:19:45,285 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2025-05-15 23:19:46,010 INFO [loop_until]: Function succeeded after 0s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-05-15 23:19:51,729 INFO [loop_until]: Function succeeded after 6s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-05-15 23:19:57,444 INFO [loop_until]: Function succeeded after 12s (rc=0) - failed to find expected output: Simulation com.forgerock.pyrock.Loadtest started... - retry 2025-05-15 23:20:03,173 INFO [loop_until]: Function succeeded after 17s (rc=0) - expected pattern found 2025-05-15 23:20:03,173 INFO [loop_until]: OK (rc = 0) 2025-05-15 23:20:03,173 DEBUG --- stdout --- 2025-05-15 23:20:03,173 DEBUG 2025-05-16 00:19:47,000 DEBUG Creating results folder /results/gatling/preload-with-gatling 2025-05-16 00:19:47,006 INFO pom.xml and Gatling Scala files copied from /home/jenkins/lodestar/pyrock/shared/clients/gatling to temporary folder /tmp/tmp3rwtov2d 2025-05-16 00:19:47,009 INFO Running Gatling simulation with command: mvn gatling:test --batch-mode --quiet -Dgatling.simulationClass="com.forgerock.pyrock.Loadtest" -Dgatling.runDescription="..." -Dgatling.resultsFolder="/results/gatling/preload-with-gatling" -Dgatling.noReports=true -Dgatling.jvmArgs="-XX:+ExitOnOutOfMemoryError,-Xmx4G" -Dgatling.propagateSystemProperties=true -Dam_host=openam-perf-10m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcheck_before_preload=True -Dconcurrency=20 -Dcsvfile=/results/idm/preload.csv -Didc=true -Didm_host=openam-perf-10m.forgeblocks.com -Dlogin_cookie=3398f6a1737d73a -Dnum_users=10000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Doutput=/results/idm/user_feeder.csv -Drealm=alpha -DtestCase=idm.ManagedUsersPreload -Duser_password=**** Gatling 3.13.5 is available! (you're using 3.12.0) System property think_time_multiplier not set. No think time emulation will be done in the simulation. Simulation com.forgerock.pyrock.Loadtest started... 2025-05-15 23:20:03,173 DEBUG --- stderr --- 2025-05-15 23:20:03,173 DEBUG ________________________________________________________________________________ [2025-05-15 23:20:03] preload-with-gatling wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 2h. Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING Order is running since 2m, retry Order is running since 4m, retry Order is running since 6m, retry Order is running since 8m 01s, retry Order is running since 10m 01s, retry Order is running since 12m 01s, retry Order is running since 14m 02s, retry Order is running since 16m 02s, retry Order is running since 18m 02s, retry Order is running since 20m 03s, retry Order is running since 22m 03s, retry Order is running since 24m 03s, retry Order is running since 26m 04s, retry Order is running since 28m 04s, retry Order is running since 30m 04s, retry Order is running since 32m 05s, retry Order is running since 34m 05s, retry Order is running since 36m 05s, retry Order is running since 38m 06s, retry Order is running since 40m 06s, retry Order is running since 42m 06s, retry Order is running since 44m 06s, retry Order is running since 46m 07s, retry Order is running since 48m 07s, retry Order is running since 50m 07s, retry Order is running since 52m 08s, retry Order is running since 54m 08s, retry Order is running since 56m 08s, retry Order is running since 58m 09s, retry Order is running since 1h, retry Order is running since 1h 02m, retry Order is running since 1h 04m, retry Order is running since 1h 06m, retry Order is running since 1h 08m, retry Order is running since 1h 10m, retry Order is running since 1h 12m, retry Order is running since 1h 14m, retry Order is running since 1h 16m, retry Order is running since 1h 18m, retry Order is running since 1h 20m, retry Order is running since 1h 22m, retry Order is running since 1h 24m, retry Order is running since 1h 26m, retry Order is running since 1h 28m, retry Order is running since 1h 30m, retry Order is running since 1h 32m, retry Order is running since 1h 34m, retry Order is running since 1h 36m, retry Order is running since 1h 38m, retry Order is running since 1h 40m, retry Order is running since 1h 42m, retry Order is running since 1h 44m, retry Order is running since 1h 46m, retry Order is running since 1h 48m, retry Order is running since 1h 50m, retry Order is running since 1h 52m, retry Order is running since 1h 54m, retry Order is running since 1h 56m, retry Order is running since 1h 58m, retry Order is running since 2h, retry ERROR : Order is still running after timeout 2h Setting result to TIMEOUT ________________________________________________________________________________ [2025-05-16 01:20:21] preload-with-gatling stop_order : Stop the order on overseer pod ________________________________________________________________________________ 2025-05-16 01:20:21,775 INFO 2025-05-16 01:20:21,775 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-05-16 01:20:21,775 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-05-16 01:20:22,076 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-05-16 01:20:22,076 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:22,076 DEBUG --- stdout --- 2025-05-16 01:20:22,076 DEBUG overseer-0-898c87646-8747d 2025-05-16 01:20:22,076 DEBUG --- stderr --- 2025-05-16 01:20:22,076 DEBUG 2025-05-16 01:20:22,076 INFO 2025-05-16 01:20:22,076 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d -- rm -f /results/orders/order.json 2025-05-16 01:20:22,076 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-05-16 01:20:22,858 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:22,858 DEBUG --- stdout --- 2025-05-16 01:20:22,858 DEBUG 2025-05-16 01:20:22,858 DEBUG --- stderr --- 2025-05-16 01:20:22,858 DEBUG Order ran during 2h ________________________________________________________________________________ [2025-05-16 01:20:22] preload-with-gatling download_logs : Download individual order files ________________________________________________________________________________ 2025-05-16 01:20:22 - PASS: file already downloaded /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/order.json 2025-05-16 01:20:22,859 INFO 2025-05-16 01:20:22,859 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-05-16 01:20:22,859 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-05-16 01:20:23,132 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-05-16 01:20:23,132 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:23,132 DEBUG --- stdout --- 2025-05-16 01:20:23,132 DEBUG overseer-0-898c87646-8747d 2025-05-16 01:20:23,132 DEBUG --- stderr --- 2025-05-16 01:20:23,132 DEBUG Checking if file exists on pod 'overseer-0-898c87646-8747d', wait max 3 minutes if not. 2025-05-16 01:20:23,133 INFO 2025-05-16 01:20:23,133 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d --stdin -- bash -c 'if [ -f "/results/orders/preload-with-gatling/rc.txt" ]; then echo "/results/orders/preload-with-gatling/rc.txt file found"; exit 0; else echo "/results/orders/preload-with-gatling/rc.txt file not found"; exit 1; fi' | grep "/results/orders/preload-with-gatling/rc.txt file found" 2025-05-16 01:20:23,133 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-05-16 01:20:23,929 ERROR [loop_until]: Function failed after 0s (rc=1 - expected to be in [0]) - retry; CMD output was: /results/orders/preload-with-gatling/rc.txt file not found CMD error was: command terminated with exit code 1 2025-05-16 01:20:29,711 INFO [loop_until]: Function succeeded after 6s (rc=0) - expected pattern found 2025-05-16 01:20:29,711 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:29,711 DEBUG --- stdout --- 2025-05-16 01:20:29,711 DEBUG /results/orders/preload-with-gatling/rc.txt file found 2025-05-16 01:20:29,711 DEBUG --- stderr --- 2025-05-16 01:20:29,711 DEBUG 2025-05-16 01:20:29,711 INFO 2025-05-16 01:20:29,711 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d --stdin -- du -k /results/orders/preload-with-gatling/rc.txt | grep "^\d+ .+$" 2025-05-16 01:20:29,711 INFO [loop_until]: (max_time=240, interval=10, expected_rc=[0] 2025-05-16 01:20:30,497 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-05-16 01:20:30,497 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:30,497 DEBUG --- stdout --- 2025-05-16 01:20:30,497 DEBUG 4 /results/orders/preload-with-gatling/rc.txt 2025-05-16 01:20:30,497 DEBUG --- stderr --- 2025-05-16 01:20:30,497 DEBUG - Download file (size 4kB) source : /results/orders/preload-with-gatling/rc.txt @ overseer-0-898c87646-8747d destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/rc.txt 2025-05-16 01:20:30,498 INFO 2025-05-16 01:20:30,498 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-898c87646-8747d:/results/orders/preload-with-gatling/rc.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/rc.txt 2025-05-16 01:20:30,498 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-05-16 01:20:31,225 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:31,225 DEBUG --- stdout --- 2025-05-16 01:20:31,225 DEBUG tar: Removing leading `/' from member names 2025-05-16 01:20:31,225 DEBUG --- stderr --- 2025-05-16 01:20:31,225 DEBUG 2025-05-16 01:20:31,226 INFO 2025-05-16 01:20:31,226 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-05-16 01:20:31,226 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-05-16 01:20:31,498 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-05-16 01:20:31,498 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:31,498 DEBUG --- stdout --- 2025-05-16 01:20:31,498 DEBUG overseer-0-898c87646-8747d 2025-05-16 01:20:31,498 DEBUG --- stderr --- 2025-05-16 01:20:31,498 DEBUG Checking if file exists on pod 'overseer-0-898c87646-8747d', wait max 3 minutes if not. 2025-05-16 01:20:31,499 INFO 2025-05-16 01:20:31,500 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d --stdin -- bash -c 'if [ -f "/results/orders/preload-with-gatling/stderr.txt" ]; then echo "/results/orders/preload-with-gatling/stderr.txt file found"; exit 0; else echo "/results/orders/preload-with-gatling/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/preload-with-gatling/stderr.txt file found" 2025-05-16 01:20:31,500 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-05-16 01:20:32,285 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-05-16 01:20:32,285 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:32,285 DEBUG --- stdout --- 2025-05-16 01:20:32,285 DEBUG /results/orders/preload-with-gatling/stderr.txt file found 2025-05-16 01:20:32,285 DEBUG --- stderr --- 2025-05-16 01:20:32,285 DEBUG 2025-05-16 01:20:32,285 INFO 2025-05-16 01:20:32,285 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d --stdin -- du -k /results/orders/preload-with-gatling/stderr.txt | grep "^\d+ .+$" 2025-05-16 01:20:32,285 INFO [loop_until]: (max_time=240, interval=10, expected_rc=[0] 2025-05-16 01:20:33,073 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-05-16 01:20:33,073 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:33,073 DEBUG --- stdout --- 2025-05-16 01:20:33,073 DEBUG 0 /results/orders/preload-with-gatling/stderr.txt 2025-05-16 01:20:33,073 DEBUG --- stderr --- 2025-05-16 01:20:33,073 DEBUG - Download file (size 0B) source : /results/orders/preload-with-gatling/stderr.txt @ overseer-0-898c87646-8747d destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/stderr.txt 2025-05-16 01:20:33,074 INFO 2025-05-16 01:20:33,074 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-898c87646-8747d:/results/orders/preload-with-gatling/stderr.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/stderr.txt 2025-05-16 01:20:33,074 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-05-16 01:20:33,843 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:33,843 DEBUG --- stdout --- 2025-05-16 01:20:33,843 DEBUG tar: Removing leading `/' from member names 2025-05-16 01:20:33,843 DEBUG --- stderr --- 2025-05-16 01:20:33,843 DEBUG 2025-05-16 01:20:33,844 INFO 2025-05-16 01:20:33,844 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-05-16 01:20:33,844 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-05-16 01:20:34,118 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-05-16 01:20:34,118 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:34,118 DEBUG --- stdout --- 2025-05-16 01:20:34,118 DEBUG overseer-0-898c87646-8747d 2025-05-16 01:20:34,118 DEBUG --- stderr --- 2025-05-16 01:20:34,118 DEBUG Checking if file exists on pod 'overseer-0-898c87646-8747d', wait max 3 minutes if not. 2025-05-16 01:20:34,119 INFO 2025-05-16 01:20:34,120 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d --stdin -- bash -c 'if [ -f "/results/orders/preload-with-gatling/stdout.txt" ]; then echo "/results/orders/preload-with-gatling/stdout.txt file found"; exit 0; else echo "/results/orders/preload-with-gatling/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/preload-with-gatling/stdout.txt file found" 2025-05-16 01:20:34,120 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-05-16 01:20:34,900 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-05-16 01:20:34,901 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:34,901 DEBUG --- stdout --- 2025-05-16 01:20:34,901 DEBUG /results/orders/preload-with-gatling/stdout.txt file found 2025-05-16 01:20:34,901 DEBUG --- stderr --- 2025-05-16 01:20:34,901 DEBUG 2025-05-16 01:20:34,901 INFO 2025-05-16 01:20:34,901 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d --stdin -- du -k /results/orders/preload-with-gatling/stdout.txt | grep "^\d+ .+$" 2025-05-16 01:20:34,901 INFO [loop_until]: (max_time=240, interval=10, expected_rc=[0] 2025-05-16 01:20:35,687 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-05-16 01:20:35,687 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:35,687 DEBUG --- stdout --- 2025-05-16 01:20:35,687 DEBUG 2092 /results/orders/preload-with-gatling/stdout.txt 2025-05-16 01:20:35,687 DEBUG --- stderr --- 2025-05-16 01:20:35,687 DEBUG - Download file (size 2.092MB) source : /results/orders/preload-with-gatling/stdout.txt @ overseer-0-898c87646-8747d destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/stdout.txt 2025-05-16 01:20:35,688 INFO 2025-05-16 01:20:35,688 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-898c87646-8747d:/results/orders/preload-with-gatling/stdout.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/stdout.txt 2025-05-16 01:20:35,688 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-05-16 01:20:36,856 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:36,856 DEBUG --- stdout --- 2025-05-16 01:20:36,856 DEBUG tar: Removing leading `/' from member names 2025-05-16 01:20:36,856 DEBUG --- stderr --- 2025-05-16 01:20:36,856 DEBUG ________________________________________________________________________________ [2025-05-16 01:20:36] preload-with-gatling show_cmd : Display order command ________________________________________________________________________________ 2025-05-16 01:20:36,857 INFO 2025-05-16 01:20:36,857 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2025-05-16 01:20:36,857 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2025-05-16 01:20:37,132 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2025-05-16 01:20:37,132 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:37,132 DEBUG --- stdout --- 2025-05-16 01:20:37,132 DEBUG overseer-0-898c87646-8747d 2025-05-16 01:20:37,132 DEBUG --- stderr --- 2025-05-16 01:20:37,132 DEBUG Checking if file exists on pod 'overseer-0-898c87646-8747d', wait max 3 minutes if not. 2025-05-16 01:20:37,133 INFO 2025-05-16 01:20:37,133 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d --stdin -- bash -c 'if [ -f "/results/orders/preload-with-gatling/cmd.txt" ]; then echo "/results/orders/preload-with-gatling/cmd.txt file found"; exit 0; else echo "/results/orders/preload-with-gatling/cmd.txt file not found"; exit 1; fi' | grep "/results/orders/preload-with-gatling/cmd.txt file found" 2025-05-16 01:20:37,133 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-05-16 01:20:37,944 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-05-16 01:20:37,945 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:37,945 DEBUG --- stdout --- 2025-05-16 01:20:37,945 DEBUG /results/orders/preload-with-gatling/cmd.txt file found 2025-05-16 01:20:37,945 DEBUG --- stderr --- 2025-05-16 01:20:37,945 DEBUG 2025-05-16 01:20:37,945 INFO 2025-05-16 01:20:37,945 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-898c87646-8747d --stdin -- du -k /results/orders/preload-with-gatling/cmd.txt | grep "^\d+ .+$" 2025-05-16 01:20:37,945 INFO [loop_until]: (max_time=240, interval=10, expected_rc=[0] 2025-05-16 01:20:38,729 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2025-05-16 01:20:38,729 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:38,729 DEBUG --- stdout --- 2025-05-16 01:20:38,729 DEBUG 4 /results/orders/preload-with-gatling/cmd.txt 2025-05-16 01:20:38,730 DEBUG --- stderr --- 2025-05-16 01:20:38,730 DEBUG - Download file (size 4kB) source : /results/orders/preload-with-gatling/cmd.txt @ overseer-0-898c87646-8747d destination : /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/cmd.txt 2025-05-16 01:20:38,730 INFO 2025-05-16 01:20:38,731 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-898c87646-8747d:/results/orders/preload-with-gatling/cmd.txt /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/cmd.txt 2025-05-16 01:20:38,731 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2025-05-16 01:20:39,430 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:39,430 DEBUG --- stdout --- 2025-05-16 01:20:39,431 DEBUG tar: Removing leading `/' from member names 2025-05-16 01:20:39,431 DEBUG --- stderr --- 2025-05-16 01:20:39,431 DEBUG 2025-05-16 01:20:39,431 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/cmd.txt is small (1 lines), printing all lines: 2025-05-16 01:20:39,431 INFO ----- output ----- /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class="com.forgerock.pyrock.Loadtest" --run-description="idm.ManagedUsersPreload@perf-10m" --results-folder="/results/gatling/preload-with-gatling" --no-reports --simulation-jvm-args="-XX:+ExitOnOutOfMemoryError -Xmx4G" --simulation-system-properties="-Dam_host=openam-perf-10m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=M@ythe4th -DamadminUsername=xiaosong.lou@pingidentity.com -Dcheck_before_preload=True -Dconcurrency=20 -Dcsvfile=/results/idm/preload.csv -Didc=true -Didm_host=openam-perf-10m.forgeblocks.com -Dlogin_cookie=3398f6a1737d73a -Dnum_users=10000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Doutput=/results/idm/user_feeder.csv -Drealm=alpha -DtestCase=idm.ManagedUsersPreload -Duser_password=T35!tr0c-k123" ________________________________________________________________________________ [2025-05-16 01:20:39] preload-with-gatling show_rc : Display rc order ________________________________________________________________________________ 2025-05-16 01:20:39,431 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/rc.txt is small (1 lines), printing all lines: 2025-05-16 01:20:39,431 INFO ----- output ----- CMD SUCCESS: command returns code -9 (expected [0, -9, 143, 137, -15]) ________________________________________________________________________________ [2025-05-16 01:20:39] preload-with-gatling show_stdout : Display stdout order ________________________________________________________________________________ 2025-05-16 01:20:39,434 DEBUG [print_head_tail]: Print head (100 lines) and tail (100 lines) of input file (/mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/stdout.txt is 31550 lines): 2025-05-16 01:20:39,438 INFO ----- output ----- 2025-05-16 00:19:47,000 DEBUG Creating results folder /results/gatling/preload-with-gatling 2025-05-16 00:19:47,006 INFO pom.xml and Gatling Scala files copied from /home/jenkins/lodestar/pyrock/shared/clients/gatling to temporary folder /tmp/tmp3rwtov2d 2025-05-16 00:19:47,009 INFO Running Gatling simulation with command: mvn gatling:test --batch-mode --quiet -Dgatling.simulationClass="com.forgerock.pyrock.Loadtest" -Dgatling.runDescription="..." -Dgatling.resultsFolder="/results/gatling/preload-with-gatling" -Dgatling.noReports=true -Dgatling.jvmArgs="-XX:+ExitOnOutOfMemoryError,-Xmx4G" -Dgatling.propagateSystemProperties=true -Dam_host=openam-perf-10m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcheck_before_preload=True -Dconcurrency=20 -Dcsvfile=/results/idm/preload.csv -Didc=true -Didm_host=openam-perf-10m.forgeblocks.com -Dlogin_cookie=3398f6a1737d73a -Dnum_users=10000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Doutput=/results/idm/user_feeder.csv -Drealm=alpha -DtestCase=idm.ManagedUsersPreload -Duser_password=**** Gatling 3.13.5 is available! (you're using 3.12.0) System property think_time_multiplier not set. No think time emulation will be done in the simulation. Simulation com.forgerock.pyrock.Loadtest started... ================================================================================ 2025-05-15 23:20:04 GMT 5s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=786 KO=0 ) > restLoginInitiate (OK=1 KO=0 ) > restLoginUsernamePasswordCallback (OK=1 KO=0 ) > Skip 2FA (OK=1 KO=0 ) > User Authorize PKCE stage (OK=1 KO=0 ) > User AccessToken PKCE stage (OK=1 KO=0 ) > CreateManagedUser (OK=781 KO=0 ) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ---- idm.ManagedUsersPreload --------------------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / active: 20 / done: 0 ================================================================================ ================================================================================ 2025-05-15 23:20:09 GMT 9s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=1961 KO=0 ) > restLoginInitiate (OK=1 KO=0 ) > restLoginUsernamePasswordCallback (OK=1 KO=0 ) > Skip 2FA (OK=1 KO=0 ) > User Authorize PKCE stage (OK=1 KO=0 ) > User AccessToken PKCE stage (OK=1 KO=0 ) > CreateManagedUser (OK=1956 KO=0 ) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ---- idm.ManagedUsersPreload --------------------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / active: 20 / done: 0 ================================================================================ ================================================================================ 2025-05-15 23:20:14 GMT 14s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=3124 KO=0 ) > restLoginInitiate (OK=1 KO=0 ) > restLoginUsernamePasswordCallback (OK=1 KO=0 ) > Skip 2FA (OK=1 KO=0 ) > User Authorize PKCE stage (OK=1 KO=0 ) > User AccessToken PKCE stage (OK=1 KO=0 ) > CreateManagedUser (OK=3119 KO=0 ) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ---- idm.ManagedUsersPreload --------------------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / active: 20 / done: 0 ================================================================================ ================================================================================ 2025-05-15 23:20:19 GMT 19s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=4282 KO=0 ) > restLoginInitiate (OK=1 KO=0 ) > restLoginUsernamePasswordCallback (OK=1 KO=0 ) > Skip 2FA (OK=1 KO=0 ) > User Authorize PKCE stage (OK=1 KO=0 ) > User AccessToken PKCE stage (OK=1 KO=0 ) > CreateManagedUser (OK=4277 KO=0 ) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ---- idm.ManagedUsersPreload --------------------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / active: 20 / done: 0 ================================================================================ ================================================================================ 2025-05-15 23:20:24 GMT 24s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=5575 KO=0 ) > restLoginInitiate (OK=1 KO=0 ) > restLoginUsernamePasswordCallback (OK=1 KO=0 ) > Skip 2FA (OK=1 KO=0 ) > User Authorize PKCE stage (OK=1 KO=0 ) > User AccessToken PKCE stage (OK=1 KO=0 ) > CreateManagedUser (OK=5570 KO=0 ) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% [...] > CreateManagedUser (OK=1857933 KO=1 ) ---- Errors -------------------------------------------------------------------- > status.find.in(201,403), but actually found 502 1 (100.0%) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ---- idm.ManagedUsersPreload --------------------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / active: 20 / done: 0 ================================================================================ ================================================================================ 2025-05-16 01:20:04 GMT 7204s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=1859841 KO=1 ) > restLoginInitiate (OK=120 KO=0 ) > restLoginUsernamePasswordCallback (OK=120 KO=0 ) > Skip 2FA (OK=120 KO=0 ) > User Authorize PKCE stage (OK=120 KO=0 ) > User AccessToken PKCE stage (OK=120 KO=0 ) > CreateManagedUser (OK=1859241 KO=1 ) ---- Errors -------------------------------------------------------------------- > status.find.in(201,403), but actually found 502 1 (100.0%) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ---- idm.ManagedUsersPreload --------------------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / active: 20 / done: 0 ================================================================================ ================================================================================ 2025-05-16 01:20:09 GMT 7209s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=1861149 KO=1 ) > restLoginInitiate (OK=120 KO=0 ) > restLoginUsernamePasswordCallback (OK=120 KO=0 ) > Skip 2FA (OK=120 KO=0 ) > User Authorize PKCE stage (OK=120 KO=0 ) > User AccessToken PKCE stage (OK=120 KO=0 ) > CreateManagedUser (OK=1860549 KO=1 ) ---- Errors -------------------------------------------------------------------- > status.find.in(201,403), but actually found 502 1 (100.0%) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ---- idm.ManagedUsersPreload --------------------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / active: 20 / done: 0 ================================================================================ ================================================================================ 2025-05-16 01:20:14 GMT 7214s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=1862479 KO=1 ) > restLoginInitiate (OK=120 KO=0 ) > restLoginUsernamePasswordCallback (OK=120 KO=0 ) > Skip 2FA (OK=120 KO=0 ) > User Authorize PKCE stage (OK=120 KO=0 ) > User AccessToken PKCE stage (OK=120 KO=0 ) > CreateManagedUser (OK=1861879 KO=1 ) ---- Errors -------------------------------------------------------------------- > status.find.in(201,403), but actually found 502 1 (100.0%) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ---- idm.ManagedUsersPreload --------------------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / active: 20 / done: 0 ================================================================================ ================================================================================ 2025-05-16 01:20:19 GMT 7219s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=1863817 KO=1 ) > restLoginInitiate (OK=120 KO=0 ) > restLoginUsernamePasswordCallback (OK=120 KO=0 ) > Skip 2FA (OK=120 KO=0 ) > User Authorize PKCE stage (OK=120 KO=0 ) > User AccessToken PKCE stage (OK=120 KO=0 ) > CreateManagedUser (OK=1863217 KO=1 ) ---- Errors -------------------------------------------------------------------- > status.find.in(201,403), but actually found 502 1 (100.0%) ---- getToken ------------------------------------------------------------------ [--------------------------------------------------------------------------] 0% waiting: 0 / active: 1 / done: 0 ---- idm.ManagedUsersPreload --------------------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / active: 20 / done: 0 ================================================================================ ________________________________________________________________________________ [2025-05-16 01:20:39] preload-with-gatling show_stderr : Display stderr order ________________________________________________________________________________ 2025-05-16 01:20:39,439 DEBUG [print_head_tail]: Input file /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_login_tree/05_Data_Init/04_preload-with-gatling/stderr.txt is small (0 lines), printing all lines: 2025-05-16 01:20:39,439 INFO ----- output ----- ________________________________________________________________________________ [2025-05-16 01:20:39] preload-with-gatling check_rc_and_logs : Check order's outputs ________________________________________________________________________________ CMD SUCCESS: command returns code -9 (expected [0, -9, 143, 137, -15]) PASS: no error found in rc log PASS: no error found in stdout log PASS: no error found in stderr log ________________________________________________________________________________ [2025-05-16 01:20:39] preload-with-gatling print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Logs for pod: {pod} 2025-05-16 01:20:39,711 INFO 2025-05-16 01:20:39,711 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm logs overseer-0-898c87646-8747d 2025-05-16 01:20:39,711 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2025-05-16 01:20:40,131 INFO [loop_until]: OK (rc = 0) 2025-05-16 01:20:40,131 DEBUG --- stdout --- 2025-05-16 01:20:40,131 DEBUG [2025-05-16 00:15:31] httpd web_server : Server is running at http://localhost:8080 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:15:31] overseer: Waiting for order /results/orders/order.json ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 10.106.40.13 - - [16/May/2025 00:15:35] "GET / HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:15:37] "GET / HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:17:28] "GET /orders/status.txt HTTP/1.1" 200 - [2025-05-16 00:17:30] overseer: Start new order cleanup (id 1) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:17:30] order cleanup: Start command ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:17:30] order cleanup: Running command : /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class="com.forgerock.pyrock.Loadtest" --run-description="idm.ManagedUsersDelete@perf-10m" --results-folder="/results/gatling/cleanup" --no-reports --simulation-jvm-args="-XX:+ExitOnOutOfMemoryError -Xmx4G" --simulation-system-properties=" -Dam_host=openam-perf-10m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcheck_before_preload=True -Dconcurrency=30 -Dcsvfile=/results/idm/user_feeder.csv -Didc=true -Didm_host=openam-perf-10m.forgeblocks.com -Dlogin_cookie=3398f6a1737d73a -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Drealm=alpha -DtestCase=idm.ManagedUsersDelete -Duser_password=**** [2025-05-16 00:17:30] order cleanup: Http access : http://localhost:8080/orders/cleanup [2025-05-16 00:17:30] order cleanup: Full command stored in : /results/orders/cleanup/cmd.txt [2025-05-16 00:17:30] order cleanup: RC file : /results/orders/cleanup/rc.txt [2025-05-16 00:17:30] order cleanup: Stdout file : /results/orders/cleanup/stdout.txt [2025-05-16 00:17:30] order cleanup: Stderr file : /results/orders/cleanup/stderr.txt [2025-05-16 00:17:30] order cleanup: Process 70 has been successfully launched [2025-05-16 00:17:47] order cleanup: CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) [2025-05-16 00:17:47] overseer: order stopped by itself cleanup : /results/orders/cleanup ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:17:47] overseer: Waiting for order /results/orders/order.json ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:17:47] overseer: Http access : http://localhost:8080 10.106.40.13 - - [16/May/2025 00:17:53] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:05] "GET /orders/cleanup/rc.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:05] "GET /orders/status.txt HTTP/1.1" 200 - [2025-05-16 00:18:07] overseer: Start new order prepare-workload (id 2) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:18:07] order prepare-workload: Start command ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:18:07] order prepare-workload: Running command : /lodestar/pyrock/shared/scripts/idm/prepare_workload.py --preload_records 10000000 --preload_output /results/idm/preload.csv --benchmark_records 1500000 --benchmark_output /results/idm/benchmark_users.csv [2025-05-16 00:18:07] order prepare-workload: Http access : http://localhost:8080/orders/prepare-workload [2025-05-16 00:18:07] order prepare-workload: Full command stored in : /results/orders/prepare-workload/cmd.txt [2025-05-16 00:18:07] order prepare-workload: RC file : /results/orders/prepare-workload/rc.txt [2025-05-16 00:18:07] order prepare-workload: Stdout file : /results/orders/prepare-workload/stdout.txt [2025-05-16 00:18:07] order prepare-workload: Stderr file : /results/orders/prepare-workload/stderr.txt [2025-05-16 00:18:07] order prepare-workload: Process 504 has been successfully launched 10.106.40.13 - - [16/May/2025 00:18:12] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:17] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:23] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:28] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:33] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:38] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:44] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:49] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:54] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:18:59] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:19:05] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:19:10] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:19:15] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:19:20] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:19:26] "GET /orders/status.txt HTTP/1.1" 200 - [2025-05-16 00:19:29] order prepare-workload: CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) [2025-05-16 00:19:29] overseer: order stopped by itself prepare-workload : /results/orders/prepare-workload ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:19:30] overseer: Waiting for order /results/orders/order.json ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:19:30] overseer: Http access : http://localhost:8080 10.106.40.13 - - [16/May/2025 00:19:31] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:19:43] "GET /orders/prepare-workload/rc.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:19:43] "GET /orders/status.txt HTTP/1.1" 200 - [2025-05-16 00:19:45] overseer: Start new order preload-with-gatling (id 3) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:19:45] order preload-with-gatling: Start command ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 00:19:45] order preload-with-gatling: Running command : /lodestar/lodestar.py perf run-gatling --mvn-quiet --simulation-class="com.forgerock.pyrock.Loadtest" --run-description="idm.ManagedUsersPreload@perf-10m" --results-folder="/results/gatling/preload-with-gatling" --no-reports --simulation-jvm-args="-XX:+ExitOnOutOfMemoryError -Xmx4G" --simulation-system-properties="-Dam_host=openam-perf-10m.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=**** -DamadminUsername=xiaosong.lou@pingidentity.com -Dcheck_before_preload=True -Dconcurrency=20 -Dcsvfile=/results/idm/preload.csv -Didc=true -Didm_host=openam-perf-10m.forgeblocks.com -Dlogin_cookie=3398f6a1737d73a -Dnum_users=10000000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Doutput=/results/idm/user_feeder.csv -Drealm=alpha -DtestCase=idm.ManagedUsersPreload -Duser_password=**** [2025-05-16 00:19:45] order preload-with-gatling: Http access : http://localhost:8080/orders/preload-with-gatling [2025-05-16 00:19:45] order preload-with-gatling: Full command stored in : /results/orders/preload-with-gatling/cmd.txt [2025-05-16 00:19:45] order preload-with-gatling: RC file : /results/orders/preload-with-gatling/rc.txt [2025-05-16 00:19:45] order preload-with-gatling: Stdout file : /results/orders/preload-with-gatling/stdout.txt [2025-05-16 00:19:45] order preload-with-gatling: Stderr file : /results/orders/preload-with-gatling/stderr.txt [2025-05-16 00:19:45] order preload-with-gatling: Process 603 has been successfully launched 10.106.40.13 - - [16/May/2025 00:22:03] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:24:03] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:26:04] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:28:04] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:30:04] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:32:05] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:34:05] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:36:05] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:38:06] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:40:06] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:42:06] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:44:06] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:46:07] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:48:07] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:50:07] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:52:08] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:54:08] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:56:08] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 00:58:09] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:00:09] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:02:09] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:04:10] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:06:10] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:08:10] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:10:10] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:12:11] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:14:11] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:16:11] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:18:12] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:20:12] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:22:12] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:24:13] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:26:13] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:28:13] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:30:14] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:32:14] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:34:14] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:36:15] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:38:15] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:40:15] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:42:15] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:44:16] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:46:16] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:48:16] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:50:17] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:52:17] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:54:17] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:56:18] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 01:58:18] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:00:18] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:02:18] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:04:19] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:06:19] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:08:19] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:10:20] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:12:20] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:14:20] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:16:21] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:18:21] "GET /orders/status.txt HTTP/1.1" 200 - 10.106.40.13 - - [16/May/2025 02:20:21] "GET /orders/status.txt HTTP/1.1" 200 - [2025-05-16 02:20:23] overseer: Stop order preload-with-gatling : /results/orders/preload-with-gatling ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 02:20:23] order preload-with-gatling: Stop command ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 02:20:23] order preload-with-gatling: Killing process pid=603 with signal 15 2025-05-16 02:20:23,929 INFO [kill_process]: Killing process pid=603 with signal 15 2025-05-16 02:20:23,931 INFO [kill_process]: Wait for the process to stop... 2025-05-16 02:20:24,931 INFO . 2025-05-16 02:20:24,931 INFO [kill_process]: Process 603 has been successfully stopped with RC=-9 [2025-05-16 02:20:24] order preload-with-gatling: CMD SUCCESS: command returns code -9 (expected [0, -9, 143, 137, -15]) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 02:20:25] overseer: Waiting for order /results/orders/order.json ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2025-05-16 02:20:25] overseer: Http access : http://localhost:8080 10.106.40.13 - - [16/May/2025 02:20:39] "GET /orders/preload-with-gatling/rc.txt HTTP/1.1" 200 - 2025-05-16 01:20:40,131 DEBUG --- stderr --- 2025-05-16 01:20:40,131 DEBUG ________________________________________________________________________________ [2025-05-16 01:20:40] preload-with-gatling post : Post ________________________________________________________________________________ Task has been successfully stopped