--Task-- name: authn-users enabled: True class_name: GatlingTask source_name: overseer-0 source_namespace: >default< target_name: am target_namespace: >default< start: 0 stop: 100 timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: [] wait_for: [] options: {'simulation': 'com.forgerock.gatling.Loadtest', 'testCase': 'idc.login', 'testMode': 'open', 'throughput': 1800, 'rampup-duration': 30, 'java-opts': '-Dnum_users=200000 -Drealm=alpha -Duser_password=T35tr0ck123*'} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [26/Sep/2022 22:10:23] authn-users pre : N/A ________________________________________________________________________________ [26/Sep/2022 22:10:23] - INFO: setting default timeout for task - 2x global_duration or 300 seconds (whatever is higher): [26/Sep/2022 22:10:23] - INFO: Timeout for this Task has changed to 2h (7200 seconds) Pod log file : /results/orders/authn-users/stdout.txt Pod log : /results/orders/authn-users ________________________________________________________________________________ [26/Sep/2022 22:10:23] authn-users print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ /lodestar/clients/gatling-for-pyrock.sh -s com.forgerock.gatling.Loadtest -sf /lodestar/pyrock/shared/clients/gatling-open/ -rd com.forgerock.gatling.Loadtest@IDC -rf /results/gatling/authn-users --no-reports ________________________________________________________________________________ [26/Sep/2022 22:10:23] authn-users create_order : Create order for overseer with the command to run ________________________________________________________________________________ Userpool size is not defined for the task in conf.yaml and cannot find how many users (entries) are in id-repo, please check the conf.yaml file. Default for selected Gatling simulation will be used. [26/Sep/2022 22:10:23] - INFO: Get overseer pod resources requests memory [loop_until]: kubectl --namespace=xlou get pods overseer-0-c7fc5cc47-x4c97 -o jsonpath={.spec.containers[].resources.requests.memory} [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- 4Gi --- stderr --- Order file : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/order.json { "name": "authn-users", "command": "/lodestar/clients/gatling-for-pyrock.sh -s com.forgerock.gatling.Loadtest -sf /lodestar/pyrock/shared/clients/gatling-open/ -rd com.forgerock.gatling.Loadtest@IDC -rf /results/gatling/authn-users --no-reports", "out-dir": "/results/orders/authn-users", "environment": { "JAVA_OPTS": "-Dam_host=openam-perf-100m-testing.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=M@ythe4th -DamadminUsername=xiaosong.lou@forgerock.com -Didc=true -Dnum_users=200000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Drampup_duration=30 -Drealm=alpha -DtestCase=idc.login -DtestMode=open -Dthroughput=1800 -Duser_password=T35tr0ck123* -Xmx4G -Dduration=7992" } } ________________________________________________________________________________ [26/Sep/2022 22:10:23] authn-users loop_until_ready : Loop until overseer pod is ready to accept order ________________________________________________________________________________ [loop_until]: curl -k https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" [loop_until]: (max_time=120, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- WAITING --- stderr --- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 30 0 --:--:-- --:--:-- --:--:-- 30 ________________________________________________________________________________ [26/Sep/2022 22:10:24] authn-users start_order : Send the order to the overseer pod ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou cp /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/order.json overseer-0-c7fc5cc47-x4c97:/results/orders/order.json [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- Order has been successfully forwarded to overseer [26/Sep/2022 22:10:25] - INFO: Check if Gatling simulation started successfully [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-c7fc5cc47-x4c97 -- head /results/orders/authn-users/stdout.txt | grep "started..." [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - failed to find expected output: started... - retry [loop_until]: Function succeeded after 6s (rc=0) - failed to find expected output: started... - retry [loop_until]: Function succeeded after 12s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- GATLING_HOME is set to /home/jenkins/lodestar/clients/gatling Simulation com.forgerock.gatling.Loadtest started... --- stderr --- Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Setting result to FAIL ________________________________________________________________________________ [26/Sep/2022 22:13:48] authn-users show_stdout : Display stdout order ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- Checking pod: overseer-0-c7fc5cc47-x4c97 for file exists, if not wait. [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-c7fc5cc47-x4c97 -i -- bash -c 'if [ -f "/results/orders/authn-users/stdout.txt" ]; then echo "/results/orders/authn-users/stdout.txt file found"; exit 0; else echo "/results/orders/authn-users/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/authn-users/stdout.txt file found" [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- /results/orders/authn-users/stdout.txt file found --- stderr --- [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [run_command]: kubectl --namespace=xlou exec overseer-0-c7fc5cc47-x4c97 -i -- du -k /results/orders/authn-users/stdout.txt [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- 746280 /results/orders/authn-users/stdout.txt --- stderr --- - Download file (size 746280k) [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- source : /results/orders/authn-users/stdout.txt @ overseer-0-c7fc5cc47-x4c97 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/stdout.txt [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou cp overseer-0-c7fc5cc47-x4c97:/results/orders/authn-users/stdout.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/stdout.txt [loop_until]: (max_time=-1, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- tar: Removing leading `/' from member names --- stderr --- [print_head_tail]: Print head (100 lines) and tail (100 lines) of input file (/mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/stdout.txt is 9512366 lines): ----- output ----- GATLING_HOME is set to /home/jenkins/lodestar/clients/gatling Simulation com.forgerock.gatling.Loadtest started... ================================================================================ 2022-09-26 23:10:38 5s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=1456 KO=0 ) > restLoginInitiate (OK=734 KO=0 ) > restLoginUsernamePasswordCallback (OK=722 KO=0 ) ---- idc.login ----------------------------------------------------------------- [- ] 0% waiting: 14406491 / active: 22 / done: 722 ================================================================================ 23:10:38.954 [INFO ] i.g.h.e.r.DefaultStatsProcessor - Request 'restLoginUsernamePasswordCallback' failed for user 855: status.find.is(200), but actually found 401 23:10:38.966 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: restLoginUsernamePasswordCallback: KO status.find.is(200), but actually found 401 ========================= Session: Session(idc.login,855,1664230238892,Map(gatling.http.cache.baseUrl -> https://openam-perf-100m-testing.forgeblocks.com:443, gatling.http.cache.dns -> io.gatling.http.cache.DnsCacheSupport$$anon$1@39cf4ea3, username -> user.855, gatling.http.ssl.sslContexts -> SslContexts(io.netty.handler.ssl.ReferenceCountedOpenSslClientContext@3ade78fc,None), callbacks -> {"code":401,"reason":"Unauthorized","message":"Login failure"}, gatling.http.cookies -> CookieJar(Map(CookieKey(amlbcookie,openam-perf-100m-testing.forgeblocks.com,/) -> StoredCookie(amlbcookie=01, domain=openam-perf-100m-testing.forgeblocks.com, path=/, secure, HTTPOnly,false,false,1664230238915))), password -> T35tr0ck123*),0,KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$789/0x0000000800847040@5b64e610) ========================= HTTP request: POST https://openam-perf-100m-testing.forgeblocks.com/am/json/realms/alpha/authenticate?authIndexType=service&authIndexValue=SimpleLogin headers= Accept-API-Version: resource=2.0, protocol=1.0 Accept: application/json cookie: amlbcookie=01 host: openam-perf-100m-testing.forgeblocks.com content-type: application/json content-length: 2190 cookies= amlbcookie=01, domain=openam-perf-100m-testing.forgeblocks.com, path=/, secure, HTTPOnly byteArraysBody={"authId":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhdXRoSW5kZXhWYWx1ZSI6IlNpbXBsZUxvZ2luIiwib3RrIjoiNWFhOWdiNThocHBmYTBqN3YyY3RycWVxYTkiLCJhdXRoSW5kZXhUeXBlIjoic2VydmljZSIsInJlYWxtIjoiL2FscGhhIiwic2Vzc2lvbklkIjoiKkFBSlRTUUFDTURJQUJIUjVjR1VBQ0VwWFZGOUJWVlJJQUFKVE1RQUNNREUuKmV5SjBlWEFpT2lKS1YxUWlMQ0pqZEhraU9pSktWMVFpTENKaGJHY2lPaUpJVXpJMU5pSjkuWlhsS01HVllRV2xQYVVwTFZqRlJhVXhEU214aWJVMXBUMmxLUWsxVVNUUlJNRXBFVEZWb1ZFMXFWVEpKYVhkcFdWZDRia2xxYjJsYVIyeDVTVzR3TGk1WVJUZE5abmMxUzFOV2FsWlNWVTl4TFVkWkxWOVJMbG8yWjE5dVJ6TnJMVGhwWTFoalRuQXROa1F5UlRBMlJVOVNTMHR6T0dFMFVFaG5WbFJmU25KaGRTMUVUWGhFUmtWb2JVZGxjMkozV0ZWVmJFczRiR1pUVkVKWVZrVjVlblJDVkhRdFkwZzVTazFXVmtOSmQwRlhjbnBRVTFGNE1qRk9URzFGWDJoM2FIazRNa05TY1V4WVMzZG9hM0pOTTJOVVNUSmZTRVpDWm5GQ1FtZHBVRk5IYm1zM1pXdFdiMFY0WjFCaFEyOWlRV3B3WldjdFRsb3pkVTAxYVVsQlpqSlZYM2xYYUU5elRGRmhZbmhCWVROWlVURjVUR2c0UWxsbVFrdFRablJoYjJ4S2RYUnhiSGsyVWsxVVdrTkphbTB5T0hGUk9YbE9WRmwwV2tSc1FsTmpWRm81Y1VGUVVHdDZaVEpWZEhSck0wUjBXbFEwZVZkblVpMHdkVjlCVlRjelpqSjNWekZETW1KSVdXSnZhVWs1VlVsVGVGZGFORTVxV0ZSRGVISlNkbGQ2UWsxNFRIUjNjWEJyUmtwblRXTkVWRlJRWVU5RFpYVmxjMmhCTFd0UFdURjJZa1EwVkRWUmVGQmFhbVJqVVhSTFVscExYMUpRYm5CemJIaE5VRmRuWkhOSGVHcHdXRWhSWkdKVE9GTkVXR05QTmpkRE1XeEtRbU0wWVhVMFEzcFJWR3czVnpaTFVUUmljM3BQWTFGMk0zWmlZa3A1UmtsSmRUWnNTSFZhVVdsc2MxSm5TM1IzTFVkV1FsaDVOVGxFZW13MGJEQXhaazl6V1dGQk1FczVlbGszU205bE5tUm9VWE4yYURCQ04wOW5VMjFOTUhacFgxUXRWbTAzVFUxTmJ6Wk9iVGRmVUhSdllXbFFibTkyYTNOU01rNU1lRmRNU1c5RE9URjJaaTFhVEhkNVQyOVJNbmMyY1V4Vk5VWnNUbFowYW04emFESjFaRFp4Wm5RMGFGSnRPWEJrVGtNeloyTnpTSEpDVEZadVdrSnBWSFZNVlcwNFh6VmxXSE5oUlZsbVgzTnRlRmhUTUZCd1QwNDVSR1ZPYzNwSFowRnFaRzlHY0dSaFpFazFURVpYTm1KdU0waHRjamcwZW1Rd2VGVktVRVp0U0dkUmQxSlJWemxRVG14MFJHRllOMUF4Ykhnd1VsWkhXRTFMWW1aeVFqUkJYMlozTTA4d2JEVXdMbHB6VWs0NGRVbDBhVFZLWDAxUmR6ZHJXV1prTTNjLkEyaGlwQ0pjemo1RVRzbVhCOGFmdVZSYVZRQXhDZElfT0NsdzFOekF6eHMiLCJleHAiOjE2NjQyMzA1MzgsImlhdCI6MTY2NDIzMDIzOH0.p9GH9bfUpbyCTGYuPdgX5GIuIwJtdbNfI7PqkB4_xWM","callbacks":[{"type":"NameCallback","output":[{"name":"prompt","value":"User Name"}],"input":[{"name":"IDToken1","value":"user.855"}],"_id":0},{"type":"PasswordCallback","output":[{"name":"prompt","value":"Password"}],"input":[{"name":"IDToken2","value":"T35tr0ck123*"}],"_id":1}]} ========================= HTTP response: status= 401 headers= x-frame-options: SAMEORIGIN x-content-type-options: nosniff cache-control: private content-api-version: resource=2.1 expires: 0 pragma: no-cache content-type: application/json content-length: 62 date: Mon, 26 Sep 2022 22:10:38 GMT x-forgerock-transactionid: 1664230238925-011e7633448f7e369803-859 strict-transport-security: max-age=31536000; includeSubDomains; preload; Via: 1.1 google Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 body= {"code":401,"reason":"Unauthorized","message":"Login failure"} <<<<<<<<<<<<<<<<<<<<<<<<< 23:10:38.967 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-3' crashed with 'j.u.NoSuchElementException: key not found: callbacks', forwarding to the next one 23:10:38.986 [INFO ] i.g.h.e.r.DefaultStatsProcessor - Request 'restloginProgressiveProfileCallback' failed for user 855: jsonPath($.tokenId).find.exists, found nothing 23:10:38.988 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: restloginProgressiveProfileCallback: KO jsonPath($.tokenId).find.exists, found nothing ========================= Session: Session(idc.login,855,1664230238892,Map(gatling.http.cache.baseUrl -> https://openam-perf-100m-testing.forgeblocks.com:443, gatling.http.cache.dns -> io.gatling.http.cache.DnsCacheSupport$$anon$1@39cf4ea3, username -> user.855, gatling.http.ssl.sslContexts -> SslContexts(io.netty.handler.ssl.ReferenceCountedOpenSslClientContext@3ade78fc,None), callbacks -> {"code":401,"reason":"Unauthorized","message":"Login failure"}, gatling.http.cookies -> CookieJar(Map(CookieKey(amlbcookie,openam-perf-100m-testing.forgeblocks.com,/) -> StoredCookie(amlbcookie=01, domain=openam-perf-100m-testing.forgeblocks.com, path=/, secure, HTTPOnly,false,false,1664230238986))), password -> T35tr0ck123*),16,KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$789/0x0000000800847040@5b64e610) ========================= HTTP request: POST https://openam-perf-100m-testing.forgeblocks.com/am/json/realms/alpha/authenticate?authIndexType=service&authIndexValue=SimpleLogin headers= Accept-API-Version: protocol=2.1,resource=1.0 Accept: application/json cookie: amlbcookie=01 host: openam-perf-100m-testing.forgeblocks.com content-type: application/json content-length: 62 cookies= amlbcookie=01, domain=openam-perf-100m-testing.forgeblocks.com, path=/, secure, HTTPOnly byteArraysBody={"code":401,"reason":"Unauthorized","message":"Login failure"} ========================= HTTP response: status= 200 headers= x-frame-options: SAMEORIGIN x-content-type-options: nosniff cache-control: private content-api-version: resource=1.2 expires: 0 pragma: no-cache set-cookie: amlbcookie=01; Path=/; Domain=openam-perf-100m-testing.forgeblocks.com; Secure; HttpOnly; SameSite=none content-type: application/json content-length: 2170 date: Mon, 26 Sep 2022 22:10:38 GMT x-forgerock-transactionid: 1664230238703-a5ad6d17b3014e663878-729 strict-transport-security: max-age=31536000; includeSubDomains; preload; Via: 1.1 google Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 [...]

Error: Server Error

The server encountered a temporary error and could not complete your request.

Please try again in 30 seconds.

<<<<<<<<<<<<<<<<<<<<<<<<< 23:13:33.509 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-6' crashed with 'j.u.NoSuchElementException: No attribute named 'callbacks' is defined', forwarding to the next one 23:13:33.509 [ERROR] i.g.h.a.HttpRequestAction - 'httpRequest-5' failed to execute: No attribute named 'callbacks' is defined 23:13:33.509 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-3' crashed with 'j.u.NoSuchElementException: No attribute named 'callbacks' is defined', forwarding to the next one 23:13:33.509 [ERROR] i.g.h.a.HttpRequestAction - 'httpRequest-2' failed to execute: No attribute named 'callbacks' is defined 23:13:33.509 [ERROR] i.g.h.a.c.AddCookieBuilder$$anon$1 - 'addCookie-1' failed to execute: No attribute named 'tokenId' is defined 23:13:33.509 [INFO ] i.g.h.e.r.DefaultStatsProcessor - Request 'restLoginInitiate' failed for user 280830: status.find.is(200), but actually found 502 23:13:33.509 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: restLoginInitiate: KO status.find.is(200), but actually found 502 ========================= Session: Session(idc.login,280830,1664230404482,Map(gatling.http.cache.baseUrl -> https://openam-perf-100m-testing.forgeblocks.com:443, gatling.http.cache.dns -> io.gatling.http.cache.DnsCacheSupport$$anon$1@39cf4ea3, username -> user.80834, gatling.http.ssl.sslContexts -> SslContexts(io.netty.handler.ssl.ReferenceCountedOpenSslClientContext@66931966,None), gatling.http.referer -> https://openam-perf-100m-testing.forgeblocks.com/am/json/realms/alpha/authenticate?authIndexType=service&authIndexValue=SimpleLogin, password -> T35tr0ck123*),0,KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$789/0x0000000800847040@5b64e610) ========================= HTTP request: POST https://openam-perf-100m-testing.forgeblocks.com/am/json/realms/alpha/authenticate?authIndexType=service&authIndexValue=SimpleLogin headers= Accept-API-Version: resource=2.0, protocol=1.0 Accept: application/json host: openam-perf-100m-testing.forgeblocks.com content-type: application/json content-length: 0 byteArraysBody= ========================= HTTP response: status= 502 Bad Gateway headers= Content-Type: text/html; charset=UTF-8 Referrer-Policy: no-referrer Content-Length: 332 Date: Mon, 26 Sep 2022 22:13:33 GMT Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 body= 502 Server Error

Error: Server Error

The server encountered a temporary error and could not complete your request.

Please try again in 30 seconds.

<<<<<<<<<<<<<<<<<<<<<<<<< 23:13:33.510 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-6' crashed with 'j.u.NoSuchElementException: No attribute named 'callbacks' is defined', forwarding to the next one 23:13:33.510 [ERROR] i.g.h.a.HttpRequestAction - 'httpRequest-5' failed to execute: No attribute named 'callbacks' is defined 23:13:33.510 [ERROR] i.g.c.a.b.SessionHookBuilder$$anon$1 - 'hook-3' crashed with 'j.u.NoSuchElementException: No attribute named 'callbacks' is defined', forwarding to the next one 23:13:33.510 [ERROR] i.g.h.a.HttpRequestAction - 'httpRequest-2' failed to execute: No attribute named 'callbacks' is defined 23:13:33.510 [ERROR] i.g.h.a.c.AddCookieBuilder$$anon$1 - 'addCookie-1' failed to execute: No attribute named 'tokenId' is defined ================================================================================ 2022-09-26 23:13:33 180s elapsed ---- Requests ------------------------------------------------------------------ > Global (OK=231482 KO=203082) > restLoginInitiate (OK=128445 KO=152266) > restLoginUsernamePasswordCallback (OK=103037 KO=25408 ) > restloginProgressiveProfileCallback (OK=0 KO=25408 ) ---- Errors -------------------------------------------------------------------- > status.find.is(200), but actually found 502 201146 (39.63%) > restLoginUsernamePasswordCallback: Failed to build request: No 152266 (30.00%) attribute named 'callbacks' is defined > restloginProgressiveProfileCallback: Failed to build request: 152266 (30.00%) No attribute named 'callbacks' is defined > status.find.is(200), but actually found 503 695 ( 0.14%) > status.find.is(200), but actually found 400 678 ( 0.13%) > status.find.is(200), but actually found 401 294 ( 0.06%) > jsonPath($.tokenId).find.exists, found nothing 269 ( 0.05%) ---- idc.login ----------------------------------------------------------------- [#- ] 1% waiting: 14109879 / active: 16645 / done: 280711 ================================================================================ ---------------------- [26/Sep/2022 23:13:33] Create tar file authn-users.tar.gz ---------------------- RESULTS_FOLDER=/results/gatling/ RESULT_FOLDER=authn-users SIMULATION_PATH=authn-users/loadtest-20220926221032593 PWD=/results/gatling/authn-users/loadtest-20220926221032593 [cmd] tar -czvf /results/gatling//authn-users.tar.gz --exclude=simulation.log * /results/gatling ---------------------- [26/Sep/2022 23:13:33] Update /results/gatling//latest ---------------------- ---------------------- [26/Sep/2022 23:13:33] Update /results/gatling//latest.tar.gz ---------------------- ________________________________________________________________________________ [26/Sep/2022 22:14:13] authn-users show_stderr : Display stderr order ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- Checking pod: overseer-0-c7fc5cc47-x4c97 for file exists, if not wait. [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-c7fc5cc47-x4c97 -i -- bash -c 'if [ -f "/results/orders/authn-users/stderr.txt" ]; then echo "/results/orders/authn-users/stderr.txt file found"; exit 0; else echo "/results/orders/authn-users/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/authn-users/stderr.txt file found" [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- /results/orders/authn-users/stderr.txt file found --- stderr --- [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [run_command]: kubectl --namespace=xlou exec overseer-0-c7fc5cc47-x4c97 -i -- du -k /results/orders/authn-users/stderr.txt [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- 4 /results/orders/authn-users/stderr.txt --- stderr --- - Download file (size 4k) [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- source : /results/orders/authn-users/stderr.txt @ overseer-0-c7fc5cc47-x4c97 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/stderr.txt [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou cp overseer-0-c7fc5cc47-x4c97:/results/orders/authn-users/stderr.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/stderr.txt [loop_until]: (max_time=-1, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- tar: Removing leading `/' from member names --- stderr --- [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/stderr.txt is small (1 lines), printing all lines: ----- output ----- /lodestar/clients/gatling-for-pyrock.sh: line 23: 11170 Terminated /lodestar/clients/gatling/bin/gatling.sh ${OPTIONS} [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [26/Sep/2022 22:14:17] - INFO: Traceback (most recent call last): File "/mnt/disks/data/xslou/lodestar-fork/pyrock/lib/scheduler/tasks/Task.py", line 524, in run self._run() File "/mnt/disks/data/xslou/lodestar-fork/pyrock/lib/scheduler/tasks/OverseerTask.py", line 139, in _run raise FailException(f'order on {self.source.pod_name} stopped prematurely, ' shared.lib.utils.exception.FailException: order on overseer-0-c7fc5cc47-x4c97 stopped prematurely, should stop after 1h Task has been successfully stopped