--Task-- name: authn-users enabled: True class_name: GatlingTask source_name: overseer-0 source_namespace: >default< target_name: am target_namespace: >default< start: 0 stop: 100 timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: [] wait_for: [] options: {'simulation': 'com.forgerock.gatling.Loadtest', 'testCase': 'idc.login', 'testMode': 'open', 'throughput': 1000, 'rampup-duration': 30, 'java-opts': '-Dnum_users=200000 -Drealm=alpha -Duser_password=T35tr0ck123'} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [21/Sep/2022 21:56:16] authn-users pre : N/A ________________________________________________________________________________ [21/Sep/2022 21:56:16] - INFO: setting default timeout for task - 2x global_duration or 300 seconds (whatever is higher): [21/Sep/2022 21:56:16] - INFO: Timeout for this Task has changed to 1h (3600 seconds) Pod log file : /results/orders/authn-users/stdout.txt Pod log : /results/orders/authn-users ________________________________________________________________________________ [21/Sep/2022 21:56:16] authn-users print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ /lodestar/clients/gatling-for-pyrock.sh -s com.forgerock.gatling.Loadtest -sf /lodestar/pyrock/shared/clients/gatling-open/ -rd com.forgerock.gatling.Loadtest@IDC -rf /results/gatling/authn-users --no-reports ________________________________________________________________________________ [21/Sep/2022 21:56:16] authn-users create_order : Create order for overseer with the command to run ________________________________________________________________________________ Userpool size is not defined for the task in conf.yaml and cannot find how many users (entries) are in id-repo, please check the conf.yaml file. Default for selected Gatling simulation will be used. [21/Sep/2022 21:56:16] - INFO: Get overseer pod resources requests memory [loop_until]: kubectl --namespace=xlou get pods overseer-0-c7fc5cc47-x4c97 -o jsonpath={.spec.containers[].resources.requests.memory} [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- 4Gi --- stderr --- Order file : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/order.json { "name": "authn-users", "command": "/lodestar/clients/gatling-for-pyrock.sh -s com.forgerock.gatling.Loadtest -sf /lodestar/pyrock/shared/clients/gatling-open/ -rd com.forgerock.gatling.Loadtest@IDC -rf /results/gatling/authn-users --no-reports", "out-dir": "/results/orders/authn-users", "environment": { "JAVA_OPTS": "-Dam_host=openam-fr-pit-testing18.forgeblocks.com -Dam_oauth2_scope=openid^fr:idm:* -DamadminPassword=M@ythe4th -DamadminUsername=xiaosong.lou@forgerock.com -Didc=true -Dnum_users=200000 -Doauth2_client_id=idmAdminClient -Doauth2_redirect_uri=/platform/appAuthHelperRedirect.html -Drampup_duration=30 -Drealm=alpha -DtestCase=idc.login -DtestMode=open -Dthroughput=1000 -Duser_password=T35tr0ck123 -Xmx4G -Dduration=3996" } } ________________________________________________________________________________ [21/Sep/2022 21:56:17] authn-users loop_until_ready : Loop until overseer pod is ready to accept order ________________________________________________________________________________ [loop_until]: curl -k https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" [loop_until]: (max_time=120, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- WAITING --- stderr --- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 27 0 --:--:-- --:--:-- --:--:-- 27 ________________________________________________________________________________ [21/Sep/2022 21:56:17] authn-users start_order : Send the order to the overseer pod ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou cp /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_am_authn/05_scenario/01_authn-users/order.json overseer-0-c7fc5cc47-x4c97:/results/orders/order.json [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- Order has been successfully forwarded to overseer [21/Sep/2022 21:56:19] - INFO: Check if Gatling simulation started successfully [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-c7fc5cc47-x4c97 -- head /results/orders/authn-users/stdout.txt | grep "started..." [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - failed to find expected output: started... - retry [loop_until]: Function succeeded after 6s (rc=0) - failed to find expected output: started... - retry [loop_until]: Function succeeded after 12s (rc=0) - failed to find expected output: started... - retry [loop_until]: Function succeeded after 18s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- GATLING_HOME is set to /home/jenkins/lodestar/clients/gatling Simulation com.forgerock.gatling.Loadtest started... 22:56:32.060 [INFO ] i.g.h.e.r.DefaultStatsProcessor - Request 'restLoginUsernamePasswordCallback' failed for user 2: status.find.is(200), but actually found 401 22:56:32.060 [INFO ] i.g.h.e.r.DefaultStatsProcessor - Request 'restLoginUsernamePasswordCallback' failed for user 3: status.find.is(200), but actually found 401 22:56:32.072 [DEBUG] i.g.h.e.r.DefaultStatsProcessor - >>>>>>>>>>>>>>>>>>>>>>>>>> Request: restLoginUsernamePasswordCallback: KO status.find.is(200), but actually found 401 ========================= Session: --- stderr --- Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. Checking if the task is still running... Task is still running as expected. ________________________________________________________________________________ [21/Sep/2022 22:26:21] authn-users stop_order : Stop the order on overseer pod ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-c7fc5cc47-x4c97 -- rm -f /results/orders/order.json [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- Order ran during 30m 02s ________________________________________________________________________________ [21/Sep/2022 22:26:22] authn-users download_logs : Download order.tar.gz ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- Checking pod: overseer-0-c7fc5cc47-x4c97 for file exists, if not wait. [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-c7fc5cc47-x4c97 --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-c7fc5cc47-x4c97 -i -- bash -c 'if [ -f "/results/orders/authn-users/order.tar.gz" ]; then echo "/results/orders/authn-users/order.tar.gz file found"; exit 0; else echo "/results/orders/authn-users/order.tar.gz file not found"; exit 1; fi' | grep "/results/orders/authn-users/order.tar.gz file found" [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: Function failed after 0s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 6s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 12s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 18s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 23s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 29s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 35s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 41s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 47s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 53s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 58s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 04s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 10s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 16s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 22s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 27s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 33s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 39s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 45s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 51s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 1m 56s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 02s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 08s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 14s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 20s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 25s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 31s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 37s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 43s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 49s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 2m 54s (rc=1 - expected to be in [0]) - retry [loop_until]: Function failed after 3m (rc=1 - expected to be in [0]) - retry [loop_until]: rc=1, expected [0] --- stdout --- /results/orders/authn-users/order.tar.gz file not found --- stderr --- command terminated with exit code 1 [21/Sep/2022 22:29:23] - INFO: Traceback (most recent call last): File "/mnt/disks/data/xslou/lodestar-fork/pyrock/lib/scheduler/tasks/Task.py", line 524, in run self._run() File "/mnt/disks/data/xslou/lodestar-fork/pyrock/lib/scheduler/tasks/OverseerTask.py", line 146, in _run self.download_logs() File "/mnt/disks/data/xslou/lodestar-fork/pyrock/lib/scheduler/tasks/OverseerTask.py", line 248, in download_logs self.source.download(local_file=local_zip_file, remote_file=remote_zip_file, extract=True) File "/mnt/disks/data/xslou/lodestar-fork/pyrock/lib/components/overseer.py", line 275, in download kubectl(args=f'exec {self.pod_name} -i -- {file_exits_cmd}', expected_output=f'{remote_file} file found', File "/mnt/disks/data/xslou/lodestar-fork/shared/lib/cloud_utils.py", line 87, in kubectl return cmd.loop_until(command, max_time=max_time, interval=interval, verbose=verbose, expected_rc=expected_rc, File "/mnt/disks/data/xslou/lodestar-fork/shared/lib/utils/cmd/cmd.py", line 527, in loop_until raise FailException shared.lib.utils.exception.FailException: None Setting result to FAIL Task has been successfully stopped