--Task-- name: MakeFeeders_ds-idrepo enabled: True class_name: MakeFeedersTask source_name: overseer-0 source_namespace: >default< target_name: ds-idrepo target_namespace: >default< start: 0 stop: None timeout: no timeout loop: False interval: None dependencies: ['MakeUsers_ds-idrepo'] wait_for: [] options: {'out_dir': '/results/ds-idrepo', 'ldif_file': '/results/ds-idrepo/user_entries.ldif', 'zip_file': '/results/ds-idrepo/feeders.tar.gz'} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [04/Aug/2022 19:51:51] MakeFeeders_ds-idrepo pre : Checking task config ________________________________________________________________________________ Pod log file : /results/orders/MakeFeeders_ds-idrepo/stdout.txt Pod log : /results/orders/MakeFeeders_ds-idrepo ________________________________________________________________________________ [04/Aug/2022 19:51:51] MakeFeeders_ds-idrepo print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ python3 /lodestar/pyrock/shared/scripts/ds/generate_feeders.py --ldif-file /results/ds-idrepo/user_entries.ldif --out-dir /results/ds-idrepo --zip-file /results/ds-idrepo/feeders.tar.gz ________________________________________________________________________________ [04/Aug/2022 19:51:51] MakeFeeders_ds-idrepo create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo/order.json { "name": "MakeFeeders_ds-idrepo", "command": "python3 /lodestar/pyrock/shared/scripts/ds/generate_feeders.py --ldif-file /results/ds-idrepo/user_entries.ldif --out-dir /results/ds-idrepo --zip-file /results/ds-idrepo/feeders.tar.gz", "out-dir": "/results/orders/MakeFeeders_ds-idrepo" } ________________________________________________________________________________ [04/Aug/2022 19:51:51] MakeFeeders_ds-idrepo loop_until_ready : Loop until overseer pod is ready to accept order ________________________________________________________________________________ [loop_until]: curl -k https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" [loop_until]: (max_time=120, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- WAITING --- stderr --- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 22 0 --:--:-- --:--:-- --:--:-- 22 ________________________________________________________________________________ [04/Aug/2022 19:51:51] MakeFeeders_ds-idrepo start_order : Send the order to the overseer pod ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-668b4996f-mj826 --- stderr --- [loop_until]: kubectl --namespace=xlou cp /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo/order.json overseer-0-668b4996f-mj826:/results/orders/order.json [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- Order has been successfully forwarded to overseer ________________________________________________________________________________ [04/Aug/2022 19:51:53] MakeFeeders_ds-idrepo wait_stop_by_itself : Wait the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 59s Run command curl --fail -k https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt And exit (continue) if status == WAITING Order is running since 5s, retry Order is running since 10s, retry Order is running since 15s, retry Order is running since 21s, retry Order is running since 26s, retry Order is running since 31s, retry PASS : Order stopped by itself after 37s and before timeout (59s) ________________________________________________________________________________ [04/Aug/2022 19:52:30] MakeFeeders_ds-idrepo stop_order : Stop the order on overseer pod ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-668b4996f-mj826 --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-668b4996f-mj826 -- rm -f /results/orders/order.json [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- Order ran during 38s ________________________________________________________________________________ [04/Aug/2022 19:52:31] MakeFeeders_ds-idrepo download_logs : Download order.tar.gz ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-668b4996f-mj826 --- stderr --- Checking pod: overseer-0-668b4996f-mj826 for file exists, if not wait. [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-668b4996f-mj826 --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-668b4996f-mj826 -i -- bash -c 'if [ -f "/results/orders/MakeFeeders_ds-idrepo/order.tar.gz" ]; then echo "/results/orders/MakeFeeders_ds-idrepo/order.tar.gz file found"; exit 0; else echo "/results/orders/MakeFeeders_ds-idrepo/order.tar.gz file not found"; exit 1; fi' | grep "/results/orders/MakeFeeders_ds-idrepo/order.tar.gz file found" [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- /results/orders/MakeFeeders_ds-idrepo/order.tar.gz file found --- stderr --- [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-668b4996f-mj826 --- stderr --- [run_command]: kubectl --namespace=xlou exec overseer-0-668b4996f-mj826 -i -- du -k /results/orders/MakeFeeders_ds-idrepo/order.tar.gz [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- 4 /results/orders/MakeFeeders_ds-idrepo/order.tar.gz --- stderr --- - Download file (size 4k) [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-668b4996f-mj826 --- stderr --- source : /results/orders/MakeFeeders_ds-idrepo/order.tar.gz @ overseer-0-668b4996f-mj826 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo/order.tar.gz [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods -l app=overseer-0 -o jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-668b4996f-mj826 --- stderr --- [loop_until]: kubectl --namespace=xlou cp overseer-0-668b4996f-mj826:/results/orders/MakeFeeders_ds-idrepo/order.tar.gz /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo/order.tar.gz [loop_until]: (max_time=-1, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- tar: Removing leading `/' from member names --- stderr --- - Extract archive zip file : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo/order.tar.gz destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo ________________________________________________________________________________ [04/Aug/2022 19:52:36] MakeFeeders_ds-idrepo show_cmd : Display order command ________________________________________________________________________________ [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo/cmd.txt is small (1 lines), printing all lines: ----- output ----- python3 /lodestar/pyrock/shared/scripts/ds/generate_feeders.py --ldif-file /results/ds-idrepo/user_entries.ldif --out-dir /results/ds-idrepo --zip-file /results/ds-idrepo/feeders.tar.gz ________________________________________________________________________________ [04/Aug/2022 19:52:36] MakeFeeders_ds-idrepo show_rc : Display rc order ________________________________________________________________________________ [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo/rc.txt is small (1 lines), printing all lines: ----- output ----- CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) ________________________________________________________________________________ [04/Aug/2022 19:52:36] MakeFeeders_ds-idrepo show_stdout : Display stdout order ________________________________________________________________________________ [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo/stdout.txt is small (12 lines), printing all lines: ----- output ----- ----------------------------- Generating feeders, please wait..... ----------------------------- Files generated in 24s - dn file : /results/ds-idrepo/feeders/ds-idrepo.dn - login file : /results/ds-idrepo/feeders/ds-idrepo.login - bind file : /results/ds-idrepo/feeders/ds-idrepo.bind --------------------- Zipping /results/ds-idrepo/feeders, please wait..... --------------------- Zip generated in 8s - zip file : /results/ds-idrepo/feeders.tar.gz Finished without any error in 33s. ________________________________________________________________________________ [04/Aug/2022 19:52:36] MakeFeeders_ds-idrepo show_stderr : Display stderr order ________________________________________________________________________________ [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/saml2/03_Data_init/02_MakeFeeders_ds-idrepo/stderr.txt is small (0 lines), printing all lines: ----- output ----- ________________________________________________________________________________ [04/Aug/2022 19:52:36] MakeFeeders_ds-idrepo check_rc_and_logs : Check order's outputs ________________________________________________________________________________ PASS: no error found in rc log PASS: no error found in stdout log PASS: no error found in stderr log ________________________________________________________________________________ [04/Aug/2022 19:52:36] MakeFeeders_ds-idrepo print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Task result is INIT no need to display logs. ________________________________________________________________________________ [04/Aug/2022 19:52:36] MakeFeeders_ds-idrepo post : Check output ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped