--Task-- name: MakeFeeders_ds-idrepo enabled: True class_name: MakeFeedersTask source_name: overseer-0 source_namespace: >default< target_name: ds-idrepo target_namespace: >default< start: 0.0 stop: None timeout: no timeout loop: False interval: None dependencies: ['MakeUsers_ds-idrepo'] wait_for: [] preceding_task: None options: {'out_dir': '/results/ds-idrepo', 'ldif_file': '/results/ds-idrepo/user_entries.ldif', 'zip_file': '/results/ds-idrepo/feeders.tar.gz'} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [2024-04-25 17:38:32] MakeFeeders_ds-idrepo pre : Checking task config ________________________________________________________________________________ Pod log file : /results/orders/MakeFeeders_ds-idrepo/stdout.txt Pod log : /results/orders/MakeFeeders_ds-idrepo ________________________________________________________________________________ [2024-04-25 17:38:32] MakeFeeders_ds-idrepo print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ python3 /lodestar/pyrock/shared/scripts/ds/generate_feeders.py --ldif-file /results/ds-idrepo/user_entries.ldif --out-dir /results/ds-idrepo --zip-file /results/ds-idrepo/feeders.tar.gz ________________________________________________________________________________ [2024-04-25 17:38:32] MakeFeeders_ds-idrepo create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file: /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/order.json { "name": "MakeFeeders_ds-idrepo", "command": "python3 /lodestar/pyrock/shared/scripts/ds/generate_feeders.py --ldif-file /results/ds-idrepo/user_entries.ldif --out-dir /results/ds-idrepo --zip-file /results/ds-idrepo/feeders.tar.gz", "out-dir": "/results/orders/MakeFeeders_ds-idrepo" } ________________________________________________________________________________ [2024-04-25 17:38:32] MakeFeeders_ds-idrepo loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2024-04-25 17:38:32,449 INFO 2024-04-25 17:38:32,449 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2024-04-25 17:38:32,449 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2024-04-25 17:38:32,748 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-25 17:38:32,748 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:38:32,748 DEBUG --- stdout --- 2024-04-25 17:38:32,749 DEBUG WAITING 2024-04-25 17:38:32,749 DEBUG --- stderr --- 2024-04-25 17:38:32,749 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 25 0 --:--:-- --:--:-- --:--:-- 25 ________________________________________________________________________________ [2024-04-25 17:38:32] MakeFeeders_ds-idrepo start_order : Send the order to the overseer pod ________________________________________________________________________________ 2024-04-25 17:38:32,749 INFO 2024-04-25 17:38:32,749 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:38:32,749 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:38:33,074 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:38:33,074 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:38:33,074 DEBUG --- stdout --- 2024-04-25 17:38:33,074 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:38:33,074 DEBUG --- stderr --- 2024-04-25 17:38:33,074 DEBUG 2024-04-25 17:38:33,074 INFO 2024-04-25 17:38:33,074 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/order.json overseer-0-56c5c978d9-v8wm9:/results/orders/order.json 2024-04-25 17:38:33,074 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-25 17:38:34,438 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:38:34,438 DEBUG --- stdout --- 2024-04-25 17:38:34,438 DEBUG 2024-04-25 17:38:34,438 DEBUG --- stderr --- 2024-04-25 17:38:34,438 DEBUG Order has been successfully forwarded to overseer ________________________________________________________________________________ [2024-04-25 17:38:34] MakeFeeders_ds-idrepo wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 59s Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING Order is running since 5s, retry Order is running since 10s, retry Order is running since 15s, retry Order is running since 20s, retry Order is running since 26s, retry PASS : Order stopped by itself after 31s and before timeout (59s) ________________________________________________________________________________ [2024-04-25 17:39:05] MakeFeeders_ds-idrepo stop_order : Stop the order on overseer pod ________________________________________________________________________________ 2024-04-25 17:39:05,776 INFO 2024-04-25 17:39:05,776 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:05,776 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:06,079 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:06,079 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:06,080 DEBUG --- stdout --- 2024-04-25 17:39:06,080 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:06,080 DEBUG --- stderr --- 2024-04-25 17:39:06,080 DEBUG 2024-04-25 17:39:06,080 INFO 2024-04-25 17:39:06,080 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-56c5c978d9-v8wm9 -- rm -f /results/orders/order.json 2024-04-25 17:39:06,080 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-25 17:39:06,840 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:06,840 DEBUG --- stdout --- 2024-04-25 17:39:06,840 DEBUG 2024-04-25 17:39:06,840 DEBUG --- stderr --- 2024-04-25 17:39:06,840 DEBUG Order ran during 32s ________________________________________________________________________________ [2024-04-25 17:39:06] MakeFeeders_ds-idrepo download_logs : Download individual order files ________________________________________________________________________________ [2024-04-25 17:39:06] - PASS: file already downloaded /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/order.json 2024-04-25 17:39:06,841 INFO 2024-04-25 17:39:06,841 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:06,841 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:07,161 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:07,162 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:07,162 DEBUG --- stdout --- 2024-04-25 17:39:07,162 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:07,162 DEBUG --- stderr --- 2024-04-25 17:39:07,162 DEBUG Checking pod: overseer-0-56c5c978d9-v8wm9 for file exists, if not wait. 2024-04-25 17:39:07,162 INFO 2024-04-25 17:39:07,162 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:07,162 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:07,477 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:07,478 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:07,478 DEBUG --- stdout --- 2024-04-25 17:39:07,478 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:07,478 DEBUG --- stderr --- 2024-04-25 17:39:07,478 DEBUG 2024-04-25 17:39:07,479 INFO 2024-04-25 17:39:07,479 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-56c5c978d9-v8wm9 --stdin -- bash -c 'if [ -f "/results/orders/MakeFeeders_ds-idrepo/rc.txt" ]; then echo "/results/orders/MakeFeeders_ds-idrepo/rc.txt file found"; exit 0; else echo "/results/orders/MakeFeeders_ds-idrepo/rc.txt file not found"; exit 1; fi' | grep "/results/orders/MakeFeeders_ds-idrepo/rc.txt file found" 2024-04-25 17:39:07,479 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-25 17:39:08,311 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-25 17:39:08,311 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:08,311 DEBUG --- stdout --- 2024-04-25 17:39:08,311 DEBUG /results/orders/MakeFeeders_ds-idrepo/rc.txt file found 2024-04-25 17:39:08,311 DEBUG --- stderr --- 2024-04-25 17:39:08,311 DEBUG 2024-04-25 17:39:08,311 INFO 2024-04-25 17:39:08,311 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:08,311 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:08,614 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:08,614 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:08,614 DEBUG --- stdout --- 2024-04-25 17:39:08,614 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:08,614 DEBUG --- stderr --- 2024-04-25 17:39:08,614 DEBUG 2024-04-25 17:39:08,614 INFO 2024-04-25 17:39:08,615 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-56c5c978d9-v8wm9 --stdin -- du -k /results/orders/MakeFeeders_ds-idrepo/rc.txt 2024-04-25 17:39:09,436 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-25 17:39:09,436 DEBUG --- stdout --- 2024-04-25 17:39:09,436 DEBUG 4 /results/orders/MakeFeeders_ds-idrepo/rc.txt 2024-04-25 17:39:09,436 DEBUG --- stderr --- 2024-04-25 17:39:09,436 DEBUG 2024-04-25 17:39:09,436 INFO - Download file (size 4k) 2024-04-25 17:39:09,437 INFO 2024-04-25 17:39:09,437 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:09,437 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:09,761 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:09,762 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:09,762 DEBUG --- stdout --- 2024-04-25 17:39:09,762 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:09,762 DEBUG --- stderr --- 2024-04-25 17:39:09,762 DEBUG source : /results/orders/MakeFeeders_ds-idrepo/rc.txt @ overseer-0-56c5c978d9-v8wm9 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/rc.txt 2024-04-25 17:39:09,762 INFO 2024-04-25 17:39:09,762 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:09,762 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:10,066 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:10,066 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:10,066 DEBUG --- stdout --- 2024-04-25 17:39:10,066 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:10,067 DEBUG --- stderr --- 2024-04-25 17:39:10,067 DEBUG 2024-04-25 17:39:10,067 INFO 2024-04-25 17:39:10,067 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-56c5c978d9-v8wm9:/results/orders/MakeFeeders_ds-idrepo/rc.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/rc.txt 2024-04-25 17:39:10,067 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-25 17:39:10,818 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:10,818 DEBUG --- stdout --- 2024-04-25 17:39:10,818 DEBUG tar: Removing leading `/' from member names 2024-04-25 17:39:10,818 DEBUG --- stderr --- 2024-04-25 17:39:10,818 DEBUG 2024-04-25 17:39:10,819 INFO 2024-04-25 17:39:10,819 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:10,819 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:11,149 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:11,149 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:11,149 DEBUG --- stdout --- 2024-04-25 17:39:11,149 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:11,149 DEBUG --- stderr --- 2024-04-25 17:39:11,149 DEBUG Checking pod: overseer-0-56c5c978d9-v8wm9 for file exists, if not wait. 2024-04-25 17:39:11,150 INFO 2024-04-25 17:39:11,150 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:11,150 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:11,483 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:11,483 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:11,483 DEBUG --- stdout --- 2024-04-25 17:39:11,483 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:11,483 DEBUG --- stderr --- 2024-04-25 17:39:11,483 DEBUG 2024-04-25 17:39:11,484 INFO 2024-04-25 17:39:11,484 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-56c5c978d9-v8wm9 --stdin -- bash -c 'if [ -f "/results/orders/MakeFeeders_ds-idrepo/stderr.txt" ]; then echo "/results/orders/MakeFeeders_ds-idrepo/stderr.txt file found"; exit 0; else echo "/results/orders/MakeFeeders_ds-idrepo/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/MakeFeeders_ds-idrepo/stderr.txt file found" 2024-04-25 17:39:11,484 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-25 17:39:12,313 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-25 17:39:12,313 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:12,313 DEBUG --- stdout --- 2024-04-25 17:39:12,314 DEBUG /results/orders/MakeFeeders_ds-idrepo/stderr.txt file found 2024-04-25 17:39:12,314 DEBUG --- stderr --- 2024-04-25 17:39:12,314 DEBUG 2024-04-25 17:39:12,314 INFO 2024-04-25 17:39:12,314 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:12,314 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:12,614 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:12,614 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:12,614 DEBUG --- stdout --- 2024-04-25 17:39:12,614 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:12,614 DEBUG --- stderr --- 2024-04-25 17:39:12,614 DEBUG 2024-04-25 17:39:12,614 INFO 2024-04-25 17:39:12,615 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-56c5c978d9-v8wm9 --stdin -- du -k /results/orders/MakeFeeders_ds-idrepo/stderr.txt 2024-04-25 17:39:13,443 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-25 17:39:13,444 DEBUG --- stdout --- 2024-04-25 17:39:13,444 DEBUG 0 /results/orders/MakeFeeders_ds-idrepo/stderr.txt 2024-04-25 17:39:13,444 DEBUG --- stderr --- 2024-04-25 17:39:13,444 DEBUG 2024-04-25 17:39:13,444 INFO - Download file (size 0k) 2024-04-25 17:39:13,444 INFO 2024-04-25 17:39:13,444 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:13,444 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:13,755 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:13,755 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:13,755 DEBUG --- stdout --- 2024-04-25 17:39:13,756 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:13,756 DEBUG --- stderr --- 2024-04-25 17:39:13,756 DEBUG source : /results/orders/MakeFeeders_ds-idrepo/stderr.txt @ overseer-0-56c5c978d9-v8wm9 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stderr.txt 2024-04-25 17:39:13,756 INFO 2024-04-25 17:39:13,756 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:13,756 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:14,061 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:14,061 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:14,061 DEBUG --- stdout --- 2024-04-25 17:39:14,061 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:14,061 DEBUG --- stderr --- 2024-04-25 17:39:14,061 DEBUG 2024-04-25 17:39:14,062 INFO 2024-04-25 17:39:14,062 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-56c5c978d9-v8wm9:/results/orders/MakeFeeders_ds-idrepo/stderr.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stderr.txt 2024-04-25 17:39:14,062 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-25 17:39:14,822 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:14,822 DEBUG --- stdout --- 2024-04-25 17:39:14,822 DEBUG tar: Removing leading `/' from member names 2024-04-25 17:39:14,822 DEBUG --- stderr --- 2024-04-25 17:39:14,822 DEBUG 2024-04-25 17:39:14,823 INFO 2024-04-25 17:39:14,823 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:14,823 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:15,150 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:15,151 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:15,151 DEBUG --- stdout --- 2024-04-25 17:39:15,151 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:15,151 DEBUG --- stderr --- 2024-04-25 17:39:15,151 DEBUG Checking pod: overseer-0-56c5c978d9-v8wm9 for file exists, if not wait. 2024-04-25 17:39:15,151 INFO 2024-04-25 17:39:15,151 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:15,151 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:15,457 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:15,457 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:15,457 DEBUG --- stdout --- 2024-04-25 17:39:15,458 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:15,458 DEBUG --- stderr --- 2024-04-25 17:39:15,458 DEBUG 2024-04-25 17:39:15,458 INFO 2024-04-25 17:39:15,459 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-56c5c978d9-v8wm9 --stdin -- bash -c 'if [ -f "/results/orders/MakeFeeders_ds-idrepo/stdout.txt" ]; then echo "/results/orders/MakeFeeders_ds-idrepo/stdout.txt file found"; exit 0; else echo "/results/orders/MakeFeeders_ds-idrepo/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/MakeFeeders_ds-idrepo/stdout.txt file found" 2024-04-25 17:39:15,459 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-25 17:39:16,277 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-25 17:39:16,279 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:16,279 DEBUG --- stdout --- 2024-04-25 17:39:16,279 DEBUG /results/orders/MakeFeeders_ds-idrepo/stdout.txt file found 2024-04-25 17:39:16,279 DEBUG --- stderr --- 2024-04-25 17:39:16,279 DEBUG 2024-04-25 17:39:16,279 INFO 2024-04-25 17:39:16,279 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:16,279 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:16,592 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:16,593 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:16,593 DEBUG --- stdout --- 2024-04-25 17:39:16,593 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:16,593 DEBUG --- stderr --- 2024-04-25 17:39:16,593 DEBUG 2024-04-25 17:39:16,593 INFO 2024-04-25 17:39:16,593 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-56c5c978d9-v8wm9 --stdin -- du -k /results/orders/MakeFeeders_ds-idrepo/stdout.txt 2024-04-25 17:39:17,501 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-25 17:39:17,501 DEBUG --- stdout --- 2024-04-25 17:39:17,501 DEBUG 4 /results/orders/MakeFeeders_ds-idrepo/stdout.txt 2024-04-25 17:39:17,501 DEBUG --- stderr --- 2024-04-25 17:39:17,501 DEBUG 2024-04-25 17:39:17,501 INFO - Download file (size 4k) 2024-04-25 17:39:17,502 INFO 2024-04-25 17:39:17,502 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:17,502 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:17,809 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:17,809 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:17,809 DEBUG --- stdout --- 2024-04-25 17:39:17,809 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:17,809 DEBUG --- stderr --- 2024-04-25 17:39:17,809 DEBUG source : /results/orders/MakeFeeders_ds-idrepo/stdout.txt @ overseer-0-56c5c978d9-v8wm9 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stdout.txt 2024-04-25 17:39:17,810 INFO 2024-04-25 17:39:17,810 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:17,810 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:18,116 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:18,116 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:18,116 DEBUG --- stdout --- 2024-04-25 17:39:18,116 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:18,116 DEBUG --- stderr --- 2024-04-25 17:39:18,116 DEBUG 2024-04-25 17:39:18,116 INFO 2024-04-25 17:39:18,117 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-56c5c978d9-v8wm9:/results/orders/MakeFeeders_ds-idrepo/stdout.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stdout.txt 2024-04-25 17:39:18,117 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-25 17:39:18,859 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:18,859 DEBUG --- stdout --- 2024-04-25 17:39:18,859 DEBUG tar: Removing leading `/' from member names 2024-04-25 17:39:18,859 DEBUG --- stderr --- 2024-04-25 17:39:18,859 DEBUG ________________________________________________________________________________ [2024-04-25 17:39:18] MakeFeeders_ds-idrepo show_cmd : Display order command ________________________________________________________________________________ 2024-04-25 17:39:18,860 INFO 2024-04-25 17:39:18,860 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:18,860 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:19,167 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:19,167 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:19,167 DEBUG --- stdout --- 2024-04-25 17:39:19,167 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:19,167 DEBUG --- stderr --- 2024-04-25 17:39:19,167 DEBUG Checking pod: overseer-0-56c5c978d9-v8wm9 for file exists, if not wait. 2024-04-25 17:39:19,167 INFO 2024-04-25 17:39:19,167 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:19,167 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:19,473 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:19,473 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:19,473 DEBUG --- stdout --- 2024-04-25 17:39:19,473 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:19,473 DEBUG --- stderr --- 2024-04-25 17:39:19,473 DEBUG 2024-04-25 17:39:19,474 INFO 2024-04-25 17:39:19,474 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-56c5c978d9-v8wm9 --stdin -- bash -c 'if [ -f "/results/orders/MakeFeeders_ds-idrepo/cmd.txt" ]; then echo "/results/orders/MakeFeeders_ds-idrepo/cmd.txt file found"; exit 0; else echo "/results/orders/MakeFeeders_ds-idrepo/cmd.txt file not found"; exit 1; fi' | grep "/results/orders/MakeFeeders_ds-idrepo/cmd.txt file found" 2024-04-25 17:39:19,474 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-25 17:39:20,296 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-25 17:39:20,296 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:20,296 DEBUG --- stdout --- 2024-04-25 17:39:20,296 DEBUG /results/orders/MakeFeeders_ds-idrepo/cmd.txt file found 2024-04-25 17:39:20,296 DEBUG --- stderr --- 2024-04-25 17:39:20,296 DEBUG 2024-04-25 17:39:20,297 INFO 2024-04-25 17:39:20,297 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:20,297 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:20,663 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:20,664 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:20,664 DEBUG --- stdout --- 2024-04-25 17:39:20,664 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:20,664 DEBUG --- stderr --- 2024-04-25 17:39:20,664 DEBUG 2024-04-25 17:39:20,664 INFO 2024-04-25 17:39:20,664 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-56c5c978d9-v8wm9 --stdin -- du -k /results/orders/MakeFeeders_ds-idrepo/cmd.txt 2024-04-25 17:39:21,510 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-25 17:39:21,512 DEBUG --- stdout --- 2024-04-25 17:39:21,512 DEBUG 4 /results/orders/MakeFeeders_ds-idrepo/cmd.txt 2024-04-25 17:39:21,512 DEBUG --- stderr --- 2024-04-25 17:39:21,512 DEBUG 2024-04-25 17:39:21,512 INFO - Download file (size 4k) 2024-04-25 17:39:21,513 INFO 2024-04-25 17:39:21,513 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:21,513 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:21,860 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:21,860 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:21,860 DEBUG --- stdout --- 2024-04-25 17:39:21,860 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:21,861 DEBUG --- stderr --- 2024-04-25 17:39:21,861 DEBUG source : /results/orders/MakeFeeders_ds-idrepo/cmd.txt @ overseer-0-56c5c978d9-v8wm9 destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/cmd.txt 2024-04-25 17:39:21,861 INFO 2024-04-25 17:39:21,861 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-25 17:39:21,861 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-25 17:39:22,196 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-25 17:39:22,196 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:22,196 DEBUG --- stdout --- 2024-04-25 17:39:22,196 DEBUG overseer-0-56c5c978d9-v8wm9 2024-04-25 17:39:22,196 DEBUG --- stderr --- 2024-04-25 17:39:22,196 DEBUG 2024-04-25 17:39:22,197 INFO 2024-04-25 17:39:22,197 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-56c5c978d9-v8wm9:/results/orders/MakeFeeders_ds-idrepo/cmd.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/cmd.txt 2024-04-25 17:39:22,197 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-25 17:39:22,973 INFO [loop_until]: OK (rc = 0) 2024-04-25 17:39:22,973 DEBUG --- stdout --- 2024-04-25 17:39:22,973 DEBUG tar: Removing leading `/' from member names 2024-04-25 17:39:22,973 DEBUG --- stderr --- 2024-04-25 17:39:22,973 DEBUG 2024-04-25 17:39:22,973 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/cmd.txt is small (1 lines), printing all lines: 2024-04-25 17:39:22,973 DEBUG ----- output ----- 2024-04-25 17:39:22,974 DEBUG python3 /lodestar/pyrock/shared/scripts/ds/generate_feeders.py --ldif-file /results/ds-idrepo/user_entries.ldif --out-dir /results/ds-idrepo --zip-file /results/ds-idrepo/feeders.tar.gz 2024-04-25 17:39:22,974 INFO ________________________________________________________________________________ [2024-04-25 17:39:22] MakeFeeders_ds-idrepo show_rc : Display rc order ________________________________________________________________________________ 2024-04-25 17:39:22,974 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/rc.txt is small (1 lines), printing all lines: 2024-04-25 17:39:22,974 DEBUG ----- output ----- 2024-04-25 17:39:22,974 DEBUG CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) 2024-04-25 17:39:22,974 INFO ________________________________________________________________________________ [2024-04-25 17:39:22] MakeFeeders_ds-idrepo show_stdout : Display stdout order ________________________________________________________________________________ 2024-04-25 17:39:22,974 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stdout.txt is small (12 lines), printing all lines: 2024-04-25 17:39:22,974 DEBUG ----- output ----- 2024-04-25 17:39:22,974 DEBUG ----------------------------- Generating feeders, please wait..... ----------------------------- Files generated in 20s - dn file : /results/ds-idrepo/feeders/ds-idrepo.dn - login file : /results/ds-idrepo/feeders/ds-idrepo.login - bind file : /results/ds-idrepo/feeders/ds-idrepo.bind --------------------- Zipping /results/ds-idrepo/feeders, please wait..... --------------------- Zip generated in 8s - zip file : /results/ds-idrepo/feeders.tar.gz Finished without any error in 29s. 2024-04-25 17:39:22,974 INFO ________________________________________________________________________________ [2024-04-25 17:39:22] MakeFeeders_ds-idrepo show_stderr : Display stderr order ________________________________________________________________________________ 2024-04-25 17:39:22,975 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stderr.txt is small (0 lines), printing all lines: 2024-04-25 17:39:22,975 DEBUG ----- output ----- 2024-04-25 17:39:22,975 DEBUG 2024-04-25 17:39:22,975 INFO ________________________________________________________________________________ [2024-04-25 17:39:22] MakeFeeders_ds-idrepo check_rc_and_logs : Check order's outputs ________________________________________________________________________________ PASS: no error found in rc log PASS: no error found in stdout log PASS: no error found in stderr log ________________________________________________________________________________ [2024-04-25 17:39:22] MakeFeeders_ds-idrepo print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Task result is INIT no need to display logs. ________________________________________________________________________________ [2024-04-25 17:39:22] MakeFeeders_ds-idrepo post : Check output ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped