--Task-- name: MakeFeeders_ds-idrepo enabled: True class_name: MakeFeedersTask source_name: overseer-0 source_namespace: >default< target_name: ds-idrepo target_namespace: >default< start: 0.0 stop: None timeout: no timeout loop: False interval: None dependencies: ['MakeUsers_ds-idrepo'] wait_for: [] preceding_task: None options: {'out_dir': '/results/ds-idrepo', 'ldif_file': '/results/ds-idrepo/user_entries.ldif', 'zip_file': '/results/ds-idrepo/feeders.tar.gz'} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [2024-04-11 01:40:40] MakeFeeders_ds-idrepo pre : Checking task config ________________________________________________________________________________ Pod log file : /results/orders/MakeFeeders_ds-idrepo/stdout.txt Pod log : /results/orders/MakeFeeders_ds-idrepo ________________________________________________________________________________ [2024-04-11 01:40:40] MakeFeeders_ds-idrepo print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ python3 /lodestar/pyrock/shared/scripts/ds/generate_feeders.py --ldif-file /results/ds-idrepo/user_entries.ldif --out-dir /results/ds-idrepo --zip-file /results/ds-idrepo/feeders.tar.gz ________________________________________________________________________________ [2024-04-11 01:40:40] MakeFeeders_ds-idrepo create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file: /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/order.json { "name": "MakeFeeders_ds-idrepo", "command": "python3 /lodestar/pyrock/shared/scripts/ds/generate_feeders.py --ldif-file /results/ds-idrepo/user_entries.ldif --out-dir /results/ds-idrepo --zip-file /results/ds-idrepo/feeders.tar.gz", "out-dir": "/results/orders/MakeFeeders_ds-idrepo" } ________________________________________________________________________________ [2024-04-11 01:40:40] MakeFeeders_ds-idrepo loop_until_ready : Wait for preceding task to finish if it uses same Overseer as a source, then loop for max_time until Overseer pod status is READY to accept order :param max_time: periodically check overseer status for this many seconds at maximum :param interval: interval between two periodical checks :return: ________________________________________________________________________________ 2024-04-11 01:40:40,479 INFO 2024-04-11 01:40:40,480 INFO [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" 2024-04-11 01:40:40,480 INFO [loop_until]: (max_time=300, interval=5, expected_rc=[0] 2024-04-11 01:40:40,777 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-11 01:40:40,777 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:40:40,778 DEBUG --- stdout --- 2024-04-11 01:40:40,778 DEBUG WAITING 2024-04-11 01:40:40,778 DEBUG --- stderr --- 2024-04-11 01:40:40,778 DEBUG % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 26 0 --:--:-- --:--:-- --:--:-- 26 ________________________________________________________________________________ [2024-04-11 01:40:40] MakeFeeders_ds-idrepo start_order : Send the order to the overseer pod ________________________________________________________________________________ 2024-04-11 01:40:40,779 INFO 2024-04-11 01:40:40,779 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:40:40,779 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:40:41,118 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:40:41,118 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:40:41,118 DEBUG --- stdout --- 2024-04-11 01:40:41,118 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:40:41,118 DEBUG --- stderr --- 2024-04-11 01:40:41,118 DEBUG 2024-04-11 01:40:41,119 INFO 2024-04-11 01:40:41,119 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/order.json overseer-0-78845f4f8c-v2r2q:/results/orders/order.json 2024-04-11 01:40:41,119 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 01:40:42,497 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:40:42,497 DEBUG --- stdout --- 2024-04-11 01:40:42,497 DEBUG 2024-04-11 01:40:42,497 DEBUG --- stderr --- 2024-04-11 01:40:42,497 DEBUG Order has been successfully forwarded to overseer ________________________________________________________________________________ [2024-04-11 01:40:42] MakeFeeders_ds-idrepo wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 59s Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING PASS : Order stopped by itself after 2m and before timeout (59s) ________________________________________________________________________________ [2024-04-11 01:42:42] MakeFeeders_ds-idrepo stop_order : Stop the order on overseer pod ________________________________________________________________________________ 2024-04-11 01:42:42,813 INFO 2024-04-11 01:42:42,813 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:42,813 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:43,151 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:43,152 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:43,152 DEBUG --- stdout --- 2024-04-11 01:42:43,152 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:43,152 DEBUG --- stderr --- 2024-04-11 01:42:43,152 DEBUG 2024-04-11 01:42:43,152 INFO 2024-04-11 01:42:43,152 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-78845f4f8c-v2r2q -- rm -f /results/orders/order.json 2024-04-11 01:42:43,152 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 01:42:43,974 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:43,975 DEBUG --- stdout --- 2024-04-11 01:42:43,975 DEBUG 2024-04-11 01:42:43,975 DEBUG --- stderr --- 2024-04-11 01:42:43,975 DEBUG Order ran during 2m 01s ________________________________________________________________________________ [2024-04-11 01:42:43] MakeFeeders_ds-idrepo download_logs : Download individual order files ________________________________________________________________________________ [2024-04-11 01:42:43] - PASS: file already downloaded /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/order.json 2024-04-11 01:42:43,976 INFO 2024-04-11 01:42:43,976 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:43,976 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:44,284 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:44,284 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:44,284 DEBUG --- stdout --- 2024-04-11 01:42:44,284 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:44,284 DEBUG --- stderr --- 2024-04-11 01:42:44,284 DEBUG Checking pod: overseer-0-78845f4f8c-v2r2q for file exists, if not wait. 2024-04-11 01:42:44,285 INFO 2024-04-11 01:42:44,285 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:44,285 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:44,590 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:44,590 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:44,590 DEBUG --- stdout --- 2024-04-11 01:42:44,590 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:44,590 DEBUG --- stderr --- 2024-04-11 01:42:44,590 DEBUG 2024-04-11 01:42:44,591 INFO 2024-04-11 01:42:44,591 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-78845f4f8c-v2r2q --stdin -- bash -c 'if [ -f "/results/orders/MakeFeeders_ds-idrepo/rc.txt" ]; then echo "/results/orders/MakeFeeders_ds-idrepo/rc.txt file found"; exit 0; else echo "/results/orders/MakeFeeders_ds-idrepo/rc.txt file not found"; exit 1; fi' | grep "/results/orders/MakeFeeders_ds-idrepo/rc.txt file found" 2024-04-11 01:42:44,591 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 01:42:45,430 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-11 01:42:45,430 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:45,430 DEBUG --- stdout --- 2024-04-11 01:42:45,430 DEBUG /results/orders/MakeFeeders_ds-idrepo/rc.txt file found 2024-04-11 01:42:45,430 DEBUG --- stderr --- 2024-04-11 01:42:45,430 DEBUG 2024-04-11 01:42:45,430 INFO 2024-04-11 01:42:45,430 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:45,430 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:45,802 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:45,802 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:45,802 DEBUG --- stdout --- 2024-04-11 01:42:45,802 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:45,802 DEBUG --- stderr --- 2024-04-11 01:42:45,802 DEBUG 2024-04-11 01:42:45,803 INFO 2024-04-11 01:42:45,803 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-78845f4f8c-v2r2q --stdin -- du -k /results/orders/MakeFeeders_ds-idrepo/rc.txt 2024-04-11 01:42:46,637 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-11 01:42:46,637 DEBUG --- stdout --- 2024-04-11 01:42:46,637 DEBUG 4 /results/orders/MakeFeeders_ds-idrepo/rc.txt 2024-04-11 01:42:46,637 DEBUG --- stderr --- 2024-04-11 01:42:46,637 DEBUG 2024-04-11 01:42:46,637 INFO - Download file (size 4k) 2024-04-11 01:42:46,638 INFO 2024-04-11 01:42:46,638 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:46,638 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:47,031 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:47,032 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:47,032 DEBUG --- stdout --- 2024-04-11 01:42:47,032 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:47,032 DEBUG --- stderr --- 2024-04-11 01:42:47,032 DEBUG source : /results/orders/MakeFeeders_ds-idrepo/rc.txt @ overseer-0-78845f4f8c-v2r2q destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/rc.txt 2024-04-11 01:42:47,033 INFO 2024-04-11 01:42:47,033 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:47,033 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:47,405 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:47,406 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:47,406 DEBUG --- stdout --- 2024-04-11 01:42:47,406 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:47,406 DEBUG --- stderr --- 2024-04-11 01:42:47,407 DEBUG 2024-04-11 01:42:47,407 INFO 2024-04-11 01:42:47,407 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-78845f4f8c-v2r2q:/results/orders/MakeFeeders_ds-idrepo/rc.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/rc.txt 2024-04-11 01:42:47,408 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-11 01:42:48,199 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:48,200 DEBUG --- stdout --- 2024-04-11 01:42:48,200 DEBUG tar: Removing leading `/' from member names 2024-04-11 01:42:48,200 DEBUG --- stderr --- 2024-04-11 01:42:48,200 DEBUG 2024-04-11 01:42:48,200 INFO 2024-04-11 01:42:48,200 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:48,200 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:48,505 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:48,505 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:48,505 DEBUG --- stdout --- 2024-04-11 01:42:48,505 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:48,505 DEBUG --- stderr --- 2024-04-11 01:42:48,505 DEBUG Checking pod: overseer-0-78845f4f8c-v2r2q for file exists, if not wait. 2024-04-11 01:42:48,506 INFO 2024-04-11 01:42:48,506 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:48,506 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:48,807 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:48,807 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:48,808 DEBUG --- stdout --- 2024-04-11 01:42:48,808 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:48,808 DEBUG --- stderr --- 2024-04-11 01:42:48,808 DEBUG 2024-04-11 01:42:48,808 INFO 2024-04-11 01:42:48,809 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-78845f4f8c-v2r2q --stdin -- bash -c 'if [ -f "/results/orders/MakeFeeders_ds-idrepo/stderr.txt" ]; then echo "/results/orders/MakeFeeders_ds-idrepo/stderr.txt file found"; exit 0; else echo "/results/orders/MakeFeeders_ds-idrepo/stderr.txt file not found"; exit 1; fi' | grep "/results/orders/MakeFeeders_ds-idrepo/stderr.txt file found" 2024-04-11 01:42:48,809 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 01:42:49,628 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-11 01:42:49,628 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:49,628 DEBUG --- stdout --- 2024-04-11 01:42:49,628 DEBUG /results/orders/MakeFeeders_ds-idrepo/stderr.txt file found 2024-04-11 01:42:49,628 DEBUG --- stderr --- 2024-04-11 01:42:49,628 DEBUG 2024-04-11 01:42:49,629 INFO 2024-04-11 01:42:49,629 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:49,629 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:49,932 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:49,932 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:49,932 DEBUG --- stdout --- 2024-04-11 01:42:49,932 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:49,932 DEBUG --- stderr --- 2024-04-11 01:42:49,932 DEBUG 2024-04-11 01:42:49,933 INFO 2024-04-11 01:42:49,933 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-78845f4f8c-v2r2q --stdin -- du -k /results/orders/MakeFeeders_ds-idrepo/stderr.txt 2024-04-11 01:42:50,763 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-11 01:42:50,763 DEBUG --- stdout --- 2024-04-11 01:42:50,763 DEBUG 0 /results/orders/MakeFeeders_ds-idrepo/stderr.txt 2024-04-11 01:42:50,763 DEBUG --- stderr --- 2024-04-11 01:42:50,763 DEBUG 2024-04-11 01:42:50,763 INFO - Download file (size 0k) 2024-04-11 01:42:50,764 INFO 2024-04-11 01:42:50,764 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:50,764 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:51,091 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:51,091 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:51,091 DEBUG --- stdout --- 2024-04-11 01:42:51,091 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:51,092 DEBUG --- stderr --- 2024-04-11 01:42:51,092 DEBUG source : /results/orders/MakeFeeders_ds-idrepo/stderr.txt @ overseer-0-78845f4f8c-v2r2q destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stderr.txt 2024-04-11 01:42:51,092 INFO 2024-04-11 01:42:51,092 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:51,092 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:51,450 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:51,450 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:51,450 DEBUG --- stdout --- 2024-04-11 01:42:51,450 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:51,450 DEBUG --- stderr --- 2024-04-11 01:42:51,450 DEBUG 2024-04-11 01:42:51,451 INFO 2024-04-11 01:42:51,451 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-78845f4f8c-v2r2q:/results/orders/MakeFeeders_ds-idrepo/stderr.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stderr.txt 2024-04-11 01:42:51,451 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-11 01:42:52,291 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:52,291 DEBUG --- stdout --- 2024-04-11 01:42:52,292 DEBUG tar: Removing leading `/' from member names 2024-04-11 01:42:52,292 DEBUG --- stderr --- 2024-04-11 01:42:52,292 DEBUG 2024-04-11 01:42:52,292 INFO 2024-04-11 01:42:52,292 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:52,293 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:52,651 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:52,651 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:52,651 DEBUG --- stdout --- 2024-04-11 01:42:52,651 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:52,651 DEBUG --- stderr --- 2024-04-11 01:42:52,651 DEBUG Checking pod: overseer-0-78845f4f8c-v2r2q for file exists, if not wait. 2024-04-11 01:42:52,652 INFO 2024-04-11 01:42:52,652 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:52,652 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:52,984 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:52,984 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:52,984 DEBUG --- stdout --- 2024-04-11 01:42:52,984 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:52,984 DEBUG --- stderr --- 2024-04-11 01:42:52,984 DEBUG 2024-04-11 01:42:52,985 INFO 2024-04-11 01:42:52,985 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-78845f4f8c-v2r2q --stdin -- bash -c 'if [ -f "/results/orders/MakeFeeders_ds-idrepo/stdout.txt" ]; then echo "/results/orders/MakeFeeders_ds-idrepo/stdout.txt file found"; exit 0; else echo "/results/orders/MakeFeeders_ds-idrepo/stdout.txt file not found"; exit 1; fi' | grep "/results/orders/MakeFeeders_ds-idrepo/stdout.txt file found" 2024-04-11 01:42:52,985 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 01:42:53,836 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-11 01:42:53,836 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:53,837 DEBUG --- stdout --- 2024-04-11 01:42:53,837 DEBUG /results/orders/MakeFeeders_ds-idrepo/stdout.txt file found 2024-04-11 01:42:53,837 DEBUG --- stderr --- 2024-04-11 01:42:53,837 DEBUG 2024-04-11 01:42:53,837 INFO 2024-04-11 01:42:53,837 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:53,837 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:54,141 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:54,141 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:54,141 DEBUG --- stdout --- 2024-04-11 01:42:54,141 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:54,141 DEBUG --- stderr --- 2024-04-11 01:42:54,141 DEBUG 2024-04-11 01:42:54,141 INFO 2024-04-11 01:42:54,141 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-78845f4f8c-v2r2q --stdin -- du -k /results/orders/MakeFeeders_ds-idrepo/stdout.txt 2024-04-11 01:42:54,958 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-11 01:42:54,958 DEBUG --- stdout --- 2024-04-11 01:42:54,958 DEBUG 4 /results/orders/MakeFeeders_ds-idrepo/stdout.txt 2024-04-11 01:42:54,958 DEBUG --- stderr --- 2024-04-11 01:42:54,958 DEBUG 2024-04-11 01:42:54,958 INFO - Download file (size 4k) 2024-04-11 01:42:54,958 INFO 2024-04-11 01:42:54,959 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:54,959 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:55,262 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:55,263 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:55,263 DEBUG --- stdout --- 2024-04-11 01:42:55,263 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:55,263 DEBUG --- stderr --- 2024-04-11 01:42:55,263 DEBUG source : /results/orders/MakeFeeders_ds-idrepo/stdout.txt @ overseer-0-78845f4f8c-v2r2q destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stdout.txt 2024-04-11 01:42:55,263 INFO 2024-04-11 01:42:55,263 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:55,263 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:55,568 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:55,568 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:55,568 DEBUG --- stdout --- 2024-04-11 01:42:55,568 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:55,568 DEBUG --- stderr --- 2024-04-11 01:42:55,568 DEBUG 2024-04-11 01:42:55,569 INFO 2024-04-11 01:42:55,569 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-78845f4f8c-v2r2q:/results/orders/MakeFeeders_ds-idrepo/stdout.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stdout.txt 2024-04-11 01:42:55,569 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-11 01:42:56,336 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:56,336 DEBUG --- stdout --- 2024-04-11 01:42:56,336 DEBUG tar: Removing leading `/' from member names 2024-04-11 01:42:56,336 DEBUG --- stderr --- 2024-04-11 01:42:56,336 DEBUG ________________________________________________________________________________ [2024-04-11 01:42:56] MakeFeeders_ds-idrepo show_cmd : Display order command ________________________________________________________________________________ 2024-04-11 01:42:56,337 INFO 2024-04-11 01:42:56,337 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:56,337 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:56,650 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:56,651 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:56,651 DEBUG --- stdout --- 2024-04-11 01:42:56,651 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:56,651 DEBUG --- stderr --- 2024-04-11 01:42:56,651 DEBUG Checking pod: overseer-0-78845f4f8c-v2r2q for file exists, if not wait. 2024-04-11 01:42:56,651 INFO 2024-04-11 01:42:56,651 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:56,651 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:56,955 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:56,955 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:56,955 DEBUG --- stdout --- 2024-04-11 01:42:56,955 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:56,955 DEBUG --- stderr --- 2024-04-11 01:42:56,955 DEBUG 2024-04-11 01:42:56,956 INFO 2024-04-11 01:42:56,956 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-78845f4f8c-v2r2q --stdin -- bash -c 'if [ -f "/results/orders/MakeFeeders_ds-idrepo/cmd.txt" ]; then echo "/results/orders/MakeFeeders_ds-idrepo/cmd.txt file found"; exit 0; else echo "/results/orders/MakeFeeders_ds-idrepo/cmd.txt file not found"; exit 1; fi' | grep "/results/orders/MakeFeeders_ds-idrepo/cmd.txt file found" 2024-04-11 01:42:56,956 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 01:42:57,834 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-11 01:42:57,836 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:57,836 DEBUG --- stdout --- 2024-04-11 01:42:57,836 DEBUG /results/orders/MakeFeeders_ds-idrepo/cmd.txt file found 2024-04-11 01:42:57,836 DEBUG --- stderr --- 2024-04-11 01:42:57,836 DEBUG 2024-04-11 01:42:57,837 INFO 2024-04-11 01:42:57,837 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:57,837 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:58,166 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:58,167 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:58,167 DEBUG --- stdout --- 2024-04-11 01:42:58,167 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:58,167 DEBUG --- stderr --- 2024-04-11 01:42:58,167 DEBUG 2024-04-11 01:42:58,167 INFO 2024-04-11 01:42:58,167 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec overseer-0-78845f4f8c-v2r2q --stdin -- du -k /results/orders/MakeFeeders_ds-idrepo/cmd.txt 2024-04-11 01:42:59,078 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-11 01:42:59,078 DEBUG --- stdout --- 2024-04-11 01:42:59,078 DEBUG 4 /results/orders/MakeFeeders_ds-idrepo/cmd.txt 2024-04-11 01:42:59,078 DEBUG --- stderr --- 2024-04-11 01:42:59,078 DEBUG 2024-04-11 01:42:59,078 INFO - Download file (size 4k) 2024-04-11 01:42:59,079 INFO 2024-04-11 01:42:59,079 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:59,079 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:59,406 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:59,407 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:59,407 DEBUG --- stdout --- 2024-04-11 01:42:59,407 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:59,407 DEBUG --- stderr --- 2024-04-11 01:42:59,407 DEBUG source : /results/orders/MakeFeeders_ds-idrepo/cmd.txt @ overseer-0-78845f4f8c-v2r2q destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/cmd.txt 2024-04-11 01:42:59,408 INFO 2024-04-11 01:42:59,408 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 2024-04-11 01:42:59,408 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 01:42:59,731 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-11 01:42:59,731 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:42:59,731 DEBUG --- stdout --- 2024-04-11 01:42:59,731 DEBUG overseer-0-78845f4f8c-v2r2q 2024-04-11 01:42:59,731 DEBUG --- stderr --- 2024-04-11 01:42:59,731 DEBUG 2024-04-11 01:42:59,732 INFO 2024-04-11 01:42:59,732 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp --retries=100 overseer-0-78845f4f8c-v2r2q:/results/orders/MakeFeeders_ds-idrepo/cmd.txt /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/cmd.txt 2024-04-11 01:42:59,732 INFO [loop_until]: (max_time=-1, interval=5, expected_rc=[0] 2024-04-11 01:43:00,484 INFO [loop_until]: OK (rc = 0) 2024-04-11 01:43:00,484 DEBUG --- stdout --- 2024-04-11 01:43:00,484 DEBUG tar: Removing leading `/' from member names 2024-04-11 01:43:00,484 DEBUG --- stderr --- 2024-04-11 01:43:00,484 DEBUG 2024-04-11 01:43:00,484 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/cmd.txt is small (1 lines), printing all lines: 2024-04-11 01:43:00,484 DEBUG ----- output ----- 2024-04-11 01:43:00,484 DEBUG python3 /lodestar/pyrock/shared/scripts/ds/generate_feeders.py --ldif-file /results/ds-idrepo/user_entries.ldif --out-dir /results/ds-idrepo --zip-file /results/ds-idrepo/feeders.tar.gz 2024-04-11 01:43:00,484 INFO ________________________________________________________________________________ [2024-04-11 01:43:00] MakeFeeders_ds-idrepo show_rc : Display rc order ________________________________________________________________________________ 2024-04-11 01:43:00,484 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/rc.txt is small (1 lines), printing all lines: 2024-04-11 01:43:00,485 DEBUG ----- output ----- 2024-04-11 01:43:00,485 DEBUG CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) 2024-04-11 01:43:00,485 INFO ________________________________________________________________________________ [2024-04-11 01:43:00] MakeFeeders_ds-idrepo show_stdout : Display stdout order ________________________________________________________________________________ 2024-04-11 01:43:00,485 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stdout.txt is small (12 lines), printing all lines: 2024-04-11 01:43:00,485 DEBUG ----- output ----- 2024-04-11 01:43:00,485 DEBUG ----------------------------- Generating feeders, please wait..... ----------------------------- Files generated in 21s - dn file : /results/ds-idrepo/feeders/ds-idrepo.dn - login file : /results/ds-idrepo/feeders/ds-idrepo.login - bind file : /results/ds-idrepo/feeders/ds-idrepo.bind --------------------- Zipping /results/ds-idrepo/feeders, please wait..... --------------------- Zip generated in 8s - zip file : /results/ds-idrepo/feeders.tar.gz Finished without any error in 30s. 2024-04-11 01:43:00,485 INFO ________________________________________________________________________________ [2024-04-11 01:43:00] MakeFeeders_ds-idrepo show_stderr : Display stderr order ________________________________________________________________________________ 2024-04-11 01:43:00,485 DEBUG [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/authn_rest/05_Data_Init/02_MakeFeeders_ds-idrepo/stderr.txt is small (0 lines), printing all lines: 2024-04-11 01:43:00,485 DEBUG ----- output ----- 2024-04-11 01:43:00,485 DEBUG 2024-04-11 01:43:00,485 INFO ________________________________________________________________________________ [2024-04-11 01:43:00] MakeFeeders_ds-idrepo check_rc_and_logs : Check order's outputs ________________________________________________________________________________ PASS: no error found in rc log PASS: no error found in stdout log PASS: no error found in stderr log ________________________________________________________________________________ [2024-04-11 01:43:00] MakeFeeders_ds-idrepo print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Task result is INIT no need to display logs. ________________________________________________________________________________ [2024-04-11 01:43:00] MakeFeeders_ds-idrepo post : Check output ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped