--Task-- name: dump-uid enabled: True class_name: DumpIDMIDTaskLDAPSearch source_name: overseer-0 source_namespace: >default< target_name: ds-idrepo-0 target_namespace: >default< start: 0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: ['preload-with-gatling'] wait_for: [] options: {'_options_': '--ldapsearch_output /results/idm/user_info.ldif --user_output /results/idm/user_feeder.csv'} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [28/May/2023 06:22:04] dump-uid pre : N/A ________________________________________________________________________________ [28/May/2023 06:22:04] - INFO: Timeout for this Task has changed to 4h 40m (16800 seconds) [28/May/2023 06:22:04] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset Pod log file : /results/orders/dump-uid/stdout.txt Pod log : /results/orders/dump-uid ________________________________________________________________________________ [28/May/2023 06:22:04] dump-uid print_command : Display the command to be forwarded to overseer as an order ________________________________________________________________________________ /lodestar/pyrock/shared/scripts/idm/dump_idm_id_ldapsearch.py --ldapsearch_path /lodestar/clients/ds/bin/ldapsearch --ds_host ds-idrepo-0.ds-idrepo --ds_port 1389 --ds_admin_user uid=admin --ds_admin_pass wEcQVfDwdrG8TaZMLfSX2nfQQCYInCpr --ldapsearch_output /results/idm/user_info.ldif --user_output /results/idm/user_feeder.csv ________________________________________________________________________________ [28/May/2023 06:22:04] dump-uid create_order : Create order for overseer with the command to run ________________________________________________________________________________ Order file : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid/order.json { "name": "dump-uid", "command": "/lodestar/pyrock/shared/scripts/idm/dump_idm_id_ldapsearch.py --ldapsearch_path /lodestar/clients/ds/bin/ldapsearch --ds_host ds-idrepo-0.ds-idrepo --ds_port 1389 --ds_admin_user uid=admin --ds_admin_pass wEcQVfDwdrG8TaZMLfSX2nfQQCYInCpr --ldapsearch_output /results/idm/user_info.ldif --user_output /results/idm/user_feeder.csv", "out-dir": "/results/orders/dump-uid" } ________________________________________________________________________________ [28/May/2023 06:22:04] dump-uid loop_until_ready : Loop until overseer pod is ready to accept order ________________________________________________________________________________ [loop_until]: curl --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt | grep "WAITING" [loop_until]: (max_time=120, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- WAITING --- stderr --- % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 7 100 7 0 0 33 0 --:--:-- --:--:-- --:--:-- 33 ________________________________________________________________________________ [28/May/2023 06:22:04] dump-uid start_order : Send the order to the overseer pod ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-64c8c4b58-d9cvt --- stderr --- [loop_until]: kubectl --namespace=xlou cp /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid/order.json overseer-0-64c8c4b58-d9cvt:/results/orders/order.json [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- Order has been successfully forwarded to overseer ________________________________________________________________________________ [28/May/2023 06:22:06] dump-uid wait_stop_by_itself : Wait for the order to stop by itself ________________________________________________________________________________ Task may stop by itself (no stop defined in configuration file), wait until overseer ends the order - timeout is 4h 40m Waiting for status (curl --fail --insecure https://overseer-0.xlou.iam.xlou-cdm.engineeringpit.com/orders/status.txt) == WAITING PASS : Order stopped by itself after 2m and before timeout (4h 40m) ________________________________________________________________________________ [28/May/2023 06:24:06] dump-uid stop_order : Stop the order on overseer pod ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-64c8c4b58-d9cvt --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-64c8c4b58-d9cvt -- rm -f /results/orders/order.json [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- Order ran during 2m 01s ________________________________________________________________________________ [28/May/2023 06:24:07] dump-uid download_logs : Download order.tar.gz ________________________________________________________________________________ [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-64c8c4b58-d9cvt --- stderr --- Checking pod: overseer-0-64c8c4b58-d9cvt for file exists, if not wait. [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-64c8c4b58-d9cvt --- stderr --- [loop_until]: kubectl --namespace=xlou exec overseer-0-64c8c4b58-d9cvt --stdin -- bash -c 'if [ -f "/results/orders/dump-uid/order.tar.gz" ]; then echo "/results/orders/dump-uid/order.tar.gz file found"; exit 0; else echo "/results/orders/dump-uid/order.tar.gz file not found"; exit 1; fi' | grep "/results/orders/dump-uid/order.tar.gz file found" [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found [loop_until]: OK (rc = 0) --- stdout --- /results/orders/dump-uid/order.tar.gz file found --- stderr --- [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-64c8c4b58-d9cvt --- stderr --- [run_command]: kubectl --namespace=xlou exec overseer-0-64c8c4b58-d9cvt --stdin -- du -k /results/orders/dump-uid/order.tar.gz [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- 4 /results/orders/dump-uid/order.tar.gz --- stderr --- - Download file (size 4k) [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-64c8c4b58-d9cvt --- stderr --- source : /results/orders/dump-uid/order.tar.gz @ overseer-0-64c8c4b58-d9cvt destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid/order.tar.gz [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou get pods --selector app=overseer-0 --output jsonpath={.items[*].metadata.name}` | grep 1 [loop_until]: (max_time=180, interval=10, expected_rc=[0] [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found [loop_until]: OK (rc = 0) --- stdout --- overseer-0-64c8c4b58-d9cvt --- stderr --- [loop_until]: kubectl --namespace=xlou cp overseer-0-64c8c4b58-d9cvt:/results/orders/dump-uid/order.tar.gz /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid/order.tar.gz [loop_until]: (max_time=-1, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- tar: Removing leading `/' from member names --- stderr --- - Extract archive zip file : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid/order.tar.gz destination : /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid ________________________________________________________________________________ [28/May/2023 06:24:11] dump-uid show_cmd : Display order command ________________________________________________________________________________ [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid/cmd.txt is small (1 lines), printing all lines: ----- output ----- /lodestar/pyrock/shared/scripts/idm/dump_idm_id_ldapsearch.py --ldapsearch_path /lodestar/clients/ds/bin/ldapsearch --ds_host ds-idrepo-0.ds-idrepo --ds_port 1389 --ds_admin_user uid=admin --ds_admin_pass wEcQVfDwdrG8TaZMLfSX2nfQQCYInCpr --ldapsearch_output /results/idm/user_info.ldif --user_output /results/idm/user_feeder.csv ________________________________________________________________________________ [28/May/2023 06:24:11] dump-uid show_rc : Display rc order ________________________________________________________________________________ [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid/rc.txt is small (1 lines), printing all lines: ----- output ----- CMD SUCCESS: command returns code 0 (expected [0, -9, 143, 137, -15]) ________________________________________________________________________________ [28/May/2023 06:24:11] dump-uid show_stdout : Display stdout order ________________________________________________________________________________ [print_head_tail]: Print head (100 lines) and tail (100 lines) of input file (/mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid/stdout.txt is 334 lines): ----- output ----- /lodestar/clients/ds/bin/ldapsearch --noPropertiesFile --hostname ds-idrepo-0.ds-idrepo --port 1389 --bindDn "uid=admin" --bindPassword "wEcQVfDwdrG8TaZMLfSX2nfQQCYInCpr" --baseDn "ou=people,ou=identities" "&" fr-idm-uuid uid sn givenName mail fr-idm-effectiveRole [run_command]: /lodestar/clients/ds/bin/ldapsearch --noPropertiesFile --hostname ds-idrepo-0.ds-idrepo --port 1389 --bindDn "uid=admin" --bindPassword "wEcQVfDwdrG8TaZMLfSX2nfQQCYInCpr" --baseDn "ou=people,ou=identities" "&" fr-idm-uuid uid sn givenName mail fr-idm-effectiveRole Redirecting output to /results/idm/user_info.ldif [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- [print_head_tail]: Print head (80 lines) and tail (80 lines) of input file (/results/idm/user_info.ldif is 7000002 lines): ----- output ----- dn: ou=people,ou=identities dn: fr-idm-uuid=000001c7-dc0c-4192-b8ad-54d38d7c80a4,ou=people,ou=identities fr-idm-uuid: 000001c7-dc0c-4192-b8ad-54d38d7c80a4 givenName: Wang mail: user.416314@example.com sn: Xqvviv uid: user.416314 dn: fr-idm-uuid=0000132c-bc52-4224-987d-bc98dd784058,ou=people,ou=identities fr-idm-uuid: 0000132c-bc52-4224-987d-bc98dd784058 givenName: Martin mail: user.304761@example.com sn: Yjmcvm uid: user.304761 dn: fr-idm-uuid=00002446-c3ed-4cfc-a92f-d82891995656,ou=people,ou=identities fr-idm-uuid: 00002446-c3ed-4cfc-a92f-d82891995656 givenName: Smith mail: user.184338@example.com sn: Lceqkd uid: user.184338 dn: fr-idm-uuid=00002fb7-3323-435c-8c37-b510e4a8eed5,ou=people,ou=identities fr-idm-uuid: 00002fb7-3323-435c-8c37-b510e4a8eed5 givenName: Wang mail: user.735829@example.com sn: Rfoytr uid: user.735829 dn: fr-idm-uuid=00005843-1f81-4171-8050-95d6fb28612f,ou=people,ou=identities fr-idm-uuid: 00005843-1f81-4171-8050-95d6fb28612f givenName: Accola mail: user.677771@example.com sn: Ybeltt uid: user.677771 dn: fr-idm-uuid=0000598b-22fc-4672-8f74-68e4ff2fd6c8,ou=people,ou=identities fr-idm-uuid: 0000598b-22fc-4672-8f74-68e4ff2fd6c8 givenName: Aberta mail: user.539221@example.com sn: Muukgq uid: user.539221 dn: fr-idm-uuid=0000753b-8e50-4d30-947c-3d78e0c85b97,ou=people,ou=identities fr-idm-uuid: 0000753b-8e50-4d30-947c-3d78e0c85b97 [...] givenName: Martin mail: user.35646@example.com sn: Gzwiiz uid: user.35646 dn: fr-idm-uuid=ffffab4e-505d-485d-b8b2-a10afbe84df3,ou=people,ou=identities fr-idm-uuid: ffffab4e-505d-485d-b8b2-a10afbe84df3 givenName: Nkosi mail: user.968053@example.com sn: Uqwduu uid: user.968053 dn: fr-idm-uuid=ffffb638-3c62-451b-8a08-ad310c9ca403,ou=people,ou=identities fr-idm-uuid: ffffb638-3c62-451b-8a08-ad310c9ca403 givenName: Chen mail: user.576862@example.com sn: Peecbd uid: user.576862 dn: fr-idm-uuid=ffffcbdd-dc8a-4489-a8b5-c4e1c441d52b,ou=people,ou=identities fr-idm-uuid: ffffcbdd-dc8a-4489-a8b5-c4e1c441d52b givenName: Wang mail: user.235357@example.com sn: Qyqgnp uid: user.235357 dn: fr-idm-uuid=ffffd35b-de86-4ace-88bb-618c35150ed4,ou=people,ou=identities fr-idm-uuid: ffffd35b-de86-4ace-88bb-618c35150ed4 givenName: Sato mail: user.757065@example.com sn: Esbrtz uid: user.757065 dn: fr-idm-uuid=ffffdb1c-48fa-4a84-8d25-520f55cc4467,ou=people,ou=identities fr-idm-uuid: ffffdb1c-48fa-4a84-8d25-520f55cc4467 givenName: Hernandez mail: user.522859@example.com sn: Megjzj uid: user.522859 dn: fr-idm-uuid=ffffe623-ae73-4dc0-9c6f-5c6b1e3829d0,ou=people,ou=identities fr-idm-uuid: ffffe623-ae73-4dc0-9c6f-5c6b1e3829d0 givenName: Kim mail: user.522641@example.com sn: Pkqpsx uid: user.522641 --- stderr --- got key error with dn: ou=people,ou=identities ________________________________________________________________________________ [28/May/2023 06:24:11] dump-uid show_stderr : Display stderr order ________________________________________________________________________________ [print_head_tail]: Input file /mnt/disks/data/xslou/lodestar-fork/results/pyrock/simple_managed_users/05_Data_init/03_dump-uid/stderr.txt is small (0 lines), printing all lines: ----- output ----- ________________________________________________________________________________ [28/May/2023 06:24:11] dump-uid check_rc_and_logs : Check order's outputs ________________________________________________________________________________ PASS: no error found in rc log PASS: no error found in stdout log PASS: no error found in stderr log ________________________________________________________________________________ [28/May/2023 06:24:11] dump-uid print_overseer_logs : Print overseer logs from the K8s pod ________________________________________________________________________________ Task result is INIT no need to display logs. ________________________________________________________________________________ [28/May/2023 06:24:11] dump-uid post : Check output ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped