--Task-- name: Verification-ds-cts enabled: True class_name: VerificationTask source_name: ds-cts source_namespace: >default< target_name: ds-cts target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: [] wait_for: [] preceding_task: None options: {} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [2024-04-19 11:48:57] Verification-ds-cts pre : N/A ________________________________________________________________________________ [2024-04-19 11:48:57] - INFO: Timeout for this Task has changed to 14h (50400 seconds) [2024-04-19 11:48:57] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset ________________________________________________________________________________ [2024-04-19 11:48:57] Verification-ds-cts step1 : Wait until pods are in Running state and check number of running pods ________________________________________________________________________________ 2024-04-19 11:48:57,179 INFO 2024-04-19 11:48:57,179 INFO --------------------- Get expected number of pods --------------------- 2024-04-19 11:48:57,180 INFO 2024-04-19 11:48:57,180 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas} 2024-04-19 11:48:57,180 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-19 11:48:58,111 INFO [loop_until]: OK (rc = 0) 2024-04-19 11:48:58,111 DEBUG --- stdout --- 2024-04-19 11:48:58,111 DEBUG 3 2024-04-19 11:48:58,111 DEBUG --- stderr --- 2024-04-19 11:48:58,111 DEBUG 2024-04-19 11:48:58,111 INFO 2024-04-19 11:48:58,111 INFO -------------- Waiting for 3 expected pod(s) to be ready -------------- 2024-04-19 11:48:58,112 INFO 2024-04-19 11:48:58,112 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-cts --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3" 2024-04-19 11:48:58,112 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0] 2024-04-19 11:48:58,783 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-19 11:48:58,783 INFO [loop_until]: OK (rc = 0) 2024-04-19 11:48:58,783 DEBUG --- stdout --- 2024-04-19 11:48:58,784 DEBUG current:3 ready:3 replicas:3 2024-04-19 11:48:58,784 DEBUG --- stderr --- 2024-04-19 11:48:58,784 DEBUG ________________________________________________________________________________ [2024-04-19 11:48:58] Verification-ds-cts step2 : Discovering pods ________________________________________________________________________________ 2024-04-19 11:48:58,784 INFO 2024-04-19 11:48:58,784 INFO --------------------- Get expected number of pods --------------------- 2024-04-19 11:48:58,785 INFO 2024-04-19 11:48:58,785 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-cts --output jsonpath={.items[*].spec.replicas} 2024-04-19 11:48:58,785 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-19 11:48:59,194 INFO [loop_until]: OK (rc = 0) 2024-04-19 11:48:59,195 DEBUG --- stdout --- 2024-04-19 11:48:59,195 DEBUG 3 2024-04-19 11:48:59,195 DEBUG --- stderr --- 2024-04-19 11:48:59,195 DEBUG 2024-04-19 11:48:59,195 INFO 2024-04-19 11:48:59,195 INFO ---------------------------- Get pod list ---------------------------- 2024-04-19 11:48:59,196 INFO 2024-04-19 11:48:59,196 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-cts --output jsonpath={.items[*].metadata.name}` | grep 3 2024-04-19 11:48:59,196 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-19 11:49:00,139 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found 2024-04-19 11:49:00,140 INFO [loop_until]: OK (rc = 0) 2024-04-19 11:49:00,140 DEBUG --- stdout --- 2024-04-19 11:49:00,140 DEBUG ds-cts-0 ds-cts-1 ds-cts-2 2024-04-19 11:49:00,140 DEBUG --- stderr --- 2024-04-19 11:49:00,140 DEBUG ________________________________________________________________________________ [2024-04-19 11:49:00] Verification-ds-cts step3 : Check component is alive ________________________________________________________________________________ 2024-04-19 11:49:00,141 INFO Livecheck to ds-cts-0 2024-04-19 11:49:00,142 INFO 2024-04-19 11:49:00,142 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive 2024-04-19 11:49:02,910 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-19 11:49:02,930 DEBUG --- stdout --- 2024-04-19 11:49:02,931 DEBUG dn: alive: true 2024-04-19 11:49:02,931 DEBUG --- stderr --- 2024-04-19 11:49:02,931 DEBUG 2024-04-19 11:49:02,932 INFO 2024-04-19 11:49:02,933 INFO Livecheck to ds-cts-1 2024-04-19 11:49:02,933 INFO 2024-04-19 11:49:02,934 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive 2024-04-19 11:49:05,570 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-19 11:49:05,571 DEBUG --- stdout --- 2024-04-19 11:49:05,571 DEBUG dn: alive: true 2024-04-19 11:49:05,571 DEBUG --- stderr --- 2024-04-19 11:49:05,571 DEBUG 2024-04-19 11:49:05,571 INFO 2024-04-19 11:49:05,571 INFO Livecheck to ds-cts-2 2024-04-19 11:49:05,571 INFO 2024-04-19 11:49:05,572 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive 2024-04-19 11:49:07,799 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-19 11:49:07,799 DEBUG --- stdout --- 2024-04-19 11:49:07,799 DEBUG dn: alive: true 2024-04-19 11:49:07,799 DEBUG --- stderr --- 2024-04-19 11:49:07,799 DEBUG 2024-04-19 11:49:07,799 INFO PASS : component ds-cts is alive ________________________________________________________________________________ [2024-04-19 11:49:07] Verification-ds-cts step4 : Record version ________________________________________________________________________________ 2024-04-19 11:49:07,800 INFO 2024-04-19 11:49:07,800 INFO --------------------- Get DS-CTS software version --------------------- 2024-04-19 11:49:07,801 INFO 2024-04-19 11:49:07,801 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-cts-0 --container ds -- find /opt/opendj -name opendj-core.jar 2024-04-19 11:49:07,801 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0] 2024-04-19 11:49:08,851 INFO [loop_until]: OK (rc = 0) 2024-04-19 11:49:08,851 DEBUG --- stdout --- 2024-04-19 11:49:08,852 DEBUG /opt/opendj/lib/opendj-core.jar 2024-04-19 11:49:08,852 DEBUG --- stderr --- 2024-04-19 11:49:08,852 DEBUG 2024-04-19 11:49:08,852 INFO 2024-04-19 11:49:08,852 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-cts-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-cts_info/opendj-core.jar --container ds 2024-04-19 11:49:08,852 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-19 11:49:10,084 INFO [loop_until]: OK (rc = 0) 2024-04-19 11:49:10,084 DEBUG --- stdout --- 2024-04-19 11:49:10,084 DEBUG tar: Removing leading `/' from member names 2024-04-19 11:49:10,084 DEBUG --- stderr --- 2024-04-19 11:49:10,084 DEBUG Version is 7.6.0.20240412143511-db97906b8eff882e42d79d5b86aca4356 Revision is db97906b8eff882e42d79d5b86aca43562470e8d ________________________________________________________________________________ [2024-04-19 11:49:10] Verification-ds-cts step5 : Checking pod restart counts ________________________________________________________________________________ 2024-04-19 11:49:10,093 INFO 2024-04-19 11:49:10,093 INFO ------------------ Check pod ds-cts-0 restart count ------------------ 2024-04-19 11:49:10,093 INFO 2024-04-19 11:49:10,093 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-0 --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-19 11:49:10,093 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-19 11:49:10,470 INFO [loop_until]: OK (rc = 0) 2024-04-19 11:49:10,470 DEBUG --- stdout --- 2024-04-19 11:49:10,471 DEBUG 0 2024-04-19 11:49:10,471 DEBUG --- stderr --- 2024-04-19 11:49:10,471 DEBUG 2024-04-19 11:49:10,471 INFO Pod ds-cts-0 has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases 2024-04-19 11:49:10,471 INFO 2024-04-19 11:49:10,471 INFO ------------------ Check pod ds-cts-1 restart count ------------------ 2024-04-19 11:49:10,471 INFO 2024-04-19 11:49:10,471 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-1 --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-19 11:49:10,471 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-19 11:49:10,849 INFO [loop_until]: OK (rc = 0) 2024-04-19 11:49:10,850 DEBUG --- stdout --- 2024-04-19 11:49:10,850 DEBUG 0 2024-04-19 11:49:10,850 DEBUG --- stderr --- 2024-04-19 11:49:10,850 DEBUG 2024-04-19 11:49:10,850 INFO Pod ds-cts-1 has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases 2024-04-19 11:49:10,850 INFO 2024-04-19 11:49:10,850 INFO ------------------ Check pod ds-cts-2 restart count ------------------ 2024-04-19 11:49:10,850 INFO 2024-04-19 11:49:10,851 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-cts-2 --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-19 11:49:10,851 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-19 11:49:11,240 INFO [loop_until]: OK (rc = 0) 2024-04-19 11:49:11,240 DEBUG --- stdout --- 2024-04-19 11:49:11,240 DEBUG 0 2024-04-19 11:49:11,240 DEBUG --- stderr --- 2024-04-19 11:49:11,240 DEBUG 2024-04-19 11:49:11,240 INFO Pod ds-cts-2 has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases ________________________________________________________________________________ [2024-04-19 11:49:11] Verification-ds-cts post : Post method ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped