--Task-- name: Verification-ds-idrepo enabled: True class_name: VerificationTask source_name: ds-idrepo source_namespace: >default< target_name: ds-idrepo target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: [] wait_for: [] preceding_task: None options: {} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [2024-04-24 09:54:50] Verification-ds-idrepo pre : N/A ________________________________________________________________________________ [2024-04-24 09:54:50] - INFO: Timeout for this Task has changed to 8h (28800 seconds) [2024-04-24 09:54:50] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset ________________________________________________________________________________ [2024-04-24 09:54:50] Verification-ds-idrepo step1 : Wait until pods are in Running state and check number of running pods ________________________________________________________________________________ 2024-04-24 09:54:50,094 INFO 2024-04-24 09:54:50,094 INFO --------------------- Get expected number of pods --------------------- 2024-04-24 09:54:50,099 INFO 2024-04-24 09:54:50,101 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas} 2024-04-24 09:54:50,105 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 09:54:51,487 INFO [loop_until]: OK (rc = 0) 2024-04-24 09:54:51,487 DEBUG --- stdout --- 2024-04-24 09:54:51,487 DEBUG 3 2024-04-24 09:54:51,487 DEBUG --- stderr --- 2024-04-24 09:54:51,487 DEBUG 2024-04-24 09:54:51,488 INFO 2024-04-24 09:54:51,488 INFO -------------- Waiting for 3 expected pod(s) to be ready -------------- 2024-04-24 09:54:51,488 INFO 2024-04-24 09:54:51,488 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets ds-idrepo --output jsonpath="current:{.status.currentReplicas} ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "current:3 ready:3 replicas:3" 2024-04-24 09:54:51,488 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0] 2024-04-24 09:54:52,559 INFO [loop_until]: Function succeeded after 1s (rc=0) - expected pattern found 2024-04-24 09:54:52,559 INFO [loop_until]: OK (rc = 0) 2024-04-24 09:54:52,559 DEBUG --- stdout --- 2024-04-24 09:54:52,559 DEBUG current:3 ready:3 replicas:3 2024-04-24 09:54:52,559 DEBUG --- stderr --- 2024-04-24 09:54:52,559 DEBUG ________________________________________________________________________________ [2024-04-24 09:54:52] Verification-ds-idrepo step2 : Discovering pods ________________________________________________________________________________ 2024-04-24 09:54:52,560 INFO 2024-04-24 09:54:52,560 INFO --------------------- Get expected number of pods --------------------- 2024-04-24 09:54:52,561 INFO 2024-04-24 09:54:52,561 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get statefulsets --selector app=ds-idrepo --output jsonpath={.items[*].spec.replicas} 2024-04-24 09:54:52,561 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 09:54:53,313 INFO [loop_until]: OK (rc = 0) 2024-04-24 09:54:53,314 DEBUG --- stdout --- 2024-04-24 09:54:53,314 DEBUG 3 2024-04-24 09:54:53,314 DEBUG --- stderr --- 2024-04-24 09:54:53,314 DEBUG 2024-04-24 09:54:53,315 INFO 2024-04-24 09:54:53,315 INFO ---------------------------- Get pod list ---------------------------- 2024-04-24 09:54:53,315 INFO 2024-04-24 09:54:53,315 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=ds-idrepo --output jsonpath={.items[*].metadata.name}` | grep 3 2024-04-24 09:54:53,315 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-24 09:54:54,590 INFO [loop_until]: Function succeeded after 1s (rc=0) - expected number of elements found 2024-04-24 09:54:54,590 INFO [loop_until]: OK (rc = 0) 2024-04-24 09:54:54,590 DEBUG --- stdout --- 2024-04-24 09:54:54,590 DEBUG ds-idrepo-0 ds-idrepo-1 ds-idrepo-2 2024-04-24 09:54:54,590 DEBUG --- stderr --- 2024-04-24 09:54:54,590 DEBUG ________________________________________________________________________________ [2024-04-24 09:54:54] Verification-ds-idrepo step3 : Check component is alive ________________________________________________________________________________ 2024-04-24 09:54:54,591 INFO Livecheck to ds-idrepo-0 2024-04-24 09:54:54,591 INFO 2024-04-24 09:54:54,595 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive 2024-04-24 09:54:56,609 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-24 09:54:56,609 DEBUG --- stdout --- 2024-04-24 09:54:56,609 DEBUG dn: alive: true 2024-04-24 09:54:56,609 DEBUG --- stderr --- 2024-04-24 09:54:56,609 DEBUG 2024-04-24 09:54:56,609 INFO 2024-04-24 09:54:56,610 INFO Livecheck to ds-idrepo-1 2024-04-24 09:54:56,610 INFO 2024-04-24 09:54:56,611 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-1 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive 2024-04-24 09:54:58,467 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-24 09:54:58,468 DEBUG --- stdout --- 2024-04-24 09:54:58,468 DEBUG dn: alive: true 2024-04-24 09:54:58,468 DEBUG --- stderr --- 2024-04-24 09:54:58,468 DEBUG 2024-04-24 09:54:58,468 INFO 2024-04-24 09:54:58,469 INFO Livecheck to ds-idrepo-2 2024-04-24 09:54:58,469 INFO 2024-04-24 09:54:58,470 INFO [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-2 --container ds -- ldapsearch --noPropertiesFile --port 1389 --useStartTls --trustAll --bindDn "uid=admin" --bindPassword **** --baseDn "" --searchScope base "(&)" alive 2024-04-24 09:55:00,503 INFO [run_command]: OK (rc = 0 - expected to be in [0]) 2024-04-24 09:55:00,505 DEBUG --- stdout --- 2024-04-24 09:55:00,505 DEBUG dn: alive: true 2024-04-24 09:55:00,505 DEBUG --- stderr --- 2024-04-24 09:55:00,505 DEBUG 2024-04-24 09:55:00,505 INFO PASS : component ds-idrepo is alive ________________________________________________________________________________ [2024-04-24 09:55:00] Verification-ds-idrepo step4 : Record version ________________________________________________________________________________ 2024-04-24 09:55:00,506 INFO 2024-04-24 09:55:00,506 INFO ------------------- Get DS-IDREPO software version ------------------- 2024-04-24 09:55:00,507 INFO 2024-04-24 09:55:00,507 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec ds-idrepo-0 --container ds -- find /opt/opendj -name opendj-core.jar 2024-04-24 09:55:00,507 INFO [loop_until]: (max_time=30, interval=5, expected_rc=[0] 2024-04-24 09:55:01,502 INFO [loop_until]: OK (rc = 0) 2024-04-24 09:55:01,502 DEBUG --- stdout --- 2024-04-24 09:55:01,502 DEBUG /opt/opendj/lib/opendj-core.jar 2024-04-24 09:55:01,503 DEBUG --- stderr --- 2024-04-24 09:55:01,503 DEBUG 2024-04-24 09:55:01,504 INFO 2024-04-24 09:55:01,504 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm cp ds-idrepo-0:/opt/opendj/lib/opendj-core.jar /tmp/ds-idrepo_info/opendj-core.jar --container ds 2024-04-24 09:55:01,504 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 09:55:03,122 INFO [loop_until]: OK (rc = 0) 2024-04-24 09:55:03,122 DEBUG --- stdout --- 2024-04-24 09:55:03,122 DEBUG tar: Removing leading `/' from member names 2024-04-24 09:55:03,122 DEBUG --- stderr --- 2024-04-24 09:55:03,123 DEBUG Version is 7.6.0.20240418130921-7e0436f7626375c00baee240ad259d6a2 Revision is 7e0436f7626375c00baee240ad259d6a2b07d5f4 ________________________________________________________________________________ [2024-04-24 09:55:03] Verification-ds-idrepo step5 : Checking pod restart counts ________________________________________________________________________________ 2024-04-24 09:55:03,133 INFO 2024-04-24 09:55:03,133 INFO ----------------- Check pod ds-idrepo-0 restart count ----------------- 2024-04-24 09:55:03,133 INFO 2024-04-24 09:55:03,133 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-0 --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-24 09:55:03,133 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 09:55:03,530 INFO [loop_until]: OK (rc = 0) 2024-04-24 09:55:03,531 DEBUG --- stdout --- 2024-04-24 09:55:03,531 DEBUG 0 2024-04-24 09:55:03,531 DEBUG --- stderr --- 2024-04-24 09:55:03,531 DEBUG 2024-04-24 09:55:03,531 INFO Pod ds-idrepo-0 has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases 2024-04-24 09:55:03,531 INFO 2024-04-24 09:55:03,531 INFO ----------------- Check pod ds-idrepo-1 restart count ----------------- 2024-04-24 09:55:03,531 INFO 2024-04-24 09:55:03,531 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-1 --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-24 09:55:03,532 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 09:55:03,919 INFO [loop_until]: OK (rc = 0) 2024-04-24 09:55:03,920 DEBUG --- stdout --- 2024-04-24 09:55:03,920 DEBUG 0 2024-04-24 09:55:03,921 DEBUG --- stderr --- 2024-04-24 09:55:03,921 DEBUG 2024-04-24 09:55:03,921 INFO Pod ds-idrepo-1 has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases 2024-04-24 09:55:03,921 INFO 2024-04-24 09:55:03,921 INFO ----------------- Check pod ds-idrepo-2 restart count ----------------- 2024-04-24 09:55:03,923 INFO 2024-04-24 09:55:03,924 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod ds-idrepo-2 --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-24 09:55:03,924 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-24 09:55:04,316 INFO [loop_until]: OK (rc = 0) 2024-04-24 09:55:04,316 DEBUG --- stdout --- 2024-04-24 09:55:04,316 DEBUG 0 2024-04-24 09:55:04,316 DEBUG --- stderr --- 2024-04-24 09:55:04,316 DEBUG 2024-04-24 09:55:04,316 INFO Pod ds-idrepo-2 has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases ________________________________________________________________________________ [2024-04-24 09:55:04] Verification-ds-idrepo post : Post method ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped