--Task-- name: Verification-am enabled: True class_name: VerificationTask source_name: am source_namespace: >default< target_name: am target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: [] wait_for: [] preceding_task: None options: {} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [2024-04-11 06:20:23] Verification-am pre : N/A ________________________________________________________________________________ [2024-04-11 06:20:23] - INFO: Timeout for this Task has changed to 10h (36000 seconds) [2024-04-11 06:20:23] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset ________________________________________________________________________________ [2024-04-11 06:20:23] Verification-am step1 : Wait until pods are in Running state and check number of running pods ________________________________________________________________________________ 2024-04-11 06:20:23,308 INFO 2024-04-11 06:20:23,308 INFO --------------------- Get expected number of pods --------------------- 2024-04-11 06:20:23,309 INFO 2024-04-11 06:20:23,309 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas} 2024-04-11 06:20:23,331 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 06:20:24,614 INFO [loop_until]: OK (rc = 0) 2024-04-11 06:20:24,695 DEBUG --- stdout --- 2024-04-11 06:20:24,696 DEBUG 3 2024-04-11 06:20:24,696 DEBUG --- stderr --- 2024-04-11 06:20:24,696 DEBUG 2024-04-11 06:20:24,696 INFO 2024-04-11 06:20:24,696 INFO -------------- Waiting for 3 expected pod(s) to be ready -------------- 2024-04-11 06:20:24,697 INFO 2024-04-11 06:20:24,697 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments am --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:3 replicas:3" 2024-04-11 06:20:24,697 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0] 2024-04-11 06:20:25,635 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-11 06:20:25,636 INFO [loop_until]: OK (rc = 0) 2024-04-11 06:20:25,636 DEBUG --- stdout --- 2024-04-11 06:20:25,636 DEBUG ready:3 replicas:3 2024-04-11 06:20:25,636 DEBUG --- stderr --- 2024-04-11 06:20:25,636 DEBUG ________________________________________________________________________________ [2024-04-11 06:20:25] Verification-am step2 : Discovering pods ________________________________________________________________________________ 2024-04-11 06:20:25,638 INFO 2024-04-11 06:20:25,638 INFO --------------------- Get expected number of pods --------------------- 2024-04-11 06:20:25,639 INFO 2024-04-11 06:20:25,639 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas} 2024-04-11 06:20:25,639 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 06:20:26,396 INFO [loop_until]: OK (rc = 0) 2024-04-11 06:20:26,396 DEBUG --- stdout --- 2024-04-11 06:20:26,396 DEBUG 3 2024-04-11 06:20:26,396 DEBUG --- stderr --- 2024-04-11 06:20:26,396 DEBUG 2024-04-11 06:20:26,396 INFO 2024-04-11 06:20:26,396 INFO ---------------------------- Get pod list ---------------------------- 2024-04-11 06:20:26,397 INFO 2024-04-11 06:20:26,397 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}` | grep 3 2024-04-11 06:20:26,397 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-11 06:20:27,488 INFO [loop_until]: Function succeeded after 1s (rc=0) - expected number of elements found 2024-04-11 06:20:27,489 INFO [loop_until]: OK (rc = 0) 2024-04-11 06:20:27,489 DEBUG --- stdout --- 2024-04-11 06:20:27,489 DEBUG am-5b5bd8567b-6lzwk am-5b5bd8567b-8h2w4 am-5b5bd8567b-qlkw6 2024-04-11 06:20:27,489 DEBUG --- stderr --- 2024-04-11 06:20:27,489 DEBUG ________________________________________________________________________________ [2024-04-11 06:20:27] Verification-am step3 : Check component is alive ________________________________________________________________________________ 2024-04-11 06:20:27,490 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready 2024-04-11 06:20:27,490 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready" 2024-04-11 06:20:27,766 INFO [http_cmd]: http status code OK 2024-04-11 06:20:27,766 DEBUG --- status code --- 2024-04-11 06:20:27,768 DEBUG http status code is 200 (expected 200) 2024-04-11 06:20:27,768 DEBUG --- http response --- 2024-04-11 06:20:27,769 DEBUG 2024-04-11 06:20:27,769 INFO Authenticate user amadmin via REST 2024-04-11 06:20:27,770 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/" 2024-04-11 06:20:28,017 INFO [http_cmd]: http status code OK 2024-04-11 06:20:28,017 DEBUG --- status code --- 2024-04-11 06:20:28,017 DEBUG http status code is 200 (expected 200) 2024-04-11 06:20:28,017 DEBUG --- http response --- 2024-04-11 06:20:28,017 DEBUG { "tokenId": "8QL4MPFLoVVL4sd79biiYcPFAXo.*AAJTSQACMDIAAlNLABwwRHpXUC9KcCtYeWhvZU1HRGVsVGd5YlB4RFk9AAR0eXBlAANDVFMAAlMxAAIwMQ..*", "successUrl": "/am/console", "realm": "/" } PASS : component am is alive ________________________________________________________________________________ [2024-04-11 06:20:28] Verification-am step4 : Record version ________________________________________________________________________________ 2024-04-11 06:20:28,022 INFO 2024-04-11 06:20:28,022 INFO ----------------------- Get AM software version ----------------------- 2024-04-11 06:20:28,024 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version 2024-04-11 06:20:28,679 INFO [LoginSession] Get "token_id" token for user "amadmin" 2024-04-11 06:20:28,679 DEBUG [LoginSession] re-using "token_id" token that remains valid for 2m 48s 2024-04-11 06:20:28,680 INFO [http_cmd]: curl -L --request GET --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=SJuvIQPNwJrs5zE7R55FwwWQxVc.*AAJTSQACMDIAAlNLABx2SXVPQnNyeXhkOUdpZjdjNkU3NUI0VUpSNGc9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" --cookie "route=1712816428.931.26974.449746|f60edb382037eb2df1e800d563ad78a7" "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version" 2024-04-11 06:20:28,936 INFO [http_cmd]: http status code OK 2024-04-11 06:20:28,936 DEBUG --- status code --- 2024-04-11 06:20:28,936 DEBUG http status code is 200 (expected 200) 2024-04-11 06:20:28,936 DEBUG --- http response --- 2024-04-11 06:20:28,936 DEBUG { "_id": "version", "_rev": "1507272562", "version": "7.6.0-SNAPSHOT", "fullVersion": "ForgeRock Access Management 7.6.0-SNAPSHOT Build a89ff42c02de792a1493750badb7bf8554c69559 (2024-April-09 19:07)", "revision": "a89ff42c02de792a1493750badb7bf8554c69559", "date": "2024-April-09 19:07" } Version is 7.6.0-SNAPSHOT Revision is a89ff42c02de792a1493750badb7bf8554c69559 ________________________________________________________________________________ [2024-04-11 06:20:28] Verification-am step5 : Checking pod restart counts ________________________________________________________________________________ 2024-04-11 06:20:28,940 INFO 2024-04-11 06:20:28,940 INFO ------------- Check pod am-5b5bd8567b-6lzwk restart count ------------- 2024-04-11 06:20:28,941 INFO 2024-04-11 06:20:28,941 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-5b5bd8567b-6lzwk --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-11 06:20:28,941 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 06:20:29,330 INFO [loop_until]: OK (rc = 0) 2024-04-11 06:20:29,330 DEBUG --- stdout --- 2024-04-11 06:20:29,330 DEBUG 0 2024-04-11 06:20:29,330 DEBUG --- stderr --- 2024-04-11 06:20:29,330 DEBUG 2024-04-11 06:20:29,330 INFO Pod am-5b5bd8567b-6lzwk has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases 2024-04-11 06:20:29,330 INFO 2024-04-11 06:20:29,330 INFO ------------- Check pod am-5b5bd8567b-8h2w4 restart count ------------- 2024-04-11 06:20:29,330 INFO 2024-04-11 06:20:29,331 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-5b5bd8567b-8h2w4 --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-11 06:20:29,331 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 06:20:29,889 INFO [loop_until]: OK (rc = 0) 2024-04-11 06:20:29,889 DEBUG --- stdout --- 2024-04-11 06:20:29,889 DEBUG 0 2024-04-11 06:20:29,890 DEBUG --- stderr --- 2024-04-11 06:20:29,890 DEBUG 2024-04-11 06:20:29,890 INFO Pod am-5b5bd8567b-8h2w4 has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases 2024-04-11 06:20:29,890 INFO 2024-04-11 06:20:29,890 INFO ------------- Check pod am-5b5bd8567b-qlkw6 restart count ------------- 2024-04-11 06:20:29,890 INFO 2024-04-11 06:20:29,890 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-5b5bd8567b-qlkw6 --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-11 06:20:29,890 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-11 06:20:30,455 INFO [loop_until]: OK (rc = 0) 2024-04-11 06:20:30,455 DEBUG --- stdout --- 2024-04-11 06:20:30,455 DEBUG 0 2024-04-11 06:20:30,455 DEBUG --- stderr --- 2024-04-11 06:20:30,455 DEBUG 2024-04-11 06:20:30,455 INFO Pod am-5b5bd8567b-qlkw6 has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases ________________________________________________________________________________ [2024-04-11 06:20:30] Verification-am post : Post method ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped