--Task-- name: Verification-am enabled: True class_name: VerificationTask source_name: am source_namespace: >default< target_name: am target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: [] wait_for: [] preceding_task: None options: {} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [2024-04-15 22:46:48] Verification-am pre : N/A ________________________________________________________________________________ [2024-04-15 22:46:48] - INFO: Timeout for this Task has changed to 2h (7200 seconds) [2024-04-15 22:46:48] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset ________________________________________________________________________________ [2024-04-15 22:46:48] Verification-am step1 : Wait until pods are in Running state and check number of running pods ________________________________________________________________________________ 2024-04-15 22:46:48,602 INFO 2024-04-15 22:46:48,602 INFO --------------------- Get expected number of pods --------------------- 2024-04-15 22:46:48,603 INFO 2024-04-15 22:46:48,603 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas} 2024-04-15 22:46:48,619 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-15 22:46:49,639 INFO [loop_until]: OK (rc = 0) 2024-04-15 22:46:49,639 DEBUG --- stdout --- 2024-04-15 22:46:49,640 DEBUG 3 2024-04-15 22:46:49,640 DEBUG --- stderr --- 2024-04-15 22:46:49,640 DEBUG 2024-04-15 22:46:49,640 INFO 2024-04-15 22:46:49,640 INFO -------------- Waiting for 3 expected pod(s) to be ready -------------- 2024-04-15 22:46:49,640 INFO 2024-04-15 22:46:49,640 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments am --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:3 replicas:3" 2024-04-15 22:46:49,640 INFO [loop_until]: (max_time=900, interval=30, expected_rc=[0] 2024-04-15 22:46:50,606 INFO [loop_until]: Function succeeded after 0s (rc=0) - expected pattern found 2024-04-15 22:46:50,606 INFO [loop_until]: OK (rc = 0) 2024-04-15 22:46:50,606 DEBUG --- stdout --- 2024-04-15 22:46:50,606 DEBUG ready:3 replicas:3 2024-04-15 22:46:50,607 DEBUG --- stderr --- 2024-04-15 22:46:50,607 DEBUG ________________________________________________________________________________ [2024-04-15 22:46:50] Verification-am step2 : Discovering pods ________________________________________________________________________________ 2024-04-15 22:46:50,608 INFO 2024-04-15 22:46:50,608 INFO --------------------- Get expected number of pods --------------------- 2024-04-15 22:46:50,608 INFO 2024-04-15 22:46:50,608 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=am --output jsonpath={.items[*].spec.replicas} 2024-04-15 22:46:50,608 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-15 22:46:51,259 INFO [loop_until]: OK (rc = 0) 2024-04-15 22:46:51,260 DEBUG --- stdout --- 2024-04-15 22:46:51,260 DEBUG 3 2024-04-15 22:46:51,260 DEBUG --- stderr --- 2024-04-15 22:46:51,260 DEBUG 2024-04-15 22:46:51,260 INFO 2024-04-15 22:46:51,260 INFO ---------------------------- Get pod list ---------------------------- 2024-04-15 22:46:51,260 INFO 2024-04-15 22:46:51,261 INFO [loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=am --output jsonpath={.items[*].metadata.name}` | grep 3 2024-04-15 22:46:51,261 INFO [loop_until]: (max_time=180, interval=10, expected_rc=[0] 2024-04-15 22:46:52,313 INFO [loop_until]: Function succeeded after 1s (rc=0) - expected number of elements found 2024-04-15 22:46:52,313 INFO [loop_until]: OK (rc = 0) 2024-04-15 22:46:52,313 DEBUG --- stdout --- 2024-04-15 22:46:52,313 DEBUG am-7d597cf7d6-8ctft am-7d597cf7d6-8vr8f am-7d597cf7d6-9wz9h 2024-04-15 22:46:52,313 DEBUG --- stderr --- 2024-04-15 22:46:52,313 DEBUG ________________________________________________________________________________ [2024-04-15 22:46:52] Verification-am step3 : Check component is alive ________________________________________________________________________________ 2024-04-15 22:46:52,315 INFO Livecheck to https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready 2024-04-15 22:46:52,317 INFO [http_cmd]: curl -L --request GET "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/health/ready" 2024-04-15 22:46:52,597 INFO [http_cmd]: http status code OK 2024-04-15 22:46:52,597 DEBUG --- status code --- 2024-04-15 22:46:52,597 DEBUG http status code is 200 (expected 200) 2024-04-15 22:46:52,597 DEBUG --- http response --- 2024-04-15 22:46:52,597 DEBUG 2024-04-15 22:46:52,597 INFO Authenticate user amadmin via REST 2024-04-15 22:46:52,597 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/" 2024-04-15 22:46:52,838 INFO [http_cmd]: http status code OK 2024-04-15 22:46:52,838 DEBUG --- status code --- 2024-04-15 22:46:52,838 DEBUG http status code is 200 (expected 200) 2024-04-15 22:46:52,838 DEBUG --- http response --- 2024-04-15 22:46:52,839 DEBUG { "tokenId": "A19EA7jIv4BrJZTvUJQvBrZp_D8.*AAJTSQACMDIAAlNLABxTNmNJTElzd0lHN0lCTWFiVCtTTmlCbHJlb1k9AAR0eXBlAANDVFMAAlMxAAIwMQ..*", "successUrl": "/am/console", "realm": "/" } PASS : component am is alive ________________________________________________________________________________ [2024-04-15 22:46:52] Verification-am step4 : Record version ________________________________________________________________________________ 2024-04-15 22:46:52,846 INFO 2024-04-15 22:46:52,846 INFO ----------------------- Get AM software version ----------------------- 2024-04-15 22:46:52,847 INFO Getting product version from https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version 2024-04-15 22:46:52,847 INFO [LoginSession] Get "token_id" token for user "amadmin" 2024-04-15 22:46:52,847 INFO [LoginSession] Obtaining new "token_id" token from server because it has expired 2024-04-15 22:46:52,848 INFO Authenticate user amadmin via REST 2024-04-15 22:46:52,848 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate?realm=/" 2024-04-15 22:46:53,063 INFO [http_cmd]: http status code OK 2024-04-15 22:46:53,063 DEBUG --- status code --- 2024-04-15 22:46:53,063 DEBUG http status code is 200 (expected 200) 2024-04-15 22:46:53,063 DEBUG --- http response --- 2024-04-15 22:46:53,063 DEBUG { "tokenId": "3FP0Abl_vgP94GQ8-HM_EonQCZY.*AAJTSQACMDIAAlNLABxsZWNnMHlOTndXU0xNZjZXUXNhbmFiZFlrY2c9AAR0eXBlAANDVFMAAlMxAAIwMQ..*", "successUrl": "/am/console", "realm": "/" } 2024-04-15 22:46:53,064 INFO [LoginSession] Obtaining session info 2024-04-15 22:46:53,064 INFO [http_cmd]: curl -L --request POST "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/realms/root/sessions?_action=getSessionInfo" 2024-04-15 22:46:53,301 INFO [http_cmd]: http status code OK 2024-04-15 22:46:53,302 DEBUG --- status code --- 2024-04-15 22:46:53,302 DEBUG http status code is 200 (expected 200) 2024-04-15 22:46:53,302 DEBUG --- http response --- 2024-04-15 22:46:53,302 DEBUG { "username": "amadmin", "universalId": "id=amadmin,ou=user,ou=am-config", "realm": "/", "latestAccessTime": "2024-04-15T22:46:53Z", "maxIdleExpirationTime": "2024-04-15T22:49:53Z", "maxSessionExpirationTime": "2024-04-15T22:49:52Z", "properties": { "AMCtxId": "375a847b-e922-4197-9d6a-358b574b265d-972066218" } } 2024-04-15 22:46:53,330 INFO [http_cmd]: curl -L --request GET --cookie "amlbcookie=01" --cookie "iPlanetDirectoryPro=3FP0Abl_vgP94GQ8-HM_EonQCZY.*AAJTSQACMDIAAlNLABxsZWNnMHlOTndXU0xNZjZXUXNhbmFiZFlrY2c9AAR0eXBlAANDVFMAAlMxAAIwMQ..*" --cookie "route=1713221214.024.49308.283344|f60edb382037eb2df1e800d563ad78a7" "https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version" 2024-04-15 22:46:53,552 INFO [http_cmd]: http status code OK 2024-04-15 22:46:53,553 DEBUG --- status code --- 2024-04-15 22:46:53,553 DEBUG http status code is 200 (expected 200) 2024-04-15 22:46:53,553 DEBUG --- http response --- 2024-04-15 22:46:53,554 DEBUG { "_id": "version", "_rev": "509589602", "version": "7.6.0-SNAPSHOT", "fullVersion": "ForgeRock Access Management 7.6.0-SNAPSHOT Build bbf6791c961b39ef101ebad36686ee2c3cde94bb (2024-April-12 18:28)", "revision": "bbf6791c961b39ef101ebad36686ee2c3cde94bb", "date": "2024-April-12 18:28" } Version is 7.6.0-SNAPSHOT Revision is bbf6791c961b39ef101ebad36686ee2c3cde94bb ________________________________________________________________________________ [2024-04-15 22:46:53] Verification-am step5 : Checking pod restart counts ________________________________________________________________________________ 2024-04-15 22:46:53,565 INFO 2024-04-15 22:46:53,565 INFO ------------- Check pod am-7d597cf7d6-8ctft restart count ------------- 2024-04-15 22:46:53,565 INFO 2024-04-15 22:46:53,565 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8ctft --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-15 22:46:53,566 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-15 22:46:54,110 INFO [loop_until]: OK (rc = 0) 2024-04-15 22:46:54,110 DEBUG --- stdout --- 2024-04-15 22:46:54,111 DEBUG 0 2024-04-15 22:46:54,111 DEBUG --- stderr --- 2024-04-15 22:46:54,111 DEBUG 2024-04-15 22:46:54,111 INFO Pod am-7d597cf7d6-8ctft has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases 2024-04-15 22:46:54,111 INFO 2024-04-15 22:46:54,111 INFO ------------- Check pod am-7d597cf7d6-8vr8f restart count ------------- 2024-04-15 22:46:54,111 INFO 2024-04-15 22:46:54,111 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-8vr8f --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-15 22:46:54,112 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-15 22:46:54,615 INFO [loop_until]: OK (rc = 0) 2024-04-15 22:46:54,615 DEBUG --- stdout --- 2024-04-15 22:46:54,616 DEBUG 0 2024-04-15 22:46:54,616 DEBUG --- stderr --- 2024-04-15 22:46:54,616 DEBUG 2024-04-15 22:46:54,616 INFO Pod am-7d597cf7d6-8vr8f has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases 2024-04-15 22:46:54,616 INFO 2024-04-15 22:46:54,616 INFO ------------- Check pod am-7d597cf7d6-9wz9h restart count ------------- 2024-04-15 22:46:54,617 INFO 2024-04-15 22:46:54,618 INFO [loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod am-7d597cf7d6-9wz9h --output jsonpath={.status.containerStatuses[*].restartCount} 2024-04-15 22:46:54,619 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 2024-04-15 22:46:55,447 INFO [loop_until]: OK (rc = 0) 2024-04-15 22:46:55,447 DEBUG --- stdout --- 2024-04-15 22:46:55,447 DEBUG 0 2024-04-15 22:46:55,447 DEBUG --- stderr --- 2024-04-15 22:46:55,448 DEBUG 2024-04-15 22:46:55,448 INFO Pod am-7d597cf7d6-9wz9h has been restarted 0 times. SUCCESS : pod restart count (0) did not change between validation and verification phases ________________________________________________________________________________ [2024-04-15 22:46:55] Verification-am post : Post method ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped