--Task--
              name: Monitoring_start
           enabled: True
        class_name: MonitoringStart
       source_name: controller
  source_namespace: >default<
       target_name: controller
  target_namespace: >default<
             start: 0
              stop: None
           timeout: no timeout
              loop: False
          interval: None
      dependencies: []
          wait_for: []
           options: {}
        group_name: None
       Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock

________________________________________________________________________________

[2024-01-30 02:04:35] Monitoring_start pre : Initialising task parameters
________________________________________________________________________________

task will be executed on controller (localhost)
________________________________________________________________________________

[2024-01-30 02:04:35] Monitoring_start step1 : N/A
________________________________________________________________________________


****************************************    Start Lodemon   ****************************************

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm delete configmap lodemon-config
[loop_until]: (max_time=180, interval=5, expected_rc=[0, 1]
[loop_until]: OK (rc = 1)
--- stdout ---

--- stderr ---
Error from server (NotFound): configmaps "lodemon-config" not found

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm create configmap lodemon-config --from-file=/mnt/disks/data/xslou/lodestar-fork/config/config.yaml
[loop_until]: (max_time=180, interval=5, expected_rc=[0]
[loop_until]: OK (rc = 0)
--- stdout ---
configmap/lodemon-config created
--- stderr ---


[run_command]: gcloud iam service-accounts add-iam-policy-binding  pit-test@engineeringpit.iam.gserviceaccount.com                                 --role roles/iam.workloadIdentityUser                                 --member "serviceAccount:engineeringpit.svc.id.goog[xlou/k8s-svc-acct-lodemon]"
[run_command]: ERROR
--- rc ---
returned 1, expected to be in [0]
--- stdout ---

--- stderr ---
ERROR: (gcloud.iam.service-accounts.add-iam-policy-binding) PERMISSION_DENIED: Permission 'iam.serviceAccounts.setIamPolicy' denied on resource (or it may not exist).
- '@type': type.googleapis.com/google.rpc.ErrorInfo
  domain: iam.googleapis.com
  metadata:
    permission: iam.serviceAccounts.setIamPolicy
  reason: IAM_PERMISSION_DENIED

could not add the iam-policy binding to the GSA: pit-test@engineeringpit.iam.gserviceaccount.com for the member k8s-svc-acct-lodemon service account, Identity Cloud metrics will not be present in the Lodemon report.The exception thrown was: (1, , ERROR: (gcloud.iam.service-accounts.add-iam-policy-binding) PERMISSION_DENIED: Permission 'iam.serviceAccounts.setIamPolicy' denied on resource (or it may not exist).
- '@type': type.googleapis.com/google.rpc.ErrorInfo
  domain: iam.googleapis.com
  metadata:
    permission: iam.serviceAccounts.setIamPolicy
  reason: IAM_PERMISSION_DENIED
)

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm delete configmap lodemon-deployments
[loop_until]: (max_time=180, interval=5, expected_rc=[0, 1]
[loop_until]: OK (rc = 1)
--- stdout ---

--- stderr ---
Error from server (NotFound): configmaps "lodemon-deployments" not found

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm create configmap lodemon-deployments --from-file=/mnt/disks/data/xslou/lodestar-fork/config/deployments_to_monitor.yaml
[loop_until]: (max_time=180, interval=5, expected_rc=[0]
[loop_until]: OK (rc = 0)
--- stdout ---
configmap/lodemon-deployments created
--- stderr ---


-------------   Deploy the lodestarbox with lodemon profile   -------------

[run_command]: skaffold deploy --profile lodemon --config=/tmp/tmphk67bucz --status-check=true --namespace=xlou
Starting deploy...
- serviceaccount/k8s-svc-acct-lodemon created
- clusterrolebinding.rbac.authorization.k8s.io/k8s-svc-acct-crb-xlou created
- deployment.apps/lodemon created
Waiting for deployments to stabilize...
- xlou:deployment/lodemon: waiting for rollout to finish: 0 of 1 updated replicas are available...
- xlou:deployment/lodemon is ready.
Deployments stabilized in 20.676 seconds
There is a new version (2.10.0) of Skaffold available. Download it from:
https://github.com/GoogleContainerTools/skaffold/releases/tag/v2.10.0

Help improve Skaffold with our 2-minute anonymous survey: run 'skaffold survey'
To help improve the quality of this product, we collect anonymized usage data for details on what is tracked and how we use this data visit <https://skaffold.dev/docs/resources/telemetry/>. This data is handled in accordance with our privacy policy <https://policies.google.com/privacy>

You may choose to opt out of this collection by running the following command:
skaffold config set --global collect-metrics false

---------------------   Get expected number of pods   ---------------------

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=lodemon --output jsonpath={.items[*].spec.replicas}
[loop_until]: (max_time=180, interval=5, expected_rc=[0]
[loop_until]: OK (rc = 0)
--- stdout ---
1
--- stderr ---


----------------------------    Get pod list   ----------------------------

[loop_until]: awk -F" " "{print NF}" <<< `kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods --selector app=lodemon --output jsonpath={.items[*].metadata.name}` | grep 1
[loop_until]: (max_time=180, interval=10, expected_rc=[0]
[loop_until]: Function succeeded after 0s (rc=0) - expected number of elements found
[loop_until]: OK (rc = 0)
--- stdout ---
lodemon-59c988bdf9-2gc4d
--- stderr ---


------------   Check pod lodemon-59c988bdf9-2gc4d is running   ------------

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods lodemon-59c988bdf9-2gc4d -o=jsonpath={.status.phase} | grep "Running"
[loop_until]: (max_time=360, interval=5, expected_rc=[0]
[loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
[loop_until]: OK (rc = 0)
--- stdout ---
Running
--- stderr ---


[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pods lodemon-59c988bdf9-2gc4d -o=jsonpath={.status.containerStatuses[*].ready} | grep "true"
[loop_until]: (max_time=360, interval=5, expected_rc=[0]
[loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
[loop_until]: OK (rc = 0)
--- stdout ---
true
--- stderr ---


[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod lodemon-59c988bdf9-2gc4d --output jsonpath={.status.startTime}
[loop_until]: (max_time=180, interval=5, expected_rc=[0]
[loop_until]: OK (rc = 0)
--- stdout ---
2024-01-30T02:05:41Z
--- stderr ---


-----   Check pod lodemon-59c988bdf9-2gc4d filesystem is accessible   -----
[loop_until]: Function succeeded after 0s (rc=0) - expected pattern found

----------    Check pod lodemon-59c988bdf9-2gc4d restart count   ----------

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get pod lodemon-59c988bdf9-2gc4d --output jsonpath={.status.containerStatuses[*].restartCount}
[loop_until]: (max_time=180, interval=5, expected_rc=[0]
[loop_until]: OK (rc = 0)
--- stdout ---
0
--- stderr ---

Pod lodemon-59c988bdf9-2gc4d has been restarted 0 times.

---------------------   Get expected number of pods   ---------------------

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments --selector app=lodemon --output jsonpath={.items[*].spec.replicas}
[loop_until]: (max_time=180, interval=5, expected_rc=[0]
[loop_until]: OK (rc = 0)
--- stdout ---
1
--- stderr ---


--------------   Waiting for 1 expected pod(s) to be ready   --------------

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm get deployments lodemon --output jsonpath="ready:{.status.readyReplicas} replicas:{.status.replicas}" | grep "ready:1 replicas:1"
[loop_until]: (max_time=900, interval=30, expected_rc=[0]
[loop_until]: Function succeeded after 0s (rc=0) - expected pattern found
[loop_until]: OK (rc = 0)
--- stdout ---
ready:1 replicas:1
--- stderr ---

Component lodemon is alive
Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/pod-logs/20240130_020605-after-lodemon-deploy/lodemon-59c988bdf9-2gc4d.txt
Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/pod-logs/20240130_020605-after-lodemon-deploy/lodemon-59c988bdf9-2gc4d.txt
Check pod logs for errors

[loop_until]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec lodemon-59c988bdf9-2gc4d -- curl --fail --silent --show-error http://localhost:8080/monitoring/start
[loop_until]: (max_time=180, interval=5, expected_rc=[0]
[loop_until]: OK (rc = 0)
--- stdout ---
{"Status": "OK", "Message": "Monitoring has been started"}
--- stderr ---

Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/pod-logs/stack/after-lodemon-deployment/lodemon-59c988bdf9-2gc4d.txt
Dumping pod description and logs to /mnt/disks/data/xslou/lodestar-fork/results/pyrock/idc_login_tree/pod-logs/stack/after-lodemon-deployment/lodemon-59c988bdf9-2gc4d.txt
Check pod logs for errors
________________________________________________________________________________

[2024-01-30 02:06:08] Monitoring_start post : Post method
________________________________________________________________________________

Setting result to PASS
Task has been successfully stopped