--Task--
              name: Monitoring_stop
           enabled: True
        class_name: MonitoringStopAndCollectResults
       source_name: controller
  source_namespace: >default<
       target_name: controller
  target_namespace: >default<
             start: 0.0
              stop: None
           timeout: no timeout
              loop: False
          interval: None
      dependencies: []
          wait_for: []
    preceding_task: None
           options: {}
        group_name: None
       Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock

task will be executed on controller (localhost)
2025-09-11 00:40:57 - INFO: Interval for this Task has changed to 5s (5 seconds)
2025-09-11 00:40:57 - INFO: interval was set to Task default because it was unset, based on self.timeout value because task is allowed to stop by itself
________________________________________________________________________________

[2025-09-11 00:40:57] Monitoring_stop step1 : N/A
________________________________________________________________________________

2025-09-11 00:40:57,719 INFO    
2025-09-11 00:40:57,719 INFO    *****************************************   Stop Lodemon   *****************************************
2025-09-11 00:40:57,719 INFO    
2025-09-11 00:40:57,720 INFO    [run_command]: kubectl --namespace=xlou --context=gke_engineeringpit_us-east1-d_xlou-cdm exec lodemon-6fddcf76bb-nj6jh -- curl --fail --silent -m 60 --connect-timeout 60 -X POST --show-error http://localhost:8080/monitoring/stop
2025-09-11 00:40:58,263 INFO    [run_command]: OK (rc = 0 - expected to be in [0])
2025-09-11 00:40:58,264 DEBUG   --- stdout ---
2025-09-11 00:40:58,264 DEBUG   {"status":"monitoring stopped"}
2025-09-11 00:40:58,264 DEBUG   --- stderr ---
2025-09-11 00:40:58,264 DEBUG   
2025-09-11 00:40:58,264 INFO    
2025-09-11 00:40:58,264 INFO    
2025-09-11 00:40:58,264 INFO    [run_command]: gcloud iam service-accounts remove-iam-policy-binding  pit-test@engineeringpit.iam.gserviceaccount.com                     --project engineeringpit                     --role roles/iam.workloadIdentityUser                     --member "serviceAccount:engineeringpit.svc.id.goog[xlou/k8s-svc-acct-lodemon]"
2025-09-11 00:40:59,356 ERROR   [run_command]: ERROR
2025-09-11 00:40:59,356 ERROR   --- rc ---
2025-09-11 00:40:59,356 ERROR   returned 1, expected to be in [0]
2025-09-11 00:40:59,356 ERROR   --- stdout ---
2025-09-11 00:40:59,356 ERROR   
2025-09-11 00:40:59,356 ERROR   --- stderr ---
2025-09-11 00:40:59,356 ERROR   ERROR: (gcloud.iam.service-accounts.remove-iam-policy-binding) Policy binding with the specified principal, role, and condition not found!
2025-09-11 00:40:59,356 INFO    
2025-09-11 00:40:59,356 INFO    the operation gcloud iam service-accounts remove-iam-policy-binding for the SA: pit-test@engineeringpit.iam.gserviceaccount.com and member: "serviceAccount:engineeringpit.svc.id.goog[xlou/k8s-svc-acct-lodemon]" has failed. The exception thrown was: Command error: rc 1 -- stdout:  -- stderr:ERROR: (gcloud.iam.service-accounts.remove-iam-policy-binding) Policy binding with the specified principal, role, and condition not found!This is expected if the IAM policy binding was not created or we don't have access
2025-09-11 00:40:59,356 INFO    Downloading report from lodemon pod lodemon-6fddcf76bb-nj6jh:/tmp/lodemon/report.tar.gz
2025-09-11 00:40:59,357 INFO    
2025-09-11 00:40:59,357 INFO    [loop_until]: kubectl --namespace=xlou cp lodemon-6fddcf76bb-nj6jh:/tmp/lodemon/report.tar.gz /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/lodemon.tar.gz
2025-09-11 00:40:59,357 INFO    [loop_until]: (max_time=180, interval=5, expected_rc=[0]
2025-09-11 00:40:59,880 INFO    [loop_until]: OK (rc = 0)
2025-09-11 00:40:59,881 DEBUG   --- stdout ---
2025-09-11 00:40:59,881 DEBUG   tar: Removing leading `/' from member names
2025-09-11 00:40:59,881 DEBUG   --- stderr ---
2025-09-11 00:40:59,881 DEBUG   
2025-09-11 00:40:59,881 INFO    Extracting report to /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/lodemon
2025-09-11 00:41:00,678 INFO    Dumping pod description and logs to /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/pod-logs/stack/after-lodemon-stop/lodemon-6fddcf76bb-nj6jh.txt
2025-09-11 00:41:00,678 INFO    Dumping pod description and logs to /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/pod-logs/stack/after-lodemon-stop/lodemon-6fddcf76bb-nj6jh.txt
2025-09-11 00:41:00,678 INFO    Check pod logs for errors
2025-09-11 00:41:01,285 INFO    Dumping pod description and logs to /mnt/disk1/xslou/workshop/lodestar-fork/results/pyrock/idc_benchmark/pod-logs/stack/after-lodemon-stop/_cpu-info.txt
________________________________________________________________________________

[2025-09-11 00:41:01] Monitoring_stop post : Post method
________________________________________________________________________________

Setting result to PASS
Task has been successfully stopped