==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-747d57dcb8-lclbf Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Fri, 19 Apr 2024 05:45:47 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=747d57dcb8 skaffold.dev/run-id=7964740a-b536-4c2c-b9b0-ce06901e2063 Annotations: Status: Running IP: 10.106.46.166 IPs: IP: 10.106.46.166 Controlled By: ReplicaSet/lodemon-747d57dcb8 Containers: lodemon: Container ID: containerd://d19607a9b3b18a70f23619f5909c8271ce451f0daf18c04d0b88c8306023ad6d Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:03f76e6326d713f4d73dc474e0e25fbbbaecc7c1d8c0590bfd94d68ac0d98c97 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Fri, 19 Apr 2024 05:46:03 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-586b6 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-586b6: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 12:51:55 INFO 12:51:55 INFO [loop_until]: kubectl --namespace=xlou top pods 12:51:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:51:55 INFO [loop_until]: OK (rc = 0) 12:51:55 DEBUG --- stdout --- 12:51:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2341m 9433Mi am-6fd554f8f7-hvvkw 2383m 9010Mi am-6fd554f8f7-pmj4b 2452m 9133Mi ds-cts-0 10753m 13325Mi ds-cts-1 9149m 13468Mi ds-cts-2 10874m 13428Mi ds-idrepo-0 683m 3365Mi ds-idrepo-1 699m 3263Mi ds-idrepo-2 693m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-747d57dcb8-lclbf 7m 70Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 999m 763Mi 12:51:55 DEBUG --- stderr --- 12:51:55 DEBUG 12:52:01 INFO 12:52:01 INFO [loop_until]: kubectl --namespace=xlou top node 12:52:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:52:01 INFO [loop_until]: OK (rc = 0) 12:52:01 DEBUG --- stdout --- 12:52:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6128Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 758m 2% 6887Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2363m 7% 11759Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 3240Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10601m 35% 16358Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5180Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2502m 8% 12265Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3228Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2674m 8% 11372Mi 10% gke-xlou-cdm-ds-0f601751-j488 740m 2% 6361Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9553m 31% 18052Mi 16% gke-xlou-cdm-ds-0f601751-nj6q 784m 2% 7010Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 10387m 34% 16011Mi 14% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 4775Mi 8% 12:52:01 DEBUG --- stderr --- 12:52:01 DEBUG 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713527557" 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG { 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "data": { 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "result": [ 12:52:37 DEBUG { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "pod": "ds-cts-0" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG [...] 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG "pod": "ds-cts-2" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 1713527557, 12:52:37 DEBUG "0.0004968866666666666" 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 ERROR [http_cmd]: ERROR 12:52:37 ERROR --- status code --- 12:52:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:52:37 ERROR --- http response --- 12:52:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:52:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG { 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "data": { 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "result": [ 12:52:37 DEBUG { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "pod": "ds-cts-0" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG [...] 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG "pod": "ds-cts-2" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 1713527557, 12:52:37 DEBUG "13.4" 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG { 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "data": { 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "result": [ 12:52:37 DEBUG { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "pod": "ds-idrepo-0" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG [...] 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG "pod": "ds-cts-2" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 1713527557, 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 DEBUG "0" 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG ] 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG } 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG ] 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:37 DEBUG } 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG } 12:52:37 DEBUG { 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "data": { 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "result": [ 12:52:37 DEBUG { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "pod": "ds-idrepo-0" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG [...] 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG "pod": "ds-idrepo-2" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 1713527557, 12:52:37 DEBUG "0" 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713527557" 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG { 12:52:37 DEBUG { 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "data": { 12:52:37 DEBUG "data": { 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "result": [ 12:52:37 DEBUG "result": [ 12:52:37 DEBUG { 12:52:37 DEBUG { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:52:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:52:37 DEBUG }, 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG [...] 12:52:37 DEBUG [...] 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:52:37 DEBUG "pod": "ds-idrepo-2" 12:52:37 DEBUG }, 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 1713527557, 12:52:37 DEBUG 1713527557, 12:52:37 DEBUG "2831234" 12:52:37 DEBUG "4143.7829976921375" 12:52:37 DEBUG ] 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 DEBUG ] 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG { 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "data": { 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "result": [ 12:52:37 DEBUG { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG [...] 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 1713527557, 12:52:37 DEBUG "1.0326217639502935" 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG { 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "data": { 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "result": [ 12:52:37 DEBUG { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG [...] 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 1713527557, 12:52:37 DEBUG "2499.1666666666665" 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527557" 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 INFO [http_cmd]: http status code OK 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG --- status code --- 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG --- http response --- 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG ----- output ----- 12:52:37 DEBUG { 12:52:37 DEBUG { 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "status": "success", 12:52:37 DEBUG "data": { 12:52:37 DEBUG "data": { 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "resultType": "vector", 12:52:37 DEBUG "result": [ 12:52:37 DEBUG "result": [ 12:52:37 DEBUG { 12:52:37 DEBUG { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "metric": { 12:52:37 DEBUG "pod": "ds-idrepo-0" 12:52:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:52:37 DEBUG }, 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG [...] 12:52:37 DEBUG [...] 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG 12:52:37 DEBUG "pod": "ds-idrepo-2" 12:52:37 DEBUG "pod": "ds-idrepo-2" 12:52:37 DEBUG }, 12:52:37 DEBUG }, 12:52:37 DEBUG "value": [ 12:52:37 DEBUG "value": [ 12:52:37 DEBUG 1713527557, 12:52:37 DEBUG 1713527557, 12:52:37 DEBUG "0" 12:52:37 DEBUG "0" 12:52:37 DEBUG ] 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 DEBUG ] 12:52:37 DEBUG ] 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:37 DEBUG } 12:52:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527558" 12:52:38 INFO [http_cmd]: http status code OK 12:52:38 DEBUG --- status code --- 12:52:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:38 DEBUG --- http response --- 12:52:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:38 DEBUG ----- output ----- 12:52:38 DEBUG { 12:52:38 DEBUG "status": "success", 12:52:38 DEBUG "data": { 12:52:38 DEBUG "resultType": "vector", 12:52:38 DEBUG "result": [ 12:52:38 DEBUG { 12:52:38 DEBUG "metric": { 12:52:38 DEBUG "instance": "10.142.0.107:9100" 12:52:38 DEBUG }, 12:52:38 DEBUG "value": [ 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG [...] 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG "instance": "10.142.0.43:9100" 12:52:38 DEBUG }, 12:52:38 DEBUG "value": [ 12:52:38 DEBUG 1713527558, 12:52:38 DEBUG "1.0199999999999934" 12:52:38 DEBUG ] 12:52:38 DEBUG } 12:52:38 DEBUG ] 12:52:38 DEBUG } 12:52:38 DEBUG } 12:52:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527558" 12:52:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527558" 12:52:38 INFO [http_cmd]: http status code OK 12:52:38 INFO [http_cmd]: http status code OK 12:52:38 DEBUG --- status code --- 12:52:38 DEBUG --- status code --- 12:52:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:38 DEBUG --- http response --- 12:52:38 DEBUG --- http response --- 12:52:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:38 DEBUG ----- output ----- 12:52:38 DEBUG { 12:52:38 DEBUG "status": "success", 12:52:38 DEBUG "data": { 12:52:38 DEBUG "resultType": "vector", 12:52:38 DEBUG "result": [ 12:52:38 DEBUG { 12:52:38 DEBUG "metric": { 12:52:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:52:38 DEBUG ----- output ----- 12:52:38 DEBUG }, 12:52:38 DEBUG { 12:52:38 DEBUG "value": [ 12:52:38 DEBUG "status": "success", 12:52:38 DEBUG 12:52:38 DEBUG "data": { 12:52:38 DEBUG 12:52:38 DEBUG "resultType": "vector", 12:52:38 DEBUG 12:52:38 DEBUG "result": [ 12:52:38 DEBUG [...] 12:52:38 DEBUG { 12:52:38 DEBUG 12:52:38 DEBUG "metric": { 12:52:38 DEBUG 12:52:38 DEBUG "instance": "10.142.0.107:9100" 12:52:38 DEBUG 12:52:38 DEBUG }, 12:52:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:52:38 DEBUG "value": [ 12:52:38 DEBUG }, 12:52:38 DEBUG 12:52:38 DEBUG "value": [ 12:52:38 DEBUG 12:52:38 DEBUG 1713527558, 12:52:38 DEBUG 12:52:38 DEBUG "0" 12:52:38 DEBUG [...] 12:52:38 DEBUG ] 12:52:38 DEBUG 12:52:38 DEBUG } 12:52:38 DEBUG 12:52:38 DEBUG ] 12:52:38 DEBUG 12:52:38 DEBUG } 12:52:38 DEBUG "instance": "10.142.0.43:9100" 12:52:38 DEBUG } 12:52:38 DEBUG }, 12:52:38 DEBUG "value": [ 12:52:38 DEBUG 1713527558, 12:52:38 DEBUG "0.68583333333335" 12:52:38 DEBUG ] 12:52:38 DEBUG } 12:52:38 DEBUG ] 12:52:38 DEBUG } 12:52:38 DEBUG } 12:52:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527558" 12:52:38 INFO [http_cmd]: http status code OK 12:52:38 DEBUG --- status code --- 12:52:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:38 DEBUG --- http response --- 12:52:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:38 DEBUG ----- output ----- 12:52:38 DEBUG { 12:52:38 DEBUG "status": "success", 12:52:38 DEBUG "data": { 12:52:38 DEBUG "resultType": "vector", 12:52:38 DEBUG "result": [ 12:52:38 DEBUG { 12:52:38 DEBUG "metric": { 12:52:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:52:38 DEBUG }, 12:52:38 DEBUG "value": [ 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG [...] 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:52:38 DEBUG }, 12:52:38 DEBUG "value": [ 12:52:38 DEBUG 1713527558, 12:52:38 DEBUG "0" 12:52:38 DEBUG ] 12:52:38 DEBUG } 12:52:38 DEBUG ] 12:52:38 DEBUG } 12:52:38 DEBUG } 12:52:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527558" 12:52:38 INFO [http_cmd]: http status code OK 12:52:38 DEBUG --- status code --- 12:52:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:38 DEBUG --- http response --- 12:52:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:38 DEBUG ----- output ----- 12:52:38 DEBUG { 12:52:38 DEBUG "status": "success", 12:52:38 DEBUG "data": { 12:52:38 DEBUG "resultType": "vector", 12:52:38 DEBUG "result": [ 12:52:38 DEBUG { 12:52:38 DEBUG "metric": { 12:52:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:52:38 DEBUG }, 12:52:38 DEBUG "value": [ 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG [...] 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG 12:52:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:52:38 DEBUG }, 12:52:38 DEBUG "value": [ 12:52:38 DEBUG 1713527558, 12:52:38 DEBUG "0" 12:52:38 DEBUG ] 12:52:38 DEBUG } 12:52:38 DEBUG ] 12:52:38 DEBUG } 12:52:38 DEBUG } 12:52:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713527559" 12:52:39 INFO [http_cmd]: http status code OK 12:52:39 DEBUG --- status code --- 12:52:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:39 DEBUG --- http response --- 12:52:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:39 DEBUG ----- output ----- 12:52:39 DEBUG { 12:52:39 DEBUG "status": "success", 12:52:39 DEBUG "data": { 12:52:39 DEBUG "resultType": "vector", 12:52:39 DEBUG "result": [ 12:52:39 DEBUG { 12:52:39 DEBUG "metric": { 12:52:39 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:52:39 DEBUG }, 12:52:39 DEBUG "value": [ 12:52:39 DEBUG 12:52:39 DEBUG 12:52:39 DEBUG 12:52:39 DEBUG [...] 12:52:39 DEBUG 12:52:39 DEBUG 12:52:39 DEBUG 12:52:39 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:52:39 DEBUG }, 12:52:39 DEBUG "value": [ 12:52:39 DEBUG 1713527559, 12:52:39 DEBUG "1089982464" 12:52:39 DEBUG ] 12:52:39 DEBUG } 12:52:39 DEBUG ] 12:52:39 DEBUG } 12:52:39 DEBUG } 12:52:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527561" 12:52:41 INFO [http_cmd]: http status code OK 12:52:41 DEBUG --- status code --- 12:52:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:41 DEBUG --- http response --- 12:52:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:52:41 DEBUG ----- output ----- 12:52:41 DEBUG { 12:52:41 DEBUG "status": "success", 12:52:41 DEBUG "data": { 12:52:41 DEBUG "resultType": "vector", 12:52:41 DEBUG "result": [ 12:52:41 DEBUG { 12:52:41 DEBUG "metric": { 12:52:41 DEBUG "pod": "ds-cts-0" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG [...] 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG "pod": "ds-cts-2" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 1713527561, 12:52:41 DEBUG "29.966666666666665" 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG } 12:52:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713527561" 12:52:41 INFO [http_cmd]: http status code OK 12:52:41 DEBUG --- status code --- 12:52:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:41 DEBUG --- http response --- 12:52:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:41 DEBUG ----- output ----- 12:52:41 DEBUG { 12:52:41 DEBUG "status": "success", 12:52:41 DEBUG "data": { 12:52:41 DEBUG "resultType": "vector", 12:52:41 DEBUG "result": [ 12:52:41 DEBUG { 12:52:41 DEBUG "metric": { 12:52:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG [...] 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 1713527561, 12:52:41 DEBUG "55.19" 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG } 12:52:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527561" 12:52:41 INFO [http_cmd]: http status code OK 12:52:41 DEBUG --- status code --- 12:52:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:41 DEBUG --- http response --- 12:52:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:52:41 DEBUG ----- output ----- 12:52:41 DEBUG { 12:52:41 DEBUG "status": "success", 12:52:41 DEBUG "data": { 12:52:41 DEBUG "resultType": "vector", 12:52:41 DEBUG "result": [ 12:52:41 DEBUG { 12:52:41 DEBUG "metric": { 12:52:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG [...] 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG "pod": "ds-idrepo-2" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 1713527561, 12:52:41 DEBUG "0" 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG } 12:52:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527561" 12:52:41 INFO [http_cmd]: http status code OK 12:52:41 DEBUG --- status code --- 12:52:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:41 DEBUG --- http response --- 12:52:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:41 DEBUG ----- output ----- 12:52:41 DEBUG { 12:52:41 DEBUG "status": "success", 12:52:41 DEBUG "data": { 12:52:41 DEBUG "resultType": "vector", 12:52:41 DEBUG "result": [ 12:52:41 DEBUG { 12:52:41 DEBUG "metric": { 12:52:41 DEBUG "pod": "ds-idrepo-0" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG [...] 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG "pod": "ds-idrepo-2" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 1713527561, 12:52:41 DEBUG "0" 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG } 12:52:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527561" 12:52:41 INFO [http_cmd]: http status code OK 12:52:41 DEBUG --- status code --- 12:52:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:41 DEBUG --- http response --- 12:52:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:52:41 DEBUG ----- output ----- 12:52:41 DEBUG { 12:52:41 DEBUG "status": "success", 12:52:41 DEBUG "data": { 12:52:41 DEBUG "resultType": "vector", 12:52:41 DEBUG "result": [ 12:52:41 DEBUG { 12:52:41 DEBUG "metric": { 12:52:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG [...] 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG "pod": "ds-idrepo-2" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 1713527561, 12:52:41 DEBUG "0" 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG } 12:52:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527561" 12:52:41 INFO [http_cmd]: http status code OK 12:52:41 DEBUG --- status code --- 12:52:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:41 DEBUG --- http response --- 12:52:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:41 DEBUG ----- output ----- 12:52:41 DEBUG { 12:52:41 DEBUG "status": "success", 12:52:41 DEBUG "data": { 12:52:41 DEBUG "resultType": "vector", 12:52:41 DEBUG "result": [ 12:52:41 DEBUG { 12:52:41 DEBUG "metric": { 12:52:41 DEBUG "instance": "10.142.0.107:9100" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG [...] 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG 12:52:41 DEBUG "instance": "10.142.0.43:9100" 12:52:41 DEBUG }, 12:52:41 DEBUG "value": [ 12:52:41 DEBUG 1713527561, 12:52:41 DEBUG "1581211837440" 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG ] 12:52:41 DEBUG } 12:52:41 DEBUG } 12:52:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527562" 12:52:42 INFO [http_cmd]: http status code OK 12:52:42 DEBUG --- status code --- 12:52:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:42 DEBUG --- http response --- 12:52:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:52:42 DEBUG ----- output ----- 12:52:42 DEBUG { 12:52:42 DEBUG "status": "success", 12:52:42 DEBUG "data": { 12:52:42 DEBUG "resultType": "vector", 12:52:42 DEBUG "result": [ 12:52:42 DEBUG { 12:52:42 DEBUG "metric": { 12:52:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:52:42 DEBUG }, 12:52:42 DEBUG "value": [ 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG [...] 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:52:42 DEBUG }, 12:52:42 DEBUG "value": [ 12:52:42 DEBUG 1713527562, 12:52:42 DEBUG "0" 12:52:42 DEBUG ] 12:52:42 DEBUG } 12:52:42 DEBUG ] 12:52:42 DEBUG } 12:52:42 DEBUG } 12:52:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527562" 12:52:42 INFO [http_cmd]: http status code OK 12:52:42 DEBUG --- status code --- 12:52:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:42 DEBUG --- http response --- 12:52:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:42 DEBUG ----- output ----- 12:52:42 DEBUG { 12:52:42 DEBUG "status": "success", 12:52:42 DEBUG "data": { 12:52:42 DEBUG "resultType": "vector", 12:52:42 DEBUG "result": [ 12:52:42 DEBUG { 12:52:42 DEBUG "metric": { 12:52:42 DEBUG "instance": "10.142.0.107:9100" 12:52:42 DEBUG }, 12:52:42 DEBUG "value": [ 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG [...] 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG "instance": "10.142.0.43:9100" 12:52:42 DEBUG }, 12:52:42 DEBUG "value": [ 12:52:42 DEBUG 1713527562, 12:52:42 DEBUG "16386355" 12:52:42 DEBUG ] 12:52:42 DEBUG } 12:52:42 DEBUG ] 12:52:42 DEBUG } 12:52:42 DEBUG } 12:52:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527562" 12:52:42 INFO [http_cmd]: http status code OK 12:52:42 DEBUG --- status code --- 12:52:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:52:42 DEBUG --- http response --- 12:52:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:52:42 DEBUG ----- output ----- 12:52:42 DEBUG { 12:52:42 DEBUG "status": "success", 12:52:42 DEBUG "data": { 12:52:42 DEBUG "resultType": "vector", 12:52:42 DEBUG "result": [ 12:52:42 DEBUG { 12:52:42 DEBUG "metric": { 12:52:42 DEBUG "instance": "10.142.0.107:9100" 12:52:42 DEBUG }, 12:52:42 DEBUG "value": [ 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG [...] 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG 12:52:42 DEBUG "instance": "10.142.0.43:9100" 12:52:42 DEBUG }, 12:52:42 DEBUG "value": [ 12:52:42 DEBUG 1713527562, 12:52:42 DEBUG "16391794.066666666" 12:52:42 DEBUG ] 12:52:42 DEBUG } 12:52:42 DEBUG ] 12:52:42 DEBUG } 12:52:42 DEBUG } 12:52:55 INFO 12:52:55 INFO [loop_until]: kubectl --namespace=xlou top pods 12:52:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:52:55 INFO [loop_until]: OK (rc = 0) 12:52:55 DEBUG --- stdout --- 12:52:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2392m 9437Mi am-6fd554f8f7-hvvkw 2346m 9006Mi am-6fd554f8f7-pmj4b 2506m 9132Mi ds-cts-0 10109m 13374Mi ds-cts-1 10044m 13227Mi ds-cts-2 10358m 13453Mi ds-idrepo-0 706m 3364Mi ds-idrepo-1 717m 3263Mi ds-idrepo-2 677m 3191Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-747d57dcb8-lclbf 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 995m 1029Mi 12:52:55 DEBUG --- stderr --- 12:52:55 DEBUG 12:53:01 INFO 12:53:01 INFO [loop_until]: kubectl --namespace=xlou top node 12:53:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:53:02 INFO [loop_until]: OK (rc = 0) 12:53:02 DEBUG --- stdout --- 12:53:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6120Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 740m 2% 6880Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2369m 7% 11775Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3241Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10752m 36% 16341Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5204Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2531m 8% 12268Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 3212Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2607m 8% 11377Mi 10% gke-xlou-cdm-ds-0f601751-j488 754m 2% 6367Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9727m 32% 18116Mi 16% gke-xlou-cdm-ds-0f601751-nj6q 798m 2% 7026Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 10659m 35% 15772Mi 14% gke-xlou-cdm-frontend-a8771548-i82n 1074m 6% 5102Mi 8% 12:53:02 DEBUG --- stderr --- 12:53:02 DEBUG 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527617" 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527617" 12:53:37 INFO [http_cmd]: http status code OK 12:53:37 DEBUG --- status code --- 12:53:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:37 DEBUG --- http response --- 12:53:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713527617" 12:53:37 INFO [http_cmd]: http status code OK 12:53:37 DEBUG --- status code --- 12:53:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:37 DEBUG --- http response --- 12:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:53:37 DEBUG ----- output ----- 12:53:37 DEBUG { 12:53:37 DEBUG "status": "success", 12:53:37 DEBUG "data": { 12:53:37 DEBUG "resultType": "vector", 12:53:37 DEBUG "result": [ 12:53:37 DEBUG { 12:53:37 DEBUG "metric": { 12:53:37 DEBUG "pod": "ds-cts-0" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG [...] 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG "pod": "ds-cts-2" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 1713527617, 12:53:37 DEBUG "0.00022794666666666664" 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG } 12:53:37 ERROR [http_cmd]: ERROR 12:53:37 ERROR --- status code --- 12:53:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:53:37 ERROR --- http response --- 12:53:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:53:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527617" 12:53:37 INFO [http_cmd]: http status code OK 12:53:37 DEBUG --- status code --- 12:53:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:37 DEBUG --- http response --- 12:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:53:37 DEBUG ----- output ----- 12:53:37 DEBUG { 12:53:37 DEBUG "status": "success", 12:53:37 DEBUG "data": { 12:53:37 DEBUG "resultType": "vector", 12:53:37 DEBUG "result": [ 12:53:37 DEBUG { 12:53:37 DEBUG "metric": { 12:53:37 DEBUG "pod": "ds-cts-0" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG [...] 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG "pod": "ds-cts-2" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 1713527617, 12:53:37 DEBUG "2.3666666666666667" 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG } 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527617" 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713527617" 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527617" 12:53:37 INFO [http_cmd]: http status code OK 12:53:37 DEBUG --- status code --- 12:53:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:37 DEBUG --- http response --- 12:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:37 DEBUG ----- output ----- 12:53:37 DEBUG { 12:53:37 DEBUG "status": "success", 12:53:37 DEBUG "data": { 12:53:37 DEBUG "resultType": "vector", 12:53:37 DEBUG "result": [ 12:53:37 DEBUG { 12:53:37 DEBUG "metric": { 12:53:37 DEBUG "pod": "ds-idrepo-0" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG [...] 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG "pod": "ds-idrepo-2" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 1713527617, 12:53:37 DEBUG "0" 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG } 12:53:37 INFO [http_cmd]: http status code OK 12:53:37 DEBUG --- status code --- 12:53:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:37 DEBUG --- http response --- 12:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:53:37 DEBUG ----- output ----- 12:53:37 DEBUG { 12:53:37 DEBUG "status": "success", 12:53:37 DEBUG "data": { 12:53:37 DEBUG "resultType": "vector", 12:53:37 DEBUG "result": [ 12:53:37 DEBUG { 12:53:37 DEBUG "metric": { 12:53:37 DEBUG "pod": "ds-idrepo-0" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG [...] 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG "pod": "ds-cts-2" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 1713527617, 12:53:37 DEBUG "0" 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG } 12:53:37 INFO [http_cmd]: http status code OK 12:53:37 DEBUG --- status code --- 12:53:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:37 DEBUG --- http response --- 12:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:37 DEBUG ----- output ----- 12:53:37 DEBUG { 12:53:37 DEBUG "status": "success", 12:53:37 DEBUG "data": { 12:53:37 DEBUG "resultType": "vector", 12:53:37 DEBUG "result": [ 12:53:37 DEBUG { 12:53:37 DEBUG "metric": { 12:53:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG [...] 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 1713527617, 12:53:37 DEBUG "2499.866666666667" 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG } 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527617" 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527617" 12:53:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713527617" 12:53:37 INFO [http_cmd]: http status code OK 12:53:37 DEBUG --- status code --- 12:53:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:37 DEBUG --- http response --- 12:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:37 DEBUG ----- output ----- 12:53:37 DEBUG { 12:53:37 DEBUG "status": "success", 12:53:37 DEBUG "data": { 12:53:37 DEBUG "resultType": "vector", 12:53:37 DEBUG "result": [ 12:53:37 DEBUG { 12:53:37 DEBUG "metric": { 12:53:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG [...] 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 1713527617, 12:53:37 DEBUG "2680966" 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG } 12:53:37 INFO [http_cmd]: http status code OK 12:53:37 DEBUG --- status code --- 12:53:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:37 DEBUG --- http response --- 12:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:53:37 DEBUG ----- output ----- 12:53:37 DEBUG { 12:53:37 DEBUG "status": "success", 12:53:37 DEBUG "data": { 12:53:37 DEBUG "resultType": "vector", 12:53:37 DEBUG "result": [ 12:53:37 DEBUG { 12:53:37 DEBUG "metric": { 12:53:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG [...] 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 1713527617, 12:53:37 DEBUG "0" 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG } 12:53:37 INFO [http_cmd]: http status code OK 12:53:37 DEBUG --- status code --- 12:53:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:37 DEBUG --- http response --- 12:53:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:37 DEBUG ----- output ----- 12:53:37 DEBUG { 12:53:37 DEBUG "status": "success", 12:53:37 DEBUG "data": { 12:53:37 DEBUG "resultType": "vector", 12:53:37 DEBUG "result": [ 12:53:37 DEBUG { 12:53:37 DEBUG "metric": { 12:53:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG [...] 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG 12:53:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:53:37 DEBUG }, 12:53:37 DEBUG "value": [ 12:53:37 DEBUG 1713527617, 12:53:37 DEBUG "1.0326384690487995" 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG ] 12:53:37 DEBUG } 12:53:37 DEBUG } 12:53:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527618" 12:53:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527618" 12:53:38 INFO [http_cmd]: http status code OK 12:53:38 DEBUG --- status code --- 12:53:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:38 DEBUG --- http response --- 12:53:38 INFO [http_cmd]: http status code OK 12:53:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:38 DEBUG --- status code --- 12:53:38 DEBUG ----- output ----- 12:53:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:38 DEBUG { 12:53:38 DEBUG --- http response --- 12:53:38 DEBUG "status": "success", 12:53:38 DEBUG "data": { 12:53:38 DEBUG "resultType": "vector", 12:53:38 DEBUG "result": [ 12:53:38 DEBUG { 12:53:38 DEBUG "metric": { 12:53:38 DEBUG "pod": "ds-idrepo-0" 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 12:53:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:53:38 DEBUG 12:53:38 DEBUG ----- output ----- 12:53:38 DEBUG 12:53:38 DEBUG { 12:53:38 DEBUG [...] 12:53:38 DEBUG "status": "success", 12:53:38 DEBUG 12:53:38 DEBUG "data": { 12:53:38 DEBUG 12:53:38 DEBUG "resultType": "vector", 12:53:38 DEBUG 12:53:38 DEBUG "result": [ 12:53:38 DEBUG "pod": "ds-idrepo-2" 12:53:38 DEBUG { 12:53:38 DEBUG }, 12:53:38 DEBUG "metric": { 12:53:38 DEBUG "value": [ 12:53:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:53:38 DEBUG 1713527618, 12:53:38 DEBUG }, 12:53:38 DEBUG "0" 12:53:38 DEBUG "value": [ 12:53:38 DEBUG ] 12:53:38 DEBUG 12:53:38 DEBUG } 12:53:38 DEBUG 12:53:38 DEBUG ] 12:53:38 DEBUG 12:53:38 DEBUG } 12:53:38 DEBUG [...] 12:53:38 DEBUG } 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 1713527618, 12:53:38 DEBUG "0" 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG } 12:53:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527618" 12:53:38 INFO [http_cmd]: http status code OK 12:53:38 DEBUG --- status code --- 12:53:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:38 DEBUG --- http response --- 12:53:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:38 DEBUG ----- output ----- 12:53:38 DEBUG { 12:53:38 DEBUG "status": "success", 12:53:38 DEBUG "data": { 12:53:38 DEBUG "resultType": "vector", 12:53:38 DEBUG "result": [ 12:53:38 DEBUG { 12:53:38 DEBUG "metric": { 12:53:38 DEBUG "instance": "10.142.0.107:9100" 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG [...] 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG "instance": "10.142.0.43:9100" 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 1713527618, 12:53:38 DEBUG "0.8553333333333358" 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG } 12:53:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527618" 12:53:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527618" 12:53:38 INFO [http_cmd]: http status code OK 12:53:38 INFO [http_cmd]: http status code OK 12:53:38 DEBUG --- status code --- 12:53:38 DEBUG --- status code --- 12:53:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:38 DEBUG --- http response --- 12:53:38 DEBUG --- http response --- 12:53:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:38 DEBUG ----- output ----- 12:53:38 DEBUG ----- output ----- 12:53:38 DEBUG { 12:53:38 DEBUG { 12:53:38 DEBUG "status": "success", 12:53:38 DEBUG "status": "success", 12:53:38 DEBUG "data": { 12:53:38 DEBUG "data": { 12:53:38 DEBUG "resultType": "vector", 12:53:38 DEBUG "resultType": "vector", 12:53:38 DEBUG "result": [ 12:53:38 DEBUG "result": [ 12:53:38 DEBUG { 12:53:38 DEBUG { 12:53:38 DEBUG "metric": { 12:53:38 DEBUG "metric": { 12:53:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:53:38 DEBUG "instance": "10.142.0.107:9100" 12:53:38 DEBUG }, 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG [...] 12:53:38 DEBUG [...] 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:53:38 DEBUG "instance": "10.142.0.43:9100" 12:53:38 DEBUG }, 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 1713527618, 12:53:38 DEBUG 1713527618, 12:53:38 DEBUG "0" 12:53:38 DEBUG "0.6844333333332931" 12:53:38 DEBUG ] 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG } 12:53:38 DEBUG ] 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG } 12:53:38 DEBUG } 12:53:38 DEBUG } 12:53:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527618" 12:53:38 INFO [http_cmd]: http status code OK 12:53:38 DEBUG --- status code --- 12:53:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:38 DEBUG --- http response --- 12:53:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:38 DEBUG ----- output ----- 12:53:38 DEBUG { 12:53:38 DEBUG "status": "success", 12:53:38 DEBUG "data": { 12:53:38 DEBUG "resultType": "vector", 12:53:38 DEBUG "result": [ 12:53:38 DEBUG { 12:53:38 DEBUG "metric": { 12:53:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG [...] 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 1713527618, 12:53:38 DEBUG "0" 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG } 12:53:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527618" 12:53:38 INFO [http_cmd]: http status code OK 12:53:38 DEBUG --- status code --- 12:53:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:38 DEBUG --- http response --- 12:53:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:38 DEBUG ----- output ----- 12:53:38 DEBUG { 12:53:38 DEBUG "status": "success", 12:53:38 DEBUG "data": { 12:53:38 DEBUG "resultType": "vector", 12:53:38 DEBUG "result": [ 12:53:38 DEBUG { 12:53:38 DEBUG "metric": { 12:53:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG [...] 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG 12:53:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:53:38 DEBUG }, 12:53:38 DEBUG "value": [ 12:53:38 DEBUG 1713527618, 12:53:38 DEBUG "0" 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG ] 12:53:38 DEBUG } 12:53:38 DEBUG } 12:53:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713527619" 12:53:39 INFO [http_cmd]: http status code OK 12:53:39 DEBUG --- status code --- 12:53:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:39 DEBUG --- http response --- 12:53:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:39 DEBUG ----- output ----- 12:53:39 DEBUG { 12:53:39 DEBUG "status": "success", 12:53:39 DEBUG "data": { 12:53:39 DEBUG "resultType": "vector", 12:53:39 DEBUG "result": [ 12:53:39 DEBUG { 12:53:39 DEBUG "metric": { 12:53:39 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:53:39 DEBUG }, 12:53:39 DEBUG "value": [ 12:53:39 DEBUG 12:53:39 DEBUG 12:53:39 DEBUG 12:53:39 DEBUG [...] 12:53:39 DEBUG 12:53:39 DEBUG 12:53:39 DEBUG 12:53:39 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:53:39 DEBUG }, 12:53:39 DEBUG "value": [ 12:53:39 DEBUG 1713527619, 12:53:39 DEBUG "1446854656" 12:53:39 DEBUG ] 12:53:39 DEBUG } 12:53:39 DEBUG ] 12:53:39 DEBUG } 12:53:39 DEBUG } 12:53:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527621" 12:53:41 INFO [http_cmd]: http status code OK 12:53:41 DEBUG --- status code --- 12:53:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:41 DEBUG --- http response --- 12:53:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:53:41 DEBUG ----- output ----- 12:53:41 DEBUG { 12:53:41 DEBUG "status": "success", 12:53:41 DEBUG "data": { 12:53:41 DEBUG "resultType": "vector", 12:53:41 DEBUG "result": [ 12:53:41 DEBUG { 12:53:41 DEBUG "metric": { 12:53:41 DEBUG "pod": "ds-cts-0" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG [...] 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG "pod": "ds-cts-2" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 1713527621, 12:53:41 DEBUG "30" 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG } 12:53:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713527621" 12:53:41 INFO [http_cmd]: http status code OK 12:53:41 DEBUG --- status code --- 12:53:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:41 DEBUG --- http response --- 12:53:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:41 DEBUG ----- output ----- 12:53:41 DEBUG { 12:53:41 DEBUG "status": "success", 12:53:41 DEBUG "data": { 12:53:41 DEBUG "resultType": "vector", 12:53:41 DEBUG "result": [ 12:53:41 DEBUG { 12:53:41 DEBUG "metric": { 12:53:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG [...] 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 1713527621, 12:53:41 DEBUG "55.19" 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG } 12:53:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527621" 12:53:41 INFO [http_cmd]: http status code OK 12:53:41 DEBUG --- status code --- 12:53:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:41 DEBUG --- http response --- 12:53:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:41 DEBUG ----- output ----- 12:53:41 DEBUG { 12:53:41 DEBUG "status": "success", 12:53:41 DEBUG "data": { 12:53:41 DEBUG "resultType": "vector", 12:53:41 DEBUG "result": [ 12:53:41 DEBUG { 12:53:41 DEBUG "metric": { 12:53:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG [...] 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG "pod": "ds-cts-1" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 1713527621, 12:53:41 DEBUG "0" 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG } 12:53:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527621" 12:53:41 INFO [http_cmd]: http status code OK 12:53:41 DEBUG --- status code --- 12:53:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:41 DEBUG --- http response --- 12:53:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:41 DEBUG ----- output ----- 12:53:41 DEBUG { 12:53:41 DEBUG "status": "success", 12:53:41 DEBUG "data": { 12:53:41 DEBUG "resultType": "vector", 12:53:41 DEBUG "result": [ 12:53:41 DEBUG { 12:53:41 DEBUG "metric": { 12:53:41 DEBUG "pod": "ds-idrepo-0" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG [...] 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG "pod": "ds-idrepo-2" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 1713527621, 12:53:41 DEBUG "0" 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG } 12:53:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527621" 12:53:41 INFO [http_cmd]: http status code OK 12:53:41 DEBUG --- status code --- 12:53:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:41 DEBUG --- http response --- 12:53:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:41 DEBUG ----- output ----- 12:53:41 DEBUG { 12:53:41 DEBUG "status": "success", 12:53:41 DEBUG "data": { 12:53:41 DEBUG "resultType": "vector", 12:53:41 DEBUG "result": [ 12:53:41 DEBUG { 12:53:41 DEBUG "metric": { 12:53:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG [...] 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG "pod": "ds-cts-1" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 1713527621, 12:53:41 DEBUG "0" 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG } 12:53:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527621" 12:53:41 INFO [http_cmd]: http status code OK 12:53:41 DEBUG --- status code --- 12:53:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:41 DEBUG --- http response --- 12:53:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:41 DEBUG ----- output ----- 12:53:41 DEBUG { 12:53:41 DEBUG "status": "success", 12:53:41 DEBUG "data": { 12:53:41 DEBUG "resultType": "vector", 12:53:41 DEBUG "result": [ 12:53:41 DEBUG { 12:53:41 DEBUG "metric": { 12:53:41 DEBUG "instance": "10.142.0.107:9100" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG [...] 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG 12:53:41 DEBUG "instance": "10.142.0.43:9100" 12:53:41 DEBUG }, 12:53:41 DEBUG "value": [ 12:53:41 DEBUG 1713527621, 12:53:41 DEBUG "1583114048512" 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG ] 12:53:41 DEBUG } 12:53:41 DEBUG } 12:53:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527622" 12:53:42 INFO [http_cmd]: http status code OK 12:53:42 DEBUG --- status code --- 12:53:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:42 DEBUG --- http response --- 12:53:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:53:42 DEBUG ----- output ----- 12:53:42 DEBUG { 12:53:42 DEBUG "status": "success", 12:53:42 DEBUG "data": { 12:53:42 DEBUG "resultType": "vector", 12:53:42 DEBUG "result": [ 12:53:42 DEBUG { 12:53:42 DEBUG "metric": { 12:53:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:53:42 DEBUG }, 12:53:42 DEBUG "value": [ 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG [...] 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:53:42 DEBUG }, 12:53:42 DEBUG "value": [ 12:53:42 DEBUG 1713527622, 12:53:42 DEBUG "0" 12:53:42 DEBUG ] 12:53:42 DEBUG } 12:53:42 DEBUG ] 12:53:42 DEBUG } 12:53:42 DEBUG } 12:53:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527622" 12:53:42 INFO [http_cmd]: http status code OK 12:53:42 DEBUG --- status code --- 12:53:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:42 DEBUG --- http response --- 12:53:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:42 DEBUG ----- output ----- 12:53:42 DEBUG { 12:53:42 DEBUG "status": "success", 12:53:42 DEBUG "data": { 12:53:42 DEBUG "resultType": "vector", 12:53:42 DEBUG "result": [ 12:53:42 DEBUG { 12:53:42 DEBUG "metric": { 12:53:42 DEBUG "instance": "10.142.0.107:9100" 12:53:42 DEBUG }, 12:53:42 DEBUG "value": [ 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG [...] 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG "instance": "10.142.0.43:9100" 12:53:42 DEBUG }, 12:53:42 DEBUG "value": [ 12:53:42 DEBUG 1713527622, 12:53:42 DEBUG "16290096.066666666" 12:53:42 DEBUG ] 12:53:42 DEBUG } 12:53:42 DEBUG ] 12:53:42 DEBUG } 12:53:42 DEBUG } 12:53:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527622" 12:53:42 INFO [http_cmd]: http status code OK 12:53:42 DEBUG --- status code --- 12:53:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:53:42 DEBUG --- http response --- 12:53:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:53:42 DEBUG ----- output ----- 12:53:42 DEBUG { 12:53:42 DEBUG "status": "success", 12:53:42 DEBUG "data": { 12:53:42 DEBUG "resultType": "vector", 12:53:42 DEBUG "result": [ 12:53:42 DEBUG { 12:53:42 DEBUG "metric": { 12:53:42 DEBUG "instance": "10.142.0.107:9100" 12:53:42 DEBUG }, 12:53:42 DEBUG "value": [ 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG [...] 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG 12:53:42 DEBUG "instance": "10.142.0.43:9100" 12:53:42 DEBUG }, 12:53:42 DEBUG "value": [ 12:53:42 DEBUG 1713527622, 12:53:42 DEBUG "16297513.166666666" 12:53:42 DEBUG ] 12:53:42 DEBUG } 12:53:42 DEBUG ] 12:53:42 DEBUG } 12:53:42 DEBUG } 12:53:55 INFO 12:53:55 INFO [loop_until]: kubectl --namespace=xlou top pods 12:53:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:53:55 INFO [loop_until]: OK (rc = 0) 12:53:55 DEBUG --- stdout --- 12:53:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2345m 9434Mi am-6fd554f8f7-hvvkw 2423m 9014Mi am-6fd554f8f7-pmj4b 2533m 9133Mi ds-cts-0 10224m 13597Mi ds-cts-1 9295m 13254Mi ds-cts-2 10510m 13464Mi ds-idrepo-0 709m 3365Mi ds-idrepo-1 677m 3262Mi ds-idrepo-2 709m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 4m 1216Mi lodemon-747d57dcb8-lclbf 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1017m 1392Mi 12:53:55 DEBUG --- stderr --- 12:53:55 DEBUG 12:54:02 INFO 12:54:02 INFO [loop_until]: kubectl --namespace=xlou top node 12:54:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:54:02 INFO [loop_until]: OK (rc = 0) 12:54:02 DEBUG --- stdout --- 12:54:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6135Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 758m 2% 6878Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2386m 7% 11774Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3224Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10179m 34% 16424Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5181Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2528m 8% 12268Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 3174Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2698m 9% 11376Mi 10% gke-xlou-cdm-ds-0f601751-j488 722m 2% 6374Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9919m 33% 17825Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 791m 2% 7030Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 10256m 34% 16157Mi 14% gke-xlou-cdm-frontend-a8771548-i82n 1074m 6% 5440Mi 9% 12:54:02 DEBUG --- stderr --- 12:54:02 DEBUG 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527677" 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527677" 12:54:37 INFO [http_cmd]: http status code OK 12:54:37 INFO [http_cmd]: http status code OK 12:54:37 DEBUG --- status code --- 12:54:37 DEBUG --- status code --- 12:54:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:37 DEBUG --- http response --- 12:54:37 DEBUG --- http response --- 12:54:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:54:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:54:37 DEBUG ----- output ----- 12:54:37 DEBUG { 12:54:37 DEBUG "status": "success", 12:54:37 DEBUG "data": { 12:54:37 DEBUG "resultType": "vector", 12:54:37 DEBUG "result": [ 12:54:37 DEBUG { 12:54:37 DEBUG "metric": { 12:54:37 DEBUG "pod": "ds-cts-0" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG [...] 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG "pod": "ds-cts-2" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 1713527677, 12:54:37 DEBUG "0.0001" 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG } 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713527677" 12:54:37 ERROR [http_cmd]: ERROR 12:54:37 ERROR --- status code --- 12:54:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:54:37 ERROR --- http response --- 12:54:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:54:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527677" 12:54:37 INFO [http_cmd]: http status code OK 12:54:37 DEBUG --- status code --- 12:54:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:37 DEBUG --- http response --- 12:54:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:54:37 DEBUG ----- output ----- 12:54:37 DEBUG { 12:54:37 DEBUG "status": "success", 12:54:37 DEBUG "data": { 12:54:37 DEBUG "resultType": "vector", 12:54:37 DEBUG "result": [ 12:54:37 DEBUG { 12:54:37 DEBUG "metric": { 12:54:37 DEBUG "pod": "ds-cts-0" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG [...] 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG "pod": "ds-cts-2" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 1713527677, 12:54:37 DEBUG "85.8" 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG } 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527677" 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713527677" 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527677" 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527677" 12:54:37 INFO [http_cmd]: http status code OK 12:54:37 DEBUG --- status code --- 12:54:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:37 DEBUG --- http response --- 12:54:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:37 DEBUG ----- output ----- 12:54:37 DEBUG { 12:54:37 DEBUG "status": "success", 12:54:37 DEBUG "data": { 12:54:37 DEBUG "resultType": "vector", 12:54:37 DEBUG "result": [ 12:54:37 DEBUG { 12:54:37 DEBUG "metric": { 12:54:37 DEBUG "pod": "ds-idrepo-0" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG [...] 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG "pod": "ds-idrepo-2" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 1713527677, 12:54:37 DEBUG "0" 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG } 12:54:37 INFO [http_cmd]: http status code OK 12:54:37 DEBUG --- status code --- 12:54:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:37 DEBUG --- http response --- 12:54:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:54:37 DEBUG ----- output ----- 12:54:37 DEBUG { 12:54:37 DEBUG "status": "success", 12:54:37 DEBUG "data": { 12:54:37 DEBUG "resultType": "vector", 12:54:37 DEBUG "result": [ 12:54:37 DEBUG { 12:54:37 DEBUG "metric": { 12:54:37 DEBUG "pod": "ds-idrepo-0" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG [...] 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG "pod": "ds-cts-2" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 1713527677, 12:54:37 DEBUG "0" 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG } 12:54:37 INFO [http_cmd]: http status code OK 12:54:37 DEBUG --- status code --- 12:54:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:37 DEBUG --- http response --- 12:54:37 INFO [http_cmd]: http status code OK 12:54:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:37 DEBUG --- status code --- 12:54:37 DEBUG ----- output ----- 12:54:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:37 DEBUG { 12:54:37 DEBUG --- http response --- 12:54:37 DEBUG "status": "success", 12:54:37 DEBUG "data": { 12:54:37 DEBUG "resultType": "vector", 12:54:37 DEBUG "result": [ 12:54:37 DEBUG { 12:54:37 DEBUG "metric": { 12:54:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG [...] 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 1713527677, 12:54:37 DEBUG "2497.9666666666667" 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG } 12:54:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:54:37 DEBUG ----- output ----- 12:54:37 DEBUG { 12:54:37 DEBUG "status": "success", 12:54:37 DEBUG "data": { 12:54:37 DEBUG "resultType": "vector", 12:54:37 DEBUG "result": [ 12:54:37 DEBUG { 12:54:37 DEBUG "metric": { 12:54:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG [...] 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG "pod": "idm-b85f9d978-txpwv" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 1713527677, 12:54:37 DEBUG "413.3141820819732" 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG } 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527677" 12:54:37 INFO [http_cmd]: http status code OK 12:54:37 DEBUG --- status code --- 12:54:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:37 DEBUG --- http response --- 12:54:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:37 DEBUG ----- output ----- 12:54:37 DEBUG { 12:54:37 DEBUG "status": "success", 12:54:37 DEBUG "data": { 12:54:37 DEBUG "resultType": "vector", 12:54:37 DEBUG "result": [ 12:54:37 DEBUG { 12:54:37 DEBUG "metric": { 12:54:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG [...] 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 1713527677, 12:54:37 DEBUG "2530933.5" 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG } 12:54:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713527677" 12:54:37 INFO [http_cmd]: http status code OK 12:54:37 DEBUG --- status code --- 12:54:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:37 DEBUG --- http response --- 12:54:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:37 DEBUG ----- output ----- 12:54:37 DEBUG { 12:54:37 DEBUG "status": "success", 12:54:37 DEBUG "data": { 12:54:37 DEBUG "resultType": "vector", 12:54:37 DEBUG "result": [ 12:54:37 DEBUG { 12:54:37 DEBUG "metric": { 12:54:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG [...] 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG 12:54:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:54:37 DEBUG }, 12:54:37 DEBUG "value": [ 12:54:37 DEBUG 1713527677, 12:54:37 DEBUG "1.027841584262288" 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG ] 12:54:37 DEBUG } 12:54:37 DEBUG } 12:54:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527678" 12:54:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527678" 12:54:38 INFO [http_cmd]: http status code OK 12:54:38 INFO [http_cmd]: http status code OK 12:54:38 DEBUG --- status code --- 12:54:38 DEBUG --- status code --- 12:54:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:38 DEBUG --- http response --- 12:54:38 DEBUG --- http response --- 12:54:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 12:54:38 DEBUG ----- output ----- 12:54:38 DEBUG ----- output ----- 12:54:38 DEBUG { 12:54:38 DEBUG { 12:54:38 DEBUG "status": "success", 12:54:38 DEBUG "status": "success", 12:54:38 DEBUG "data": { 12:54:38 DEBUG "data": { 12:54:38 DEBUG "resultType": "vector", 12:54:38 DEBUG "resultType": "vector", 12:54:38 DEBUG "result": [ 12:54:38 DEBUG "result": [ 12:54:38 DEBUG { 12:54:38 DEBUG { 12:54:38 DEBUG "metric": { 12:54:38 DEBUG "pod": "ds-idrepo-0" 12:54:38 DEBUG "metric": { 12:54:38 DEBUG }, 12:54:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:54:38 DEBUG "value": [ 12:54:38 DEBUG }, 12:54:38 DEBUG 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG [...] 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG [...] 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG "pod": "ds-idrepo-2" 12:54:38 DEBUG 12:54:38 DEBUG }, 12:54:38 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 12:54:38 DEBUG "value": [ 12:54:38 DEBUG }, 12:54:38 DEBUG 1713527678, 12:54:38 DEBUG "0" 12:54:38 DEBUG "value": [ 12:54:38 DEBUG ] 12:54:38 DEBUG 1713527678, 12:54:38 DEBUG } 12:54:38 DEBUG "15150986.105652526" 12:54:38 DEBUG ] 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG } 12:54:38 DEBUG } 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG } 12:54:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527678" 12:54:38 INFO [http_cmd]: http status code OK 12:54:38 DEBUG --- status code --- 12:54:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:38 DEBUG --- http response --- 12:54:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:38 DEBUG ----- output ----- 12:54:38 DEBUG { 12:54:38 DEBUG "status": "success", 12:54:38 DEBUG "data": { 12:54:38 DEBUG "resultType": "vector", 12:54:38 DEBUG "result": [ 12:54:38 DEBUG { 12:54:38 DEBUG "metric": { 12:54:38 DEBUG "instance": "10.142.0.107:9100" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG [...] 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG "instance": "10.142.0.43:9100" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 1713527678, 12:54:38 DEBUG "1.155666666666665" 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG } 12:54:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527678" 12:54:38 INFO [http_cmd]: http status code OK 12:54:38 DEBUG --- status code --- 12:54:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:38 DEBUG --- http response --- 12:54:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:38 DEBUG ----- output ----- 12:54:38 DEBUG { 12:54:38 DEBUG "status": "success", 12:54:38 DEBUG "data": { 12:54:38 DEBUG "resultType": "vector", 12:54:38 DEBUG "result": [ 12:54:38 DEBUG { 12:54:38 DEBUG "metric": { 12:54:38 DEBUG "instance": "10.142.0.107:9100" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG [...] 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG "instance": "10.142.0.43:9100" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 1713527678, 12:54:38 DEBUG "0.7144333333333226" 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG } 12:54:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527678" 12:54:38 INFO [http_cmd]: http status code OK 12:54:38 DEBUG --- status code --- 12:54:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:38 DEBUG --- http response --- 12:54:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:38 DEBUG ----- output ----- 12:54:38 DEBUG { 12:54:38 DEBUG "status": "success", 12:54:38 DEBUG "data": { 12:54:38 DEBUG "resultType": "vector", 12:54:38 DEBUG "result": [ 12:54:38 DEBUG { 12:54:38 DEBUG "metric": { 12:54:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG [...] 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 1713527678, 12:54:38 DEBUG "0" 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG } 12:54:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527678" 12:54:38 INFO [http_cmd]: http status code OK 12:54:38 DEBUG --- status code --- 12:54:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:38 DEBUG --- http response --- 12:54:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:38 DEBUG ----- output ----- 12:54:38 DEBUG { 12:54:38 DEBUG "status": "success", 12:54:38 DEBUG "data": { 12:54:38 DEBUG "resultType": "vector", 12:54:38 DEBUG "result": [ 12:54:38 DEBUG { 12:54:38 DEBUG "metric": { 12:54:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG [...] 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 1713527678, 12:54:38 DEBUG "0" 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG } 12:54:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527678" 12:54:38 INFO [http_cmd]: http status code OK 12:54:38 DEBUG --- status code --- 12:54:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:38 DEBUG --- http response --- 12:54:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:38 DEBUG ----- output ----- 12:54:38 DEBUG { 12:54:38 DEBUG "status": "success", 12:54:38 DEBUG "data": { 12:54:38 DEBUG "resultType": "vector", 12:54:38 DEBUG "result": [ 12:54:38 DEBUG { 12:54:38 DEBUG "metric": { 12:54:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG [...] 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG 12:54:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:54:38 DEBUG }, 12:54:38 DEBUG "value": [ 12:54:38 DEBUG 1713527678, 12:54:38 DEBUG "0" 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG ] 12:54:38 DEBUG } 12:54:38 DEBUG } 12:54:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713527679" 12:54:39 INFO [http_cmd]: http status code OK 12:54:39 DEBUG --- status code --- 12:54:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:39 DEBUG --- http response --- 12:54:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:39 DEBUG ----- output ----- 12:54:39 DEBUG { 12:54:39 DEBUG "status": "success", 12:54:39 DEBUG "data": { 12:54:39 DEBUG "resultType": "vector", 12:54:39 DEBUG "result": [ 12:54:39 DEBUG { 12:54:39 DEBUG "metric": { 12:54:39 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:54:39 DEBUG }, 12:54:39 DEBUG "value": [ 12:54:39 DEBUG 12:54:39 DEBUG 12:54:39 DEBUG 12:54:39 DEBUG [...] 12:54:39 DEBUG 12:54:39 DEBUG 12:54:39 DEBUG 12:54:39 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:54:39 DEBUG }, 12:54:39 DEBUG "value": [ 12:54:39 DEBUG 1713527679, 12:54:39 DEBUG "1790468096" 12:54:39 DEBUG ] 12:54:39 DEBUG } 12:54:39 DEBUG ] 12:54:39 DEBUG } 12:54:39 DEBUG } 12:54:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527681" 12:54:41 INFO [http_cmd]: http status code OK 12:54:41 DEBUG --- status code --- 12:54:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:41 DEBUG --- http response --- 12:54:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:54:41 DEBUG ----- output ----- 12:54:41 DEBUG { 12:54:41 DEBUG "status": "success", 12:54:41 DEBUG "data": { 12:54:41 DEBUG "resultType": "vector", 12:54:41 DEBUG "result": [ 12:54:41 DEBUG { 12:54:41 DEBUG "metric": { 12:54:41 DEBUG "pod": "ds-cts-0" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG [...] 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG "pod": "ds-cts-2" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 1713527681, 12:54:41 DEBUG "30" 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG } 12:54:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713527681" 12:54:41 INFO [http_cmd]: http status code OK 12:54:41 DEBUG --- status code --- 12:54:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:41 DEBUG --- http response --- 12:54:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:41 DEBUG ----- output ----- 12:54:41 DEBUG { 12:54:41 DEBUG "status": "success", 12:54:41 DEBUG "data": { 12:54:41 DEBUG "resultType": "vector", 12:54:41 DEBUG "result": [ 12:54:41 DEBUG { 12:54:41 DEBUG "metric": { 12:54:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG [...] 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 1713527681, 12:54:41 DEBUG "55.19" 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG } 12:54:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527681" 12:54:41 INFO [http_cmd]: http status code OK 12:54:41 DEBUG --- status code --- 12:54:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:41 DEBUG --- http response --- 12:54:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:54:41 DEBUG ----- output ----- 12:54:41 DEBUG { 12:54:41 DEBUG "status": "success", 12:54:41 DEBUG "data": { 12:54:41 DEBUG "resultType": "vector", 12:54:41 DEBUG "result": [ 12:54:41 DEBUG { 12:54:41 DEBUG "metric": { 12:54:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG [...] 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 1713527681, 12:54:41 DEBUG "0.9440452798379175" 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG } 12:54:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527681" 12:54:41 INFO [http_cmd]: http status code OK 12:54:41 DEBUG --- status code --- 12:54:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:41 DEBUG --- http response --- 12:54:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:41 DEBUG ----- output ----- 12:54:41 DEBUG { 12:54:41 DEBUG "status": "success", 12:54:41 DEBUG "data": { 12:54:41 DEBUG "resultType": "vector", 12:54:41 DEBUG "result": [ 12:54:41 DEBUG { 12:54:41 DEBUG "metric": { 12:54:41 DEBUG "pod": "ds-idrepo-0" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG [...] 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG "pod": "ds-idrepo-2" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 1713527681, 12:54:41 DEBUG "0" 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG } 12:54:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527681" 12:54:41 INFO [http_cmd]: http status code OK 12:54:41 DEBUG --- status code --- 12:54:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:41 DEBUG --- http response --- 12:54:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:54:41 DEBUG ----- output ----- 12:54:41 DEBUG { 12:54:41 DEBUG "status": "success", 12:54:41 DEBUG "data": { 12:54:41 DEBUG "resultType": "vector", 12:54:41 DEBUG "result": [ 12:54:41 DEBUG { 12:54:41 DEBUG "metric": { 12:54:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG [...] 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG 12:54:41 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 12:54:41 DEBUG }, 12:54:41 DEBUG "value": [ 12:54:41 DEBUG 1713527681, 12:54:41 DEBUG "52859650.40147981" 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG ] 12:54:41 DEBUG } 12:54:41 DEBUG } 12:54:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527682" 12:54:42 INFO [http_cmd]: http status code OK 12:54:42 DEBUG --- status code --- 12:54:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:42 DEBUG --- http response --- 12:54:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:42 DEBUG ----- output ----- 12:54:42 DEBUG { 12:54:42 DEBUG "status": "success", 12:54:42 DEBUG "data": { 12:54:42 DEBUG "resultType": "vector", 12:54:42 DEBUG "result": [ 12:54:42 DEBUG { 12:54:42 DEBUG "metric": { 12:54:42 DEBUG "instance": "10.142.0.107:9100" 12:54:42 DEBUG }, 12:54:42 DEBUG "value": [ 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG [...] 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG "instance": "10.142.0.43:9100" 12:54:42 DEBUG }, 12:54:42 DEBUG "value": [ 12:54:42 DEBUG 1713527682, 12:54:42 DEBUG "1585253964800" 12:54:42 DEBUG ] 12:54:42 DEBUG } 12:54:42 DEBUG ] 12:54:42 DEBUG } 12:54:42 DEBUG } 12:54:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527682" 12:54:42 INFO [http_cmd]: http status code OK 12:54:42 DEBUG --- status code --- 12:54:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:42 DEBUG --- http response --- 12:54:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:54:42 DEBUG ----- output ----- 12:54:42 DEBUG { 12:54:42 DEBUG "status": "success", 12:54:42 DEBUG "data": { 12:54:42 DEBUG "resultType": "vector", 12:54:42 DEBUG "result": [ 12:54:42 DEBUG { 12:54:42 DEBUG "metric": { 12:54:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:54:42 DEBUG }, 12:54:42 DEBUG "value": [ 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG [...] 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:54:42 DEBUG }, 12:54:42 DEBUG "value": [ 12:54:42 DEBUG 1713527682, 12:54:42 DEBUG "0" 12:54:42 DEBUG ] 12:54:42 DEBUG } 12:54:42 DEBUG ] 12:54:42 DEBUG } 12:54:42 DEBUG } 12:54:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527682" 12:54:42 INFO [http_cmd]: http status code OK 12:54:42 DEBUG --- status code --- 12:54:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:42 DEBUG --- http response --- 12:54:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:42 DEBUG ----- output ----- 12:54:42 DEBUG { 12:54:42 DEBUG "status": "success", 12:54:42 DEBUG "data": { 12:54:42 DEBUG "resultType": "vector", 12:54:42 DEBUG "result": [ 12:54:42 DEBUG { 12:54:42 DEBUG "metric": { 12:54:42 DEBUG "instance": "10.142.0.107:9100" 12:54:42 DEBUG }, 12:54:42 DEBUG "value": [ 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG [...] 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG "instance": "10.142.0.43:9100" 12:54:42 DEBUG }, 12:54:42 DEBUG "value": [ 12:54:42 DEBUG 1713527682, 12:54:42 DEBUG "16208565.933333334" 12:54:42 DEBUG ] 12:54:42 DEBUG } 12:54:42 DEBUG ] 12:54:42 DEBUG } 12:54:42 DEBUG } 12:54:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527682" 12:54:42 INFO [http_cmd]: http status code OK 12:54:42 DEBUG --- status code --- 12:54:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:54:42 DEBUG --- http response --- 12:54:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:54:42 DEBUG ----- output ----- 12:54:42 DEBUG { 12:54:42 DEBUG "status": "success", 12:54:42 DEBUG "data": { 12:54:42 DEBUG "resultType": "vector", 12:54:42 DEBUG "result": [ 12:54:42 DEBUG { 12:54:42 DEBUG "metric": { 12:54:42 DEBUG "instance": "10.142.0.107:9100" 12:54:42 DEBUG }, 12:54:42 DEBUG "value": [ 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG [...] 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG 12:54:42 DEBUG "instance": "10.142.0.43:9100" 12:54:42 DEBUG }, 12:54:42 DEBUG "value": [ 12:54:42 DEBUG 1713527682, 12:54:42 DEBUG "16215423.2" 12:54:42 DEBUG ] 12:54:42 DEBUG } 12:54:42 DEBUG ] 12:54:42 DEBUG } 12:54:42 DEBUG } 12:54:55 INFO 12:54:55 INFO [loop_until]: kubectl --namespace=xlou top pods 12:54:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:54:55 INFO [loop_until]: OK (rc = 0) 12:54:55 DEBUG --- stdout --- 12:54:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2364m 9437Mi am-6fd554f8f7-hvvkw 2347m 9012Mi am-6fd554f8f7-pmj4b 2476m 9133Mi ds-cts-0 10171m 13231Mi ds-cts-1 9788m 13233Mi ds-cts-2 10114m 13179Mi ds-idrepo-0 665m 3364Mi ds-idrepo-1 677m 3262Mi ds-idrepo-2 701m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-747d57dcb8-lclbf 9m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 298m 1366Mi 12:54:55 DEBUG --- stderr --- 12:54:55 DEBUG 12:55:02 INFO 12:55:02 INFO [loop_until]: kubectl --namespace=xlou top node 12:55:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:55:02 INFO [loop_until]: OK (rc = 0) 12:55:02 DEBUG --- stdout --- 12:55:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6128Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 737m 2% 6892Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2398m 8% 11787Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 3219Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10588m 35% 16473Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5183Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2548m 8% 12267Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 3166Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2665m 8% 11379Mi 10% gke-xlou-cdm-ds-0f601751-j488 748m 2% 6370Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9316m 31% 17958Mi 16% gke-xlou-cdm-ds-0f601751-nj6q 787m 2% 7017Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 10173m 34% 15643Mi 14% gke-xlou-cdm-frontend-a8771548-i82n 1133m 7% 5518Mi 9% 12:55:02 DEBUG --- stderr --- 12:55:02 DEBUG 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527737" 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527737" 12:55:37 INFO [http_cmd]: http status code OK 12:55:37 INFO [http_cmd]: http status code OK 12:55:37 DEBUG --- status code --- 12:55:37 DEBUG --- status code --- 12:55:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:37 DEBUG --- http response --- 12:55:37 DEBUG --- http response --- 12:55:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:55:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:55:37 DEBUG ----- output ----- 12:55:37 DEBUG { 12:55:37 DEBUG "status": "success", 12:55:37 DEBUG "data": { 12:55:37 DEBUG "resultType": "vector", 12:55:37 DEBUG "result": [ 12:55:37 DEBUG { 12:55:37 DEBUG "metric": { 12:55:37 DEBUG "pod": "ds-cts-0" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG [...] 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG "pod": "ds-cts-2" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 1713527737, 12:55:37 DEBUG "0.00016666666666666666" 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG } 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713527737" 12:55:37 ERROR [http_cmd]: ERROR 12:55:37 ERROR --- status code --- 12:55:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:55:37 ERROR --- http response --- 12:55:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:55:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527737" 12:55:37 INFO [http_cmd]: http status code OK 12:55:37 DEBUG --- status code --- 12:55:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:37 DEBUG --- http response --- 12:55:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:55:37 DEBUG ----- output ----- 12:55:37 DEBUG { 12:55:37 DEBUG "status": "success", 12:55:37 DEBUG "data": { 12:55:37 DEBUG "resultType": "vector", 12:55:37 DEBUG "result": [ 12:55:37 DEBUG { 12:55:37 DEBUG "metric": { 12:55:37 DEBUG "pod": "ds-cts-0" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG [...] 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG "pod": "ds-cts-2" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 1713527737, 12:55:37 DEBUG "32.93333333333333" 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG } 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527737" 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713527737" 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527737" 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527737" 12:55:37 INFO [http_cmd]: http status code OK 12:55:37 DEBUG --- status code --- 12:55:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:37 DEBUG --- http response --- 12:55:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:55:37 DEBUG ----- output ----- 12:55:37 DEBUG { 12:55:37 DEBUG "status": "success", 12:55:37 DEBUG "data": { 12:55:37 DEBUG "resultType": "vector", 12:55:37 DEBUG "result": [ 12:55:37 DEBUG { 12:55:37 DEBUG "metric": { 12:55:37 DEBUG "pod": "ds-idrepo-0" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG [...] 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG "pod": "ds-cts-2" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 1713527737, 12:55:37 DEBUG "0" 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG } 12:55:37 INFO [http_cmd]: http status code OK 12:55:37 DEBUG --- status code --- 12:55:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:37 DEBUG --- http response --- 12:55:37 INFO [http_cmd]: http status code OK 12:55:37 INFO [http_cmd]: http status code OK 12:55:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:37 DEBUG --- status code --- 12:55:37 DEBUG --- status code --- 12:55:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:37 DEBUG --- http response --- 12:55:37 DEBUG ----- output ----- 12:55:37 DEBUG --- http response --- 12:55:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:37 DEBUG { 12:55:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:55:37 DEBUG ----- output ----- 12:55:37 DEBUG "status": "success", 12:55:37 DEBUG ----- output ----- 12:55:37 DEBUG { 12:55:37 DEBUG "data": { 12:55:37 DEBUG { 12:55:37 DEBUG "status": "success", 12:55:37 DEBUG "resultType": "vector", 12:55:37 DEBUG "status": "success", 12:55:37 DEBUG "data": { 12:55:37 DEBUG "result": [ 12:55:37 DEBUG "data": { 12:55:37 DEBUG "resultType": "vector", 12:55:37 DEBUG { 12:55:37 DEBUG "resultType": "vector", 12:55:37 DEBUG "result": [ 12:55:37 DEBUG "metric": { 12:55:37 DEBUG "result": [ 12:55:37 DEBUG { 12:55:37 DEBUG "pod": "ds-idrepo-0" 12:55:37 DEBUG { 12:55:37 DEBUG "metric": { 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:55:37 DEBUG 12:55:37 DEBUG }, 12:55:37 DEBUG "metric": { 12:55:37 DEBUG 12:55:37 DEBUG "value": [ 12:55:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG }, 12:55:37 DEBUG [...] 12:55:37 DEBUG 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG [...] 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG "pod": "ds-idrepo-2" 12:55:37 DEBUG 12:55:37 DEBUG [...] 12:55:37 DEBUG }, 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG "value": [ 12:55:37 DEBUG "pod": "idm-b85f9d978-wnhxb" 12:55:37 DEBUG 12:55:37 DEBUG 1713527737, 12:55:37 DEBUG }, 12:55:37 DEBUG 12:55:37 DEBUG "0" 12:55:37 DEBUG "value": [ 12:55:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:55:37 DEBUG ] 12:55:37 DEBUG 1713527737, 12:55:37 DEBUG }, 12:55:37 DEBUG } 12:55:37 DEBUG "0" 12:55:37 DEBUG "value": [ 12:55:37 DEBUG ] 12:55:37 DEBUG ] 12:55:37 DEBUG 1713527737, 12:55:37 DEBUG } 12:55:37 DEBUG } 12:55:37 DEBUG "2494.1" 12:55:37 DEBUG } 12:55:37 DEBUG ] 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG } 12:55:37 DEBUG } 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG } 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527737" 12:55:37 INFO [http_cmd]: http status code OK 12:55:37 DEBUG --- status code --- 12:55:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:37 DEBUG --- http response --- 12:55:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:37 DEBUG ----- output ----- 12:55:37 DEBUG { 12:55:37 DEBUG "status": "success", 12:55:37 DEBUG "data": { 12:55:37 DEBUG "resultType": "vector", 12:55:37 DEBUG "result": [ 12:55:37 DEBUG { 12:55:37 DEBUG "metric": { 12:55:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG [...] 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 1713527737, 12:55:37 DEBUG "2381342" 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG } 12:55:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713527737" 12:55:37 INFO [http_cmd]: http status code OK 12:55:37 DEBUG --- status code --- 12:55:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:37 DEBUG --- http response --- 12:55:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:37 DEBUG ----- output ----- 12:55:37 DEBUG { 12:55:37 DEBUG "status": "success", 12:55:37 DEBUG "data": { 12:55:37 DEBUG "resultType": "vector", 12:55:37 DEBUG "result": [ 12:55:37 DEBUG { 12:55:37 DEBUG "metric": { 12:55:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG [...] 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG 12:55:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:55:37 DEBUG }, 12:55:37 DEBUG "value": [ 12:55:37 DEBUG 1713527737, 12:55:37 DEBUG "1.1868943856285237" 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG ] 12:55:37 DEBUG } 12:55:37 DEBUG } 12:55:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527738" 12:55:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527738" 12:55:38 INFO [http_cmd]: http status code OK 12:55:38 DEBUG --- status code --- 12:55:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:38 DEBUG --- http response --- 12:55:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:38 DEBUG ----- output ----- 12:55:38 DEBUG { 12:55:38 DEBUG "status": "success", 12:55:38 DEBUG "data": { 12:55:38 DEBUG "resultType": "vector", 12:55:38 DEBUG "result": [ 12:55:38 DEBUG { 12:55:38 DEBUG "metric": { 12:55:38 DEBUG "pod": "ds-idrepo-0" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG [...] 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG "pod": "ds-idrepo-2" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 1713527738, 12:55:38 DEBUG "0" 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG } 12:55:38 INFO [http_cmd]: http status code OK 12:55:38 DEBUG --- status code --- 12:55:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:38 DEBUG --- http response --- 12:55:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 12:55:38 DEBUG ----- output ----- 12:55:38 DEBUG { 12:55:38 DEBUG "status": "success", 12:55:38 DEBUG "data": { 12:55:38 DEBUG "resultType": "vector", 12:55:38 DEBUG "result": [ 12:55:38 DEBUG { 12:55:38 DEBUG "metric": { 12:55:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG [...] 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG "pod": "idm-b85f9d978-wnhxb" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 1713527738, 12:55:38 DEBUG "0" 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG } 12:55:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527738" 12:55:38 INFO [http_cmd]: http status code OK 12:55:38 DEBUG --- status code --- 12:55:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:38 DEBUG --- http response --- 12:55:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:38 DEBUG ----- output ----- 12:55:38 DEBUG { 12:55:38 DEBUG "status": "success", 12:55:38 DEBUG "data": { 12:55:38 DEBUG "resultType": "vector", 12:55:38 DEBUG "result": [ 12:55:38 DEBUG { 12:55:38 DEBUG "metric": { 12:55:38 DEBUG "instance": "10.142.0.107:9100" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG [...] 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG "instance": "10.142.0.43:9100" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 1713527738, 12:55:38 DEBUG "0.9083333333333391" 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG } 12:55:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527738" 12:55:38 INFO [http_cmd]: http status code OK 12:55:38 DEBUG --- status code --- 12:55:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:38 DEBUG --- http response --- 12:55:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:38 DEBUG ----- output ----- 12:55:38 DEBUG { 12:55:38 DEBUG "status": "success", 12:55:38 DEBUG "data": { 12:55:38 DEBUG "resultType": "vector", 12:55:38 DEBUG "result": [ 12:55:38 DEBUG { 12:55:38 DEBUG "metric": { 12:55:38 DEBUG "instance": "10.142.0.107:9100" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG [...] 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG "instance": "10.142.0.43:9100" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 1713527738, 12:55:38 DEBUG "0.6613666666667314" 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG } 12:55:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527738" 12:55:38 INFO [http_cmd]: http status code OK 12:55:38 DEBUG --- status code --- 12:55:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:38 DEBUG --- http response --- 12:55:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:38 DEBUG ----- output ----- 12:55:38 DEBUG { 12:55:38 DEBUG "status": "success", 12:55:38 DEBUG "data": { 12:55:38 DEBUG "resultType": "vector", 12:55:38 DEBUG "result": [ 12:55:38 DEBUG { 12:55:38 DEBUG "metric": { 12:55:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG [...] 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 1713527738, 12:55:38 DEBUG "0" 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG } 12:55:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527738" 12:55:38 INFO [http_cmd]: http status code OK 12:55:38 DEBUG --- status code --- 12:55:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:38 DEBUG --- http response --- 12:55:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:38 DEBUG ----- output ----- 12:55:38 DEBUG { 12:55:38 DEBUG "status": "success", 12:55:38 DEBUG "data": { 12:55:38 DEBUG "resultType": "vector", 12:55:38 DEBUG "result": [ 12:55:38 DEBUG { 12:55:38 DEBUG "metric": { 12:55:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG [...] 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 1713527738, 12:55:38 DEBUG "0" 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG } 12:55:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527738" 12:55:38 INFO [http_cmd]: http status code OK 12:55:38 DEBUG --- status code --- 12:55:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:38 DEBUG --- http response --- 12:55:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:38 DEBUG ----- output ----- 12:55:38 DEBUG { 12:55:38 DEBUG "status": "success", 12:55:38 DEBUG "data": { 12:55:38 DEBUG "resultType": "vector", 12:55:38 DEBUG "result": [ 12:55:38 DEBUG { 12:55:38 DEBUG "metric": { 12:55:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG [...] 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG 12:55:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:55:38 DEBUG }, 12:55:38 DEBUG "value": [ 12:55:38 DEBUG 1713527738, 12:55:38 DEBUG "0" 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG ] 12:55:38 DEBUG } 12:55:38 DEBUG } 12:55:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713527739" 12:55:39 INFO [http_cmd]: http status code OK 12:55:39 DEBUG --- status code --- 12:55:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:39 DEBUG --- http response --- 12:55:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:39 DEBUG ----- output ----- 12:55:39 DEBUG { 12:55:39 DEBUG "status": "success", 12:55:39 DEBUG "data": { 12:55:39 DEBUG "resultType": "vector", 12:55:39 DEBUG "result": [ 12:55:39 DEBUG { 12:55:39 DEBUG "metric": { 12:55:39 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:55:39 DEBUG }, 12:55:39 DEBUG "value": [ 12:55:39 DEBUG 12:55:39 DEBUG 12:55:39 DEBUG 12:55:39 DEBUG [...] 12:55:39 DEBUG 12:55:39 DEBUG 12:55:39 DEBUG 12:55:39 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:55:39 DEBUG }, 12:55:39 DEBUG "value": [ 12:55:39 DEBUG 1713527739, 12:55:39 DEBUG "1897234432" 12:55:39 DEBUG ] 12:55:39 DEBUG } 12:55:39 DEBUG ] 12:55:39 DEBUG } 12:55:39 DEBUG } 12:55:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527741" 12:55:41 INFO [http_cmd]: http status code OK 12:55:41 DEBUG --- status code --- 12:55:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:41 DEBUG --- http response --- 12:55:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:55:41 DEBUG ----- output ----- 12:55:41 DEBUG { 12:55:41 DEBUG "status": "success", 12:55:41 DEBUG "data": { 12:55:41 DEBUG "resultType": "vector", 12:55:41 DEBUG "result": [ 12:55:41 DEBUG { 12:55:41 DEBUG "metric": { 12:55:41 DEBUG "pod": "ds-cts-0" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG [...] 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG "pod": "ds-cts-2" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 1713527741, 12:55:41 DEBUG "30" 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG } 12:55:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713527741" 12:55:41 INFO [http_cmd]: http status code OK 12:55:41 DEBUG --- status code --- 12:55:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:41 DEBUG --- http response --- 12:55:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:41 DEBUG ----- output ----- 12:55:41 DEBUG { 12:55:41 DEBUG "status": "success", 12:55:41 DEBUG "data": { 12:55:41 DEBUG "resultType": "vector", 12:55:41 DEBUG "result": [ 12:55:41 DEBUG { 12:55:41 DEBUG "metric": { 12:55:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG [...] 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 1713527741, 12:55:41 DEBUG "55.19" 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG } 12:55:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527741" 12:55:41 INFO [http_cmd]: http status code OK 12:55:41 DEBUG --- status code --- 12:55:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:41 DEBUG --- http response --- 12:55:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:41 DEBUG ----- output ----- 12:55:41 DEBUG { 12:55:41 DEBUG "status": "success", 12:55:41 DEBUG "data": { 12:55:41 DEBUG "resultType": "vector", 12:55:41 DEBUG "result": [ 12:55:41 DEBUG { 12:55:41 DEBUG "metric": { 12:55:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG [...] 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG "pod": "ds-idrepo-2" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 1713527741, 12:55:41 DEBUG "0" 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG } 12:55:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527741" 12:55:41 INFO [http_cmd]: http status code OK 12:55:41 DEBUG --- status code --- 12:55:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:41 DEBUG --- http response --- 12:55:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:41 DEBUG ----- output ----- 12:55:41 DEBUG { 12:55:41 DEBUG "status": "success", 12:55:41 DEBUG "data": { 12:55:41 DEBUG "resultType": "vector", 12:55:41 DEBUG "result": [ 12:55:41 DEBUG { 12:55:41 DEBUG "metric": { 12:55:41 DEBUG "pod": "ds-idrepo-0" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG [...] 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG "pod": "ds-idrepo-2" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 1713527741, 12:55:41 DEBUG "0" 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG } 12:55:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527741" 12:55:41 INFO [http_cmd]: http status code OK 12:55:41 DEBUG --- status code --- 12:55:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:41 DEBUG --- http response --- 12:55:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:41 DEBUG ----- output ----- 12:55:41 DEBUG { 12:55:41 DEBUG "status": "success", 12:55:41 DEBUG "data": { 12:55:41 DEBUG "resultType": "vector", 12:55:41 DEBUG "result": [ 12:55:41 DEBUG { 12:55:41 DEBUG "metric": { 12:55:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG [...] 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG 12:55:41 DEBUG "pod": "ds-idrepo-2" 12:55:41 DEBUG }, 12:55:41 DEBUG "value": [ 12:55:41 DEBUG 1713527741, 12:55:41 DEBUG "0" 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG ] 12:55:41 DEBUG } 12:55:41 DEBUG } 12:55:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527742" 12:55:42 INFO [http_cmd]: http status code OK 12:55:42 DEBUG --- status code --- 12:55:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:42 DEBUG --- http response --- 12:55:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:42 DEBUG ----- output ----- 12:55:42 DEBUG { 12:55:42 DEBUG "status": "success", 12:55:42 DEBUG "data": { 12:55:42 DEBUG "resultType": "vector", 12:55:42 DEBUG "result": [ 12:55:42 DEBUG { 12:55:42 DEBUG "metric": { 12:55:42 DEBUG "instance": "10.142.0.107:9100" 12:55:42 DEBUG }, 12:55:42 DEBUG "value": [ 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG [...] 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG "instance": "10.142.0.43:9100" 12:55:42 DEBUG }, 12:55:42 DEBUG "value": [ 12:55:42 DEBUG 1713527742, 12:55:42 DEBUG "1586803704832" 12:55:42 DEBUG ] 12:55:42 DEBUG } 12:55:42 DEBUG ] 12:55:42 DEBUG } 12:55:42 DEBUG } 12:55:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527742" 12:55:42 INFO [http_cmd]: http status code OK 12:55:42 DEBUG --- status code --- 12:55:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:42 DEBUG --- http response --- 12:55:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:55:42 DEBUG ----- output ----- 12:55:42 DEBUG { 12:55:42 DEBUG "status": "success", 12:55:42 DEBUG "data": { 12:55:42 DEBUG "resultType": "vector", 12:55:42 DEBUG "result": [ 12:55:42 DEBUG { 12:55:42 DEBUG "metric": { 12:55:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:55:42 DEBUG }, 12:55:42 DEBUG "value": [ 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG [...] 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:55:42 DEBUG }, 12:55:42 DEBUG "value": [ 12:55:42 DEBUG 1713527742, 12:55:42 DEBUG "0" 12:55:42 DEBUG ] 12:55:42 DEBUG } 12:55:42 DEBUG ] 12:55:42 DEBUG } 12:55:42 DEBUG } 12:55:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527742" 12:55:42 INFO [http_cmd]: http status code OK 12:55:42 DEBUG --- status code --- 12:55:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:42 DEBUG --- http response --- 12:55:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:42 DEBUG ----- output ----- 12:55:42 DEBUG { 12:55:42 DEBUG "status": "success", 12:55:42 DEBUG "data": { 12:55:42 DEBUG "resultType": "vector", 12:55:42 DEBUG "result": [ 12:55:42 DEBUG { 12:55:42 DEBUG "metric": { 12:55:42 DEBUG "instance": "10.142.0.107:9100" 12:55:42 DEBUG }, 12:55:42 DEBUG "value": [ 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG [...] 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG "instance": "10.142.0.43:9100" 12:55:42 DEBUG }, 12:55:42 DEBUG "value": [ 12:55:42 DEBUG 1713527742, 12:55:42 DEBUG "16295954.533333333" 12:55:42 DEBUG ] 12:55:42 DEBUG } 12:55:42 DEBUG ] 12:55:42 DEBUG } 12:55:42 DEBUG } 12:55:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527742" 12:55:42 INFO [http_cmd]: http status code OK 12:55:42 DEBUG --- status code --- 12:55:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:55:42 DEBUG --- http response --- 12:55:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:55:42 DEBUG ----- output ----- 12:55:42 DEBUG { 12:55:42 DEBUG "status": "success", 12:55:42 DEBUG "data": { 12:55:42 DEBUG "resultType": "vector", 12:55:42 DEBUG "result": [ 12:55:42 DEBUG { 12:55:42 DEBUG "metric": { 12:55:42 DEBUG "instance": "10.142.0.107:9100" 12:55:42 DEBUG }, 12:55:42 DEBUG "value": [ 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG [...] 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG 12:55:42 DEBUG "instance": "10.142.0.43:9100" 12:55:42 DEBUG }, 12:55:42 DEBUG "value": [ 12:55:42 DEBUG 1713527742, 12:55:42 DEBUG "16302699.6" 12:55:42 DEBUG ] 12:55:42 DEBUG } 12:55:42 DEBUG ] 12:55:42 DEBUG } 12:55:42 DEBUG } 12:55:55 INFO 12:55:55 INFO [loop_until]: kubectl --namespace=xlou top pods 12:55:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:55:56 INFO [loop_until]: OK (rc = 0) 12:55:56 DEBUG --- stdout --- 12:55:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2405m 9435Mi am-6fd554f8f7-hvvkw 2458m 9011Mi am-6fd554f8f7-pmj4b 2560m 9133Mi ds-cts-0 9744m 13828Mi ds-cts-1 9657m 12877Mi ds-cts-2 10289m 13608Mi ds-idrepo-0 705m 3365Mi ds-idrepo-1 694m 3262Mi ds-idrepo-2 694m 3191Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 4m 1216Mi lodemon-747d57dcb8-lclbf 7m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1097m 1834Mi 12:55:56 DEBUG --- stderr --- 12:55:56 DEBUG 12:56:02 INFO 12:56:02 INFO [loop_until]: kubectl --namespace=xlou top node 12:56:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:56:02 INFO [loop_until]: OK (rc = 0) 12:56:02 DEBUG --- stdout --- 12:56:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6137Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 767m 2% 6886Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2437m 8% 11775Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 77m 0% 3225Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10244m 34% 16459Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5173Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2547m 8% 12266Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 3129Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2696m 9% 11386Mi 10% gke-xlou-cdm-ds-0f601751-j488 740m 2% 6369Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9470m 31% 17560Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 796m 2% 7030Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 10141m 33% 15558Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 6045Mi 10% 12:56:02 DEBUG --- stderr --- 12:56:02 DEBUG 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527797" 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527797" 12:56:37 INFO [http_cmd]: http status code OK 12:56:37 DEBUG --- status code --- 12:56:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:37 DEBUG --- http response --- 12:56:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:56:37 INFO [http_cmd]: http status code OK 12:56:37 DEBUG --- status code --- 12:56:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:37 DEBUG --- http response --- 12:56:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:56:37 DEBUG ----- output ----- 12:56:37 DEBUG { 12:56:37 DEBUG "status": "success", 12:56:37 DEBUG "data": { 12:56:37 DEBUG "resultType": "vector", 12:56:37 DEBUG "result": [ 12:56:37 DEBUG { 12:56:37 DEBUG "metric": { 12:56:37 DEBUG "pod": "ds-cts-0" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG [...] 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG "pod": "ds-cts-2" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 1713527797, 12:56:37 DEBUG "0.0003333333333333333" 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713527797" 12:56:37 ERROR [http_cmd]: ERROR 12:56:37 ERROR --- status code --- 12:56:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:56:37 ERROR --- http response --- 12:56:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:56:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527797" 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527797" 12:56:37 INFO [http_cmd]: http status code OK 12:56:37 DEBUG --- status code --- 12:56:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:37 DEBUG --- http response --- 12:56:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:56:37 DEBUG ----- output ----- 12:56:37 DEBUG { 12:56:37 DEBUG "status": "success", 12:56:37 DEBUG "data": { 12:56:37 DEBUG "resultType": "vector", 12:56:37 DEBUG "result": [ 12:56:37 DEBUG { 12:56:37 DEBUG "metric": { 12:56:37 DEBUG "pod": "ds-cts-0" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG [...] 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG "pod": "ds-cts-2" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 1713527797, 12:56:37 DEBUG "102.43333333333334" 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:37 INFO [http_cmd]: http status code OK 12:56:37 DEBUG --- status code --- 12:56:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:37 DEBUG --- http response --- 12:56:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:37 DEBUG ----- output ----- 12:56:37 DEBUG { 12:56:37 DEBUG "status": "success", 12:56:37 DEBUG "data": { 12:56:37 DEBUG "resultType": "vector", 12:56:37 DEBUG "result": [ 12:56:37 DEBUG { 12:56:37 DEBUG "metric": { 12:56:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG [...] 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 1713527797, 12:56:37 DEBUG "1708.0307189769605" 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713527797" 12:56:37 INFO [http_cmd]: http status code OK 12:56:37 DEBUG --- status code --- 12:56:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:37 DEBUG --- http response --- 12:56:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:56:37 DEBUG ----- output ----- 12:56:37 DEBUG { 12:56:37 DEBUG "status": "success", 12:56:37 DEBUG "data": { 12:56:37 DEBUG "resultType": "vector", 12:56:37 DEBUG "result": [ 12:56:37 DEBUG { 12:56:37 DEBUG "metric": { 12:56:37 DEBUG "pod": "ds-idrepo-0" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG [...] 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG "pod": "ds-cts-2" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 1713527797, 12:56:37 DEBUG "0" 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527797" 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527797" 12:56:37 INFO [http_cmd]: http status code OK 12:56:37 INFO [http_cmd]: http status code OK 12:56:37 DEBUG --- status code --- 12:56:37 DEBUG --- status code --- 12:56:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:37 DEBUG --- http response --- 12:56:37 DEBUG --- http response --- 12:56:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:37 DEBUG ----- output ----- 12:56:37 DEBUG ----- output ----- 12:56:37 DEBUG { 12:56:37 DEBUG { 12:56:37 DEBUG "status": "success", 12:56:37 DEBUG "status": "success", 12:56:37 DEBUG "data": { 12:56:37 DEBUG "data": { 12:56:37 DEBUG "resultType": "vector", 12:56:37 DEBUG "resultType": "vector", 12:56:37 DEBUG "result": [ 12:56:37 DEBUG "result": [ 12:56:37 DEBUG { 12:56:37 DEBUG { 12:56:37 DEBUG "metric": { 12:56:37 DEBUG "metric": { 12:56:37 DEBUG "pod": "ds-idrepo-0" 12:56:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:56:37 DEBUG }, 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527797" 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG [...] 12:56:37 DEBUG [...] 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG "pod": "ds-idrepo-2" 12:56:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:56:37 DEBUG }, 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 1713527797, 12:56:37 DEBUG 1713527797, 12:56:37 DEBUG "0" 12:56:37 DEBUG "2493.633333333333" 12:56:37 DEBUG ] 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:37 DEBUG ] 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:37 INFO [http_cmd]: http status code OK 12:56:37 DEBUG --- status code --- 12:56:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:37 DEBUG --- http response --- 12:56:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:37 DEBUG ----- output ----- 12:56:37 DEBUG { 12:56:37 DEBUG "status": "success", 12:56:37 DEBUG "data": { 12:56:37 DEBUG "resultType": "vector", 12:56:37 DEBUG "result": [ 12:56:37 DEBUG { 12:56:37 DEBUG "metric": { 12:56:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG [...] 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 1713527797, 12:56:37 DEBUG "2231355.5" 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713527797" 12:56:37 INFO [http_cmd]: http status code OK 12:56:37 DEBUG --- status code --- 12:56:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:37 DEBUG --- http response --- 12:56:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:37 DEBUG ----- output ----- 12:56:37 DEBUG { 12:56:37 DEBUG "status": "success", 12:56:37 DEBUG "data": { 12:56:37 DEBUG "resultType": "vector", 12:56:37 DEBUG "result": [ 12:56:37 DEBUG { 12:56:37 DEBUG "metric": { 12:56:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG [...] 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG 12:56:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:56:37 DEBUG }, 12:56:37 DEBUG "value": [ 12:56:37 DEBUG 1713527797, 12:56:37 DEBUG "1.0313104485305271" 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG ] 12:56:37 DEBUG } 12:56:37 DEBUG } 12:56:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527798" 12:56:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527798" 12:56:38 INFO [http_cmd]: http status code OK 12:56:38 DEBUG --- status code --- 12:56:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:38 DEBUG --- http response --- 12:56:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:38 DEBUG ----- output ----- 12:56:38 DEBUG { 12:56:38 DEBUG "status": "success", 12:56:38 DEBUG "data": { 12:56:38 DEBUG "resultType": "vector", 12:56:38 DEBUG "result": [ 12:56:38 DEBUG { 12:56:38 DEBUG "metric": { 12:56:38 DEBUG "pod": "ds-idrepo-0" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG [...] 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG "pod": "ds-idrepo-2" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 1713527798, 12:56:38 DEBUG "0" 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG } 12:56:38 INFO [http_cmd]: http status code OK 12:56:38 DEBUG --- status code --- 12:56:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:38 DEBUG --- http response --- 12:56:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:38 DEBUG ----- output ----- 12:56:38 DEBUG { 12:56:38 DEBUG "status": "success", 12:56:38 DEBUG "data": { 12:56:38 DEBUG "resultType": "vector", 12:56:38 DEBUG "result": [ 12:56:38 DEBUG { 12:56:38 DEBUG "metric": { 12:56:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG [...] 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 1713527798, 12:56:38 DEBUG "0" 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG } 12:56:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527798" 12:56:38 INFO [http_cmd]: http status code OK 12:56:38 DEBUG --- status code --- 12:56:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:38 DEBUG --- http response --- 12:56:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:38 DEBUG ----- output ----- 12:56:38 DEBUG { 12:56:38 DEBUG "status": "success", 12:56:38 DEBUG "data": { 12:56:38 DEBUG "resultType": "vector", 12:56:38 DEBUG "result": [ 12:56:38 DEBUG { 12:56:38 DEBUG "metric": { 12:56:38 DEBUG "instance": "10.142.0.107:9100" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG [...] 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG "instance": "10.142.0.43:9100" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 1713527798, 12:56:38 DEBUG "0.7773333333333368" 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG } 12:56:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527798" 12:56:38 INFO [http_cmd]: http status code OK 12:56:38 DEBUG --- status code --- 12:56:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:38 DEBUG --- http response --- 12:56:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:38 DEBUG ----- output ----- 12:56:38 DEBUG { 12:56:38 DEBUG "status": "success", 12:56:38 DEBUG "data": { 12:56:38 DEBUG "resultType": "vector", 12:56:38 DEBUG "result": [ 12:56:38 DEBUG { 12:56:38 DEBUG "metric": { 12:56:38 DEBUG "instance": "10.142.0.107:9100" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG [...] 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG "instance": "10.142.0.43:9100" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 1713527798, 12:56:38 DEBUG "0.6234333333334036" 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG } 12:56:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527798" 12:56:38 INFO [http_cmd]: http status code OK 12:56:38 DEBUG --- status code --- 12:56:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:38 DEBUG --- http response --- 12:56:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:38 DEBUG ----- output ----- 12:56:38 DEBUG { 12:56:38 DEBUG "status": "success", 12:56:38 DEBUG "data": { 12:56:38 DEBUG "resultType": "vector", 12:56:38 DEBUG "result": [ 12:56:38 DEBUG { 12:56:38 DEBUG "metric": { 12:56:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG [...] 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 1713527798, 12:56:38 DEBUG "0" 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG } 12:56:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527798" 12:56:38 INFO [http_cmd]: http status code OK 12:56:38 DEBUG --- status code --- 12:56:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:38 DEBUG --- http response --- 12:56:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:38 DEBUG ----- output ----- 12:56:38 DEBUG { 12:56:38 DEBUG "status": "success", 12:56:38 DEBUG "data": { 12:56:38 DEBUG "resultType": "vector", 12:56:38 DEBUG "result": [ 12:56:38 DEBUG { 12:56:38 DEBUG "metric": { 12:56:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG [...] 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 1713527798, 12:56:38 DEBUG "0" 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG } 12:56:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527798" 12:56:38 INFO [http_cmd]: http status code OK 12:56:38 DEBUG --- status code --- 12:56:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:38 DEBUG --- http response --- 12:56:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:38 DEBUG ----- output ----- 12:56:38 DEBUG { 12:56:38 DEBUG "status": "success", 12:56:38 DEBUG "data": { 12:56:38 DEBUG "resultType": "vector", 12:56:38 DEBUG "result": [ 12:56:38 DEBUG { 12:56:38 DEBUG "metric": { 12:56:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG [...] 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG 12:56:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:56:38 DEBUG }, 12:56:38 DEBUG "value": [ 12:56:38 DEBUG 1713527798, 12:56:38 DEBUG "0" 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG ] 12:56:38 DEBUG } 12:56:38 DEBUG } 12:56:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713527800" 12:56:40 INFO [http_cmd]: http status code OK 12:56:40 DEBUG --- status code --- 12:56:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:40 DEBUG --- http response --- 12:56:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:40 DEBUG ----- output ----- 12:56:40 DEBUG { 12:56:40 DEBUG "status": "success", 12:56:40 DEBUG "data": { 12:56:40 DEBUG "resultType": "vector", 12:56:40 DEBUG "result": [ 12:56:40 DEBUG { 12:56:40 DEBUG "metric": { 12:56:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:56:40 DEBUG }, 12:56:40 DEBUG "value": [ 12:56:40 DEBUG 12:56:40 DEBUG 12:56:40 DEBUG 12:56:40 DEBUG [...] 12:56:40 DEBUG 12:56:40 DEBUG 12:56:40 DEBUG 12:56:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:56:40 DEBUG }, 12:56:40 DEBUG "value": [ 12:56:40 DEBUG 1713527800, 12:56:40 DEBUG "2425389056" 12:56:40 DEBUG ] 12:56:40 DEBUG } 12:56:40 DEBUG ] 12:56:40 DEBUG } 12:56:40 DEBUG } 12:56:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527801" 12:56:41 INFO [http_cmd]: http status code OK 12:56:41 DEBUG --- status code --- 12:56:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:41 DEBUG --- http response --- 12:56:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:56:41 DEBUG ----- output ----- 12:56:41 DEBUG { 12:56:41 DEBUG "status": "success", 12:56:41 DEBUG "data": { 12:56:41 DEBUG "resultType": "vector", 12:56:41 DEBUG "result": [ 12:56:41 DEBUG { 12:56:41 DEBUG "metric": { 12:56:41 DEBUG "pod": "ds-cts-0" 12:56:41 DEBUG }, 12:56:41 DEBUG "value": [ 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG [...] 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG "pod": "ds-cts-2" 12:56:41 DEBUG }, 12:56:41 DEBUG "value": [ 12:56:41 DEBUG 1713527801, 12:56:41 DEBUG "29.966666666666665" 12:56:41 DEBUG ] 12:56:41 DEBUG } 12:56:41 DEBUG ] 12:56:41 DEBUG } 12:56:41 DEBUG } 12:56:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713527801" 12:56:41 INFO [http_cmd]: http status code OK 12:56:41 DEBUG --- status code --- 12:56:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:41 DEBUG --- http response --- 12:56:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:41 DEBUG ----- output ----- 12:56:41 DEBUG { 12:56:41 DEBUG "status": "success", 12:56:41 DEBUG "data": { 12:56:41 DEBUG "resultType": "vector", 12:56:41 DEBUG "result": [ 12:56:41 DEBUG { 12:56:41 DEBUG "metric": { 12:56:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:56:41 DEBUG }, 12:56:41 DEBUG "value": [ 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG [...] 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:56:41 DEBUG }, 12:56:41 DEBUG "value": [ 12:56:41 DEBUG 1713527801, 12:56:41 DEBUG "55.19" 12:56:41 DEBUG ] 12:56:41 DEBUG } 12:56:41 DEBUG ] 12:56:41 DEBUG } 12:56:41 DEBUG } 12:56:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527801" 12:56:41 INFO [http_cmd]: http status code OK 12:56:41 DEBUG --- status code --- 12:56:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:41 DEBUG --- http response --- 12:56:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:56:41 DEBUG ----- output ----- 12:56:41 DEBUG { 12:56:41 DEBUG "status": "success", 12:56:41 DEBUG "data": { 12:56:41 DEBUG "resultType": "vector", 12:56:41 DEBUG "result": [ 12:56:41 DEBUG { 12:56:41 DEBUG "metric": { 12:56:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:56:41 DEBUG }, 12:56:41 DEBUG "value": [ 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG [...] 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 12:56:41 DEBUG }, 12:56:41 DEBUG "value": [ 12:56:41 DEBUG 1713527801, 12:56:41 DEBUG "2.0070169237142528" 12:56:41 DEBUG ] 12:56:41 DEBUG } 12:56:41 DEBUG ] 12:56:41 DEBUG } 12:56:41 DEBUG } 12:56:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527801" 12:56:41 INFO [http_cmd]: http status code OK 12:56:41 DEBUG --- status code --- 12:56:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:41 DEBUG --- http response --- 12:56:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:41 DEBUG ----- output ----- 12:56:41 DEBUG { 12:56:41 DEBUG "status": "success", 12:56:41 DEBUG "data": { 12:56:41 DEBUG "resultType": "vector", 12:56:41 DEBUG "result": [ 12:56:41 DEBUG { 12:56:41 DEBUG "metric": { 12:56:41 DEBUG "pod": "ds-idrepo-0" 12:56:41 DEBUG }, 12:56:41 DEBUG "value": [ 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG [...] 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG 12:56:41 DEBUG "pod": "ds-idrepo-2" 12:56:41 DEBUG }, 12:56:41 DEBUG "value": [ 12:56:41 DEBUG 1713527801, 12:56:41 DEBUG "0" 12:56:41 DEBUG ] 12:56:41 DEBUG } 12:56:41 DEBUG ] 12:56:41 DEBUG } 12:56:41 DEBUG } 12:56:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527801" 12:56:42 INFO [http_cmd]: http status code OK 12:56:42 DEBUG --- status code --- 12:56:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:42 DEBUG --- http response --- 12:56:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:56:42 DEBUG ----- output ----- 12:56:42 DEBUG { 12:56:42 DEBUG "status": "success", 12:56:42 DEBUG "data": { 12:56:42 DEBUG "resultType": "vector", 12:56:42 DEBUG "result": [ 12:56:42 DEBUG { 12:56:42 DEBUG "metric": { 12:56:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG [...] 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 1713527801, 12:56:42 DEBUG "18242064.40800651" 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG } 12:56:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527802" 12:56:42 INFO [http_cmd]: http status code OK 12:56:42 DEBUG --- status code --- 12:56:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:42 DEBUG --- http response --- 12:56:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:42 DEBUG ----- output ----- 12:56:42 DEBUG { 12:56:42 DEBUG "status": "success", 12:56:42 DEBUG "data": { 12:56:42 DEBUG "resultType": "vector", 12:56:42 DEBUG "result": [ 12:56:42 DEBUG { 12:56:42 DEBUG "metric": { 12:56:42 DEBUG "instance": "10.142.0.107:9100" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG [...] 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG "instance": "10.142.0.43:9100" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 1713527802, 12:56:42 DEBUG "1588450534400" 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG } 12:56:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527802" 12:56:42 INFO [http_cmd]: http status code OK 12:56:42 DEBUG --- status code --- 12:56:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:42 DEBUG --- http response --- 12:56:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:56:42 DEBUG ----- output ----- 12:56:42 DEBUG { 12:56:42 DEBUG "status": "success", 12:56:42 DEBUG "data": { 12:56:42 DEBUG "resultType": "vector", 12:56:42 DEBUG "result": [ 12:56:42 DEBUG { 12:56:42 DEBUG "metric": { 12:56:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG [...] 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 1713527802, 12:56:42 DEBUG "0" 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG } 12:56:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527802" 12:56:42 INFO [http_cmd]: http status code OK 12:56:42 DEBUG --- status code --- 12:56:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:42 DEBUG --- http response --- 12:56:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:42 DEBUG ----- output ----- 12:56:42 DEBUG { 12:56:42 DEBUG "status": "success", 12:56:42 DEBUG "data": { 12:56:42 DEBUG "resultType": "vector", 12:56:42 DEBUG "result": [ 12:56:42 DEBUG { 12:56:42 DEBUG "metric": { 12:56:42 DEBUG "instance": "10.142.0.107:9100" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG [...] 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG "instance": "10.142.0.43:9100" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 1713527802, 12:56:42 DEBUG "16409057.899999999" 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG } 12:56:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527802" 12:56:42 INFO [http_cmd]: http status code OK 12:56:42 DEBUG --- status code --- 12:56:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:56:42 DEBUG --- http response --- 12:56:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:56:42 DEBUG ----- output ----- 12:56:42 DEBUG { 12:56:42 DEBUG "status": "success", 12:56:42 DEBUG "data": { 12:56:42 DEBUG "resultType": "vector", 12:56:42 DEBUG "result": [ 12:56:42 DEBUG { 12:56:42 DEBUG "metric": { 12:56:42 DEBUG "instance": "10.142.0.107:9100" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG [...] 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG 12:56:42 DEBUG "instance": "10.142.0.43:9100" 12:56:42 DEBUG }, 12:56:42 DEBUG "value": [ 12:56:42 DEBUG 1713527802, 12:56:42 DEBUG "16415838.933333334" 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG ] 12:56:42 DEBUG } 12:56:42 DEBUG } 12:56:56 INFO 12:56:56 INFO [loop_until]: kubectl --namespace=xlou top pods 12:56:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:56:56 INFO [loop_until]: OK (rc = 0) 12:56:56 DEBUG --- stdout --- 12:56:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2348m 9435Mi am-6fd554f8f7-hvvkw 2378m 9009Mi am-6fd554f8f7-pmj4b 2543m 9138Mi ds-cts-0 10061m 13691Mi ds-cts-1 9477m 13172Mi ds-cts-2 10027m 13003Mi ds-idrepo-0 666m 3364Mi ds-idrepo-1 711m 3262Mi ds-idrepo-2 672m 3191Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 4m 1215Mi lodemon-747d57dcb8-lclbf 7m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1024m 2342Mi 12:56:56 DEBUG --- stderr --- 12:56:56 DEBUG 12:57:02 INFO 12:57:02 INFO [loop_until]: kubectl --namespace=xlou top node 12:57:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:57:02 INFO [loop_until]: OK (rc = 0) 12:57:02 DEBUG --- stdout --- 12:57:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6135Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 734m 2% 6895Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2383m 7% 11780Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 74m 0% 3237Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9651m 32% 16743Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5180Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2534m 8% 12264Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 3121Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2665m 8% 11377Mi 10% gke-xlou-cdm-ds-0f601751-j488 750m 2% 6377Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9481m 31% 17490Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 790m 2% 7015Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9605m 32% 16078Mi 14% gke-xlou-cdm-frontend-a8771548-i82n 1077m 6% 6573Mi 11% 12:57:02 DEBUG --- stderr --- 12:57:02 DEBUG 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527857" 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527857" 12:57:37 INFO [http_cmd]: http status code OK 12:57:37 DEBUG --- status code --- 12:57:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:37 DEBUG --- http response --- 12:57:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:57:37 DEBUG ----- output ----- 12:57:37 DEBUG { 12:57:37 DEBUG "status": "success", 12:57:37 DEBUG "data": { 12:57:37 DEBUG "resultType": "vector", 12:57:37 DEBUG "result": [ 12:57:37 DEBUG { 12:57:37 DEBUG "metric": { 12:57:37 DEBUG "pod": "ds-cts-0" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG [...] 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG "pod": "ds-cts-2" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 1713527857, 12:57:37 DEBUG "0.0002333333333333333" 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG } 12:57:37 INFO [http_cmd]: http status code OK 12:57:37 DEBUG --- status code --- 12:57:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:37 DEBUG --- http response --- 12:57:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713527857" 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527857" 12:57:37 ERROR [http_cmd]: ERROR 12:57:37 ERROR --- status code --- 12:57:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:57:37 ERROR --- http response --- 12:57:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:57:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:57:37 INFO [http_cmd]: http status code OK 12:57:37 DEBUG --- status code --- 12:57:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:37 DEBUG --- http response --- 12:57:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:57:37 DEBUG ----- output ----- 12:57:37 DEBUG { 12:57:37 DEBUG "status": "success", 12:57:37 DEBUG "data": { 12:57:37 DEBUG "resultType": "vector", 12:57:37 DEBUG "result": [ 12:57:37 DEBUG { 12:57:37 DEBUG "metric": { 12:57:37 DEBUG "pod": "ds-cts-0" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG [...] 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG "pod": "ds-cts-2" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 1713527857, 12:57:37 DEBUG "48.333333333333336" 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG } 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527857" 12:57:37 INFO [http_cmd]: http status code OK 12:57:37 DEBUG --- status code --- 12:57:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:37 DEBUG --- http response --- 12:57:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:57:37 DEBUG ----- output ----- 12:57:37 DEBUG { 12:57:37 DEBUG "status": "success", 12:57:37 DEBUG "data": { 12:57:37 DEBUG "resultType": "vector", 12:57:37 DEBUG "result": [ 12:57:37 DEBUG { 12:57:37 DEBUG "metric": { 12:57:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG [...] 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 1713527857, 12:57:37 DEBUG "0" 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG } 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713527857" 12:57:37 INFO [http_cmd]: http status code OK 12:57:37 DEBUG --- status code --- 12:57:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:37 DEBUG --- http response --- 12:57:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:57:37 DEBUG ----- output ----- 12:57:37 DEBUG { 12:57:37 DEBUG "status": "success", 12:57:37 DEBUG "data": { 12:57:37 DEBUG "resultType": "vector", 12:57:37 DEBUG "result": [ 12:57:37 DEBUG { 12:57:37 DEBUG "metric": { 12:57:37 DEBUG "pod": "ds-idrepo-0" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG [...] 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG "pod": "ds-cts-2" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 1713527857, 12:57:37 DEBUG "0" 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG } 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527857" 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527857" 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527857" 12:57:37 INFO [http_cmd]: http status code OK 12:57:37 DEBUG --- status code --- 12:57:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:37 DEBUG --- http response --- 12:57:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:37 DEBUG ----- output ----- 12:57:37 DEBUG { 12:57:37 DEBUG "status": "success", 12:57:37 DEBUG "data": { 12:57:37 DEBUG "resultType": "vector", 12:57:37 DEBUG "result": [ 12:57:37 DEBUG { 12:57:37 DEBUG "metric": { 12:57:37 DEBUG "pod": "ds-idrepo-0" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG [...] 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG "pod": "ds-idrepo-2" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 1713527857, 12:57:37 DEBUG "0" 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG } 12:57:37 INFO [http_cmd]: http status code OK 12:57:37 DEBUG --- status code --- 12:57:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:37 DEBUG --- http response --- 12:57:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:37 DEBUG ----- output ----- 12:57:37 DEBUG { 12:57:37 DEBUG "status": "success", 12:57:37 DEBUG "data": { 12:57:37 DEBUG "resultType": "vector", 12:57:37 DEBUG "result": [ 12:57:37 DEBUG { 12:57:37 DEBUG "metric": { 12:57:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG [...] 12:57:37 DEBUG 12:57:37 INFO [http_cmd]: http status code OK 12:57:37 DEBUG 12:57:37 DEBUG --- status code --- 12:57:37 DEBUG 12:57:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 1713527857, 12:57:37 DEBUG "2494.7666666666664" 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG ] 12:57:37 DEBUG --- http response --- 12:57:37 DEBUG } 12:57:37 DEBUG } 12:57:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:37 DEBUG ----- output ----- 12:57:37 DEBUG { 12:57:37 DEBUG "status": "success", 12:57:37 DEBUG "data": { 12:57:37 DEBUG "resultType": "vector", 12:57:37 DEBUG "result": [ 12:57:37 DEBUG { 12:57:37 DEBUG "metric": { 12:57:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG [...] 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 1713527857, 12:57:37 DEBUG "2081085.5" 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG } 12:57:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713527857" 12:57:37 INFO [http_cmd]: http status code OK 12:57:37 DEBUG --- status code --- 12:57:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:37 DEBUG --- http response --- 12:57:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:37 DEBUG ----- output ----- 12:57:37 DEBUG { 12:57:37 DEBUG "status": "success", 12:57:37 DEBUG "data": { 12:57:37 DEBUG "resultType": "vector", 12:57:37 DEBUG "result": [ 12:57:37 DEBUG { 12:57:37 DEBUG "metric": { 12:57:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG [...] 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG 12:57:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:57:37 DEBUG }, 12:57:37 DEBUG "value": [ 12:57:37 DEBUG 1713527857, 12:57:37 DEBUG "1.035763801967128" 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG ] 12:57:37 DEBUG } 12:57:37 DEBUG } 12:57:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527858" 12:57:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527858" 12:57:38 INFO [http_cmd]: http status code OK 12:57:38 DEBUG --- status code --- 12:57:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:38 DEBUG --- http response --- 12:57:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:38 DEBUG ----- output ----- 12:57:38 DEBUG { 12:57:38 DEBUG "status": "success", 12:57:38 DEBUG "data": { 12:57:38 DEBUG "resultType": "vector", 12:57:38 DEBUG "result": [ 12:57:38 DEBUG { 12:57:38 DEBUG "metric": { 12:57:38 DEBUG "pod": "ds-idrepo-0" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG [...] 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG "pod": "ds-idrepo-2" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 1713527858, 12:57:38 DEBUG "0" 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG } 12:57:38 INFO [http_cmd]: http status code OK 12:57:38 DEBUG --- status code --- 12:57:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:38 DEBUG --- http response --- 12:57:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:38 DEBUG ----- output ----- 12:57:38 DEBUG { 12:57:38 DEBUG "status": "success", 12:57:38 DEBUG "data": { 12:57:38 DEBUG "resultType": "vector", 12:57:38 DEBUG "result": [ 12:57:38 DEBUG { 12:57:38 DEBUG "metric": { 12:57:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG [...] 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 1713527858, 12:57:38 DEBUG "0" 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG } 12:57:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527858" 12:57:38 INFO [http_cmd]: http status code OK 12:57:38 DEBUG --- status code --- 12:57:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:38 DEBUG --- http response --- 12:57:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:38 DEBUG ----- output ----- 12:57:38 DEBUG { 12:57:38 DEBUG "status": "success", 12:57:38 DEBUG "data": { 12:57:38 DEBUG "resultType": "vector", 12:57:38 DEBUG "result": [ 12:57:38 DEBUG { 12:57:38 DEBUG "metric": { 12:57:38 DEBUG "instance": "10.142.0.107:9100" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG [...] 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG "instance": "10.142.0.43:9100" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 1713527858, 12:57:38 DEBUG "1.114999999999994" 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG } 12:57:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527858" 12:57:38 INFO [http_cmd]: http status code OK 12:57:38 DEBUG --- status code --- 12:57:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:38 DEBUG --- http response --- 12:57:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:38 DEBUG ----- output ----- 12:57:38 DEBUG { 12:57:38 DEBUG "status": "success", 12:57:38 DEBUG "data": { 12:57:38 DEBUG "resultType": "vector", 12:57:38 DEBUG "result": [ 12:57:38 DEBUG { 12:57:38 DEBUG "metric": { 12:57:38 DEBUG "instance": "10.142.0.107:9100" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG [...] 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG "instance": "10.142.0.43:9100" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 1713527858, 12:57:38 DEBUG "0.7178999999999406" 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG } 12:57:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527858" 12:57:38 INFO [http_cmd]: http status code OK 12:57:38 DEBUG --- status code --- 12:57:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:38 DEBUG --- http response --- 12:57:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:38 DEBUG ----- output ----- 12:57:38 DEBUG { 12:57:38 DEBUG "status": "success", 12:57:38 DEBUG "data": { 12:57:38 DEBUG "resultType": "vector", 12:57:38 DEBUG "result": [ 12:57:38 DEBUG { 12:57:38 DEBUG "metric": { 12:57:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG [...] 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 1713527858, 12:57:38 DEBUG "0" 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG } 12:57:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527858" 12:57:38 INFO [http_cmd]: http status code OK 12:57:38 DEBUG --- status code --- 12:57:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:38 DEBUG --- http response --- 12:57:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:38 DEBUG ----- output ----- 12:57:38 DEBUG { 12:57:38 DEBUG "status": "success", 12:57:38 DEBUG "data": { 12:57:38 DEBUG "resultType": "vector", 12:57:38 DEBUG "result": [ 12:57:38 DEBUG { 12:57:38 DEBUG "metric": { 12:57:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG [...] 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 1713527858, 12:57:38 DEBUG "0" 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG } 12:57:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527858" 12:57:38 INFO [http_cmd]: http status code OK 12:57:38 DEBUG --- status code --- 12:57:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:38 DEBUG --- http response --- 12:57:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:38 DEBUG ----- output ----- 12:57:38 DEBUG { 12:57:38 DEBUG "status": "success", 12:57:38 DEBUG "data": { 12:57:38 DEBUG "resultType": "vector", 12:57:38 DEBUG "result": [ 12:57:38 DEBUG { 12:57:38 DEBUG "metric": { 12:57:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG [...] 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG 12:57:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:57:38 DEBUG }, 12:57:38 DEBUG "value": [ 12:57:38 DEBUG 1713527858, 12:57:38 DEBUG "0" 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG ] 12:57:38 DEBUG } 12:57:38 DEBUG } 12:57:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713527860" 12:57:40 INFO [http_cmd]: http status code OK 12:57:40 DEBUG --- status code --- 12:57:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:40 DEBUG --- http response --- 12:57:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:40 DEBUG ----- output ----- 12:57:40 DEBUG { 12:57:40 DEBUG "status": "success", 12:57:40 DEBUG "data": { 12:57:40 DEBUG "resultType": "vector", 12:57:40 DEBUG "result": [ 12:57:40 DEBUG { 12:57:40 DEBUG "metric": { 12:57:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:57:40 DEBUG }, 12:57:40 DEBUG "value": [ 12:57:40 DEBUG 12:57:40 DEBUG 12:57:40 DEBUG 12:57:40 DEBUG [...] 12:57:40 DEBUG 12:57:40 DEBUG 12:57:40 DEBUG 12:57:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:57:40 DEBUG }, 12:57:40 DEBUG "value": [ 12:57:40 DEBUG 1713527860, 12:57:40 DEBUG "2993176576" 12:57:40 DEBUG ] 12:57:40 DEBUG } 12:57:40 DEBUG ] 12:57:40 DEBUG } 12:57:40 DEBUG } 12:57:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527861" 12:57:41 INFO [http_cmd]: http status code OK 12:57:41 DEBUG --- status code --- 12:57:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:41 DEBUG --- http response --- 12:57:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:57:41 DEBUG ----- output ----- 12:57:41 DEBUG { 12:57:41 DEBUG "status": "success", 12:57:41 DEBUG "data": { 12:57:41 DEBUG "resultType": "vector", 12:57:41 DEBUG "result": [ 12:57:41 DEBUG { 12:57:41 DEBUG "metric": { 12:57:41 DEBUG "pod": "ds-cts-0" 12:57:41 DEBUG }, 12:57:41 DEBUG "value": [ 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG [...] 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG "pod": "ds-cts-2" 12:57:41 DEBUG }, 12:57:41 DEBUG "value": [ 12:57:41 DEBUG 1713527861, 12:57:41 DEBUG "30" 12:57:41 DEBUG ] 12:57:41 DEBUG } 12:57:41 DEBUG ] 12:57:41 DEBUG } 12:57:41 DEBUG } 12:57:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713527861" 12:57:41 INFO [http_cmd]: http status code OK 12:57:41 DEBUG --- status code --- 12:57:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:41 DEBUG --- http response --- 12:57:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:41 DEBUG ----- output ----- 12:57:41 DEBUG { 12:57:41 DEBUG "status": "success", 12:57:41 DEBUG "data": { 12:57:41 DEBUG "resultType": "vector", 12:57:41 DEBUG "result": [ 12:57:41 DEBUG { 12:57:41 DEBUG "metric": { 12:57:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:57:41 DEBUG }, 12:57:41 DEBUG "value": [ 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG [...] 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:57:41 DEBUG }, 12:57:41 DEBUG "value": [ 12:57:41 DEBUG 1713527861, 12:57:41 DEBUG "55.19" 12:57:41 DEBUG ] 12:57:41 DEBUG } 12:57:41 DEBUG ] 12:57:41 DEBUG } 12:57:41 DEBUG } 12:57:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527861" 12:57:41 INFO [http_cmd]: http status code OK 12:57:41 DEBUG --- status code --- 12:57:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:41 DEBUG --- http response --- 12:57:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:57:41 DEBUG ----- output ----- 12:57:41 DEBUG { 12:57:41 DEBUG "status": "success", 12:57:41 DEBUG "data": { 12:57:41 DEBUG "resultType": "vector", 12:57:41 DEBUG "result": [ 12:57:41 DEBUG { 12:57:41 DEBUG "metric": { 12:57:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:57:41 DEBUG }, 12:57:41 DEBUG "value": [ 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG [...] 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 12:57:41 DEBUG }, 12:57:41 DEBUG "value": [ 12:57:41 DEBUG 1713527861, 12:57:41 DEBUG "0.0030940713335862132" 12:57:41 DEBUG ] 12:57:41 DEBUG } 12:57:41 DEBUG ] 12:57:41 DEBUG } 12:57:41 DEBUG } 12:57:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527861" 12:57:41 INFO [http_cmd]: http status code OK 12:57:41 DEBUG --- status code --- 12:57:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:41 DEBUG --- http response --- 12:57:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:41 DEBUG ----- output ----- 12:57:41 DEBUG { 12:57:41 DEBUG "status": "success", 12:57:41 DEBUG "data": { 12:57:41 DEBUG "resultType": "vector", 12:57:41 DEBUG "result": [ 12:57:41 DEBUG { 12:57:41 DEBUG "metric": { 12:57:41 DEBUG "pod": "ds-idrepo-0" 12:57:41 DEBUG }, 12:57:41 DEBUG "value": [ 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG [...] 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG 12:57:41 DEBUG "pod": "ds-idrepo-2" 12:57:41 DEBUG }, 12:57:41 DEBUG "value": [ 12:57:41 DEBUG 1713527861, 12:57:41 DEBUG "0" 12:57:41 DEBUG ] 12:57:41 DEBUG } 12:57:41 DEBUG ] 12:57:41 DEBUG } 12:57:41 DEBUG } 12:57:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527862" 12:57:42 INFO [http_cmd]: http status code OK 12:57:42 DEBUG --- status code --- 12:57:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:42 DEBUG --- http response --- 12:57:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 12:57:42 DEBUG ----- output ----- 12:57:42 DEBUG { 12:57:42 DEBUG "status": "success", 12:57:42 DEBUG "data": { 12:57:42 DEBUG "resultType": "vector", 12:57:42 DEBUG "result": [ 12:57:42 DEBUG { 12:57:42 DEBUG "metric": { 12:57:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG [...] 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 1713527862, 12:57:42 DEBUG "1909003.4744022258" 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG } 12:57:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527862" 12:57:42 INFO [http_cmd]: http status code OK 12:57:42 DEBUG --- status code --- 12:57:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:42 DEBUG --- http response --- 12:57:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:42 DEBUG ----- output ----- 12:57:42 DEBUG { 12:57:42 DEBUG "status": "success", 12:57:42 DEBUG "data": { 12:57:42 DEBUG "resultType": "vector", 12:57:42 DEBUG "result": [ 12:57:42 DEBUG { 12:57:42 DEBUG "metric": { 12:57:42 DEBUG "instance": "10.142.0.107:9100" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG [...] 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG "instance": "10.142.0.43:9100" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 1713527862, 12:57:42 DEBUG "1590104538112" 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG } 12:57:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527862" 12:57:42 INFO [http_cmd]: http status code OK 12:57:42 DEBUG --- status code --- 12:57:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:42 DEBUG --- http response --- 12:57:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:57:42 DEBUG ----- output ----- 12:57:42 DEBUG { 12:57:42 DEBUG "status": "success", 12:57:42 DEBUG "data": { 12:57:42 DEBUG "resultType": "vector", 12:57:42 DEBUG "result": [ 12:57:42 DEBUG { 12:57:42 DEBUG "metric": { 12:57:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG [...] 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 1713527862, 12:57:42 DEBUG "0" 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG } 12:57:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527862" 12:57:42 INFO [http_cmd]: http status code OK 12:57:42 DEBUG --- status code --- 12:57:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:42 DEBUG --- http response --- 12:57:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:42 DEBUG ----- output ----- 12:57:42 DEBUG { 12:57:42 DEBUG "status": "success", 12:57:42 DEBUG "data": { 12:57:42 DEBUG "resultType": "vector", 12:57:42 DEBUG "result": [ 12:57:42 DEBUG { 12:57:42 DEBUG "metric": { 12:57:42 DEBUG "instance": "10.142.0.107:9100" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG [...] 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG "instance": "10.142.0.43:9100" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 1713527862, 12:57:42 DEBUG "16393319.133333333" 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG } 12:57:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527862" 12:57:42 INFO [http_cmd]: http status code OK 12:57:42 DEBUG --- status code --- 12:57:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:57:42 DEBUG --- http response --- 12:57:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:57:42 DEBUG ----- output ----- 12:57:42 DEBUG { 12:57:42 DEBUG "status": "success", 12:57:42 DEBUG "data": { 12:57:42 DEBUG "resultType": "vector", 12:57:42 DEBUG "result": [ 12:57:42 DEBUG { 12:57:42 DEBUG "metric": { 12:57:42 DEBUG "instance": "10.142.0.107:9100" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG [...] 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG 12:57:42 DEBUG "instance": "10.142.0.43:9100" 12:57:42 DEBUG }, 12:57:42 DEBUG "value": [ 12:57:42 DEBUG 1713527862, 12:57:42 DEBUG "16398799.2" 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG ] 12:57:42 DEBUG } 12:57:42 DEBUG } 12:57:56 INFO 12:57:56 INFO [loop_until]: kubectl --namespace=xlou top pods 12:57:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:57:56 INFO [loop_until]: OK (rc = 0) 12:57:56 DEBUG --- stdout --- 12:57:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2495m 9438Mi am-6fd554f8f7-hvvkw 2451m 9009Mi am-6fd554f8f7-pmj4b 2485m 9133Mi ds-cts-0 10512m 13300Mi ds-cts-1 9129m 13413Mi ds-cts-2 10162m 12732Mi ds-idrepo-0 685m 3365Mi ds-idrepo-1 695m 3262Mi ds-idrepo-2 687m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-747d57dcb8-lclbf 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 989m 2892Mi 12:57:56 DEBUG --- stderr --- 12:57:56 DEBUG 12:58:02 INFO 12:58:02 INFO [loop_until]: kubectl --namespace=xlou top node 12:58:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:58:02 INFO [loop_until]: OK (rc = 0) 12:58:02 DEBUG --- stdout --- 12:58:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6142Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 760m 2% 6886Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2443m 8% 11777Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3241Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9640m 32% 16486Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5177Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2586m 8% 12264Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 3118Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2729m 9% 11374Mi 10% gke-xlou-cdm-ds-0f601751-j488 763m 2% 6359Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8979m 30% 17974Mi 16% gke-xlou-cdm-ds-0f601751-nj6q 803m 2% 7027Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9918m 33% 15305Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1078m 6% 7216Mi 12% 12:58:02 DEBUG --- stderr --- 12:58:02 DEBUG 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527917" 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527917" 12:58:37 INFO [http_cmd]: http status code OK 12:58:37 DEBUG --- status code --- 12:58:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:37 DEBUG --- http response --- 12:58:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:58:37 DEBUG ----- output ----- 12:58:37 DEBUG { 12:58:37 DEBUG "status": "success", 12:58:37 DEBUG "data": { 12:58:37 DEBUG "resultType": "vector", 12:58:37 DEBUG "result": [ 12:58:37 DEBUG { 12:58:37 DEBUG "metric": { 12:58:37 DEBUG "pod": "ds-cts-0" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG [...] 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG "pod": "ds-cts-2" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 1713527917, 12:58:37 DEBUG "0.00006666666666666667" 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG } 12:58:37 INFO [http_cmd]: http status code OK 12:58:37 DEBUG --- status code --- 12:58:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:37 DEBUG --- http response --- 12:58:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713527917" 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527917" 12:58:37 ERROR [http_cmd]: ERROR 12:58:37 ERROR --- status code --- 12:58:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:58:37 ERROR --- http response --- 12:58:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:58:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:58:37 INFO [http_cmd]: http status code OK 12:58:37 DEBUG --- status code --- 12:58:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:37 DEBUG --- http response --- 12:58:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:58:37 DEBUG ----- output ----- 12:58:37 DEBUG { 12:58:37 DEBUG "status": "success", 12:58:37 DEBUG "data": { 12:58:37 DEBUG "resultType": "vector", 12:58:37 DEBUG "result": [ 12:58:37 DEBUG { 12:58:37 DEBUG "metric": { 12:58:37 DEBUG "pod": "ds-cts-0" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG [...] 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG "pod": "ds-cts-2" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 1713527917, 12:58:37 DEBUG "10.633333333333333" 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG } 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527917" 12:58:37 INFO [http_cmd]: http status code OK 12:58:37 DEBUG --- status code --- 12:58:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:37 DEBUG --- http response --- 12:58:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:37 DEBUG ----- output ----- 12:58:37 DEBUG { 12:58:37 DEBUG "status": "success", 12:58:37 DEBUG "data": { 12:58:37 DEBUG "resultType": "vector", 12:58:37 DEBUG "result": [ 12:58:37 DEBUG { 12:58:37 DEBUG "metric": { 12:58:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG [...] 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG "pod": "ds-cts-2" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 1713527917, 12:58:37 DEBUG "30487252.526505053" 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG } 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713527917" 12:58:37 INFO [http_cmd]: http status code OK 12:58:37 DEBUG --- status code --- 12:58:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:37 DEBUG --- http response --- 12:58:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:58:37 DEBUG ----- output ----- 12:58:37 DEBUG { 12:58:37 DEBUG "status": "success", 12:58:37 DEBUG "data": { 12:58:37 DEBUG "resultType": "vector", 12:58:37 DEBUG "result": [ 12:58:37 DEBUG { 12:58:37 DEBUG "metric": { 12:58:37 DEBUG "pod": "ds-idrepo-0" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG [...] 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG "pod": "ds-cts-2" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 1713527917, 12:58:37 DEBUG "0" 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG } 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527917" 12:58:37 INFO [http_cmd]: http status code OK 12:58:37 DEBUG --- status code --- 12:58:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:37 DEBUG --- http response --- 12:58:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:37 DEBUG ----- output ----- 12:58:37 DEBUG { 12:58:37 DEBUG "status": "success", 12:58:37 DEBUG "data": { 12:58:37 DEBUG "resultType": "vector", 12:58:37 DEBUG "result": [ 12:58:37 DEBUG { 12:58:37 DEBUG "metric": { 12:58:37 DEBUG "pod": "ds-idrepo-0" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG [...] 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG "pod": "ds-idrepo-2" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 1713527917, 12:58:37 DEBUG "0" 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG } 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527917" 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527917" 12:58:37 INFO [http_cmd]: http status code OK 12:58:37 DEBUG --- status code --- 12:58:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:37 INFO [http_cmd]: http status code OK 12:58:37 DEBUG --- http response --- 12:58:37 DEBUG --- status code --- 12:58:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:37 DEBUG ----- output ----- 12:58:37 DEBUG { 12:58:37 DEBUG --- http response --- 12:58:37 DEBUG "status": "success", 12:58:37 DEBUG "data": { 12:58:37 DEBUG "resultType": "vector", 12:58:37 DEBUG "result": [ 12:58:37 DEBUG { 12:58:37 DEBUG "metric": { 12:58:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG [...] 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 1713527917, 12:58:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:37 DEBUG "2500.4" 12:58:37 DEBUG ----- output ----- 12:58:37 DEBUG ] 12:58:37 DEBUG { 12:58:37 DEBUG } 12:58:37 DEBUG "status": "success", 12:58:37 DEBUG ] 12:58:37 DEBUG "data": { 12:58:37 DEBUG } 12:58:37 DEBUG "resultType": "vector", 12:58:37 DEBUG } 12:58:37 DEBUG "result": [ 12:58:37 DEBUG { 12:58:37 DEBUG "metric": { 12:58:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG [...] 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 1713527917, 12:58:37 DEBUG "1930732" 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG } 12:58:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713527917" 12:58:37 INFO [http_cmd]: http status code OK 12:58:37 DEBUG --- status code --- 12:58:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:37 DEBUG --- http response --- 12:58:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:37 DEBUG ----- output ----- 12:58:37 DEBUG { 12:58:37 DEBUG "status": "success", 12:58:37 DEBUG "data": { 12:58:37 DEBUG "resultType": "vector", 12:58:37 DEBUG "result": [ 12:58:37 DEBUG { 12:58:37 DEBUG "metric": { 12:58:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG [...] 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG 12:58:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:58:37 DEBUG }, 12:58:37 DEBUG "value": [ 12:58:37 DEBUG 1713527917, 12:58:37 DEBUG "1.0354226906217507" 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG ] 12:58:37 DEBUG } 12:58:37 DEBUG } 12:58:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527918" 12:58:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527918" 12:58:38 INFO [http_cmd]: http status code OK 12:58:38 DEBUG --- status code --- 12:58:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:38 DEBUG --- http response --- 12:58:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:38 DEBUG ----- output ----- 12:58:38 DEBUG { 12:58:38 DEBUG "status": "success", 12:58:38 DEBUG "data": { 12:58:38 DEBUG "resultType": "vector", 12:58:38 DEBUG "result": [ 12:58:38 DEBUG { 12:58:38 DEBUG "metric": { 12:58:38 DEBUG "pod": "ds-idrepo-0" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG [...] 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG "pod": "ds-idrepo-2" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 1713527918, 12:58:38 DEBUG "0" 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG } 12:58:38 INFO [http_cmd]: http status code OK 12:58:38 DEBUG --- status code --- 12:58:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:38 DEBUG --- http response --- 12:58:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:58:38 DEBUG ----- output ----- 12:58:38 DEBUG { 12:58:38 DEBUG "status": "success", 12:58:38 DEBUG "data": { 12:58:38 DEBUG "resultType": "vector", 12:58:38 DEBUG "result": [ 12:58:38 DEBUG { 12:58:38 DEBUG "metric": { 12:58:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG [...] 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG "pod": "ds-cts-2" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 1713527918, 12:58:38 DEBUG "74024362.57672516" 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG } 12:58:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527918" 12:58:38 INFO [http_cmd]: http status code OK 12:58:38 DEBUG --- status code --- 12:58:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:38 DEBUG --- http response --- 12:58:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:38 DEBUG ----- output ----- 12:58:38 DEBUG { 12:58:38 DEBUG "status": "success", 12:58:38 DEBUG "data": { 12:58:38 DEBUG "resultType": "vector", 12:58:38 DEBUG "result": [ 12:58:38 DEBUG { 12:58:38 DEBUG "metric": { 12:58:38 DEBUG "instance": "10.142.0.107:9100" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG [...] 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG "instance": "10.142.0.43:9100" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 1713527918, 12:58:38 DEBUG "0.931333333333337" 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG } 12:58:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527918" 12:58:38 INFO [http_cmd]: http status code OK 12:58:38 DEBUG --- status code --- 12:58:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:38 DEBUG --- http response --- 12:58:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:38 DEBUG ----- output ----- 12:58:38 DEBUG { 12:58:38 DEBUG "status": "success", 12:58:38 DEBUG "data": { 12:58:38 DEBUG "resultType": "vector", 12:58:38 DEBUG "result": [ 12:58:38 DEBUG { 12:58:38 DEBUG "metric": { 12:58:38 DEBUG "instance": "10.142.0.107:9100" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG [...] 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG "instance": "10.142.0.43:9100" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 1713527918, 12:58:38 DEBUG "0.6717333333333481" 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG } 12:58:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527918" 12:58:38 INFO [http_cmd]: http status code OK 12:58:38 DEBUG --- status code --- 12:58:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:38 DEBUG --- http response --- 12:58:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:38 DEBUG ----- output ----- 12:58:38 DEBUG { 12:58:38 DEBUG "status": "success", 12:58:38 DEBUG "data": { 12:58:38 DEBUG "resultType": "vector", 12:58:38 DEBUG "result": [ 12:58:38 DEBUG { 12:58:38 DEBUG "metric": { 12:58:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG [...] 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 1713527918, 12:58:38 DEBUG "0" 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG } 12:58:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527918" 12:58:38 INFO [http_cmd]: http status code OK 12:58:38 DEBUG --- status code --- 12:58:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:38 DEBUG --- http response --- 12:58:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:38 DEBUG ----- output ----- 12:58:38 DEBUG { 12:58:38 DEBUG "status": "success", 12:58:38 DEBUG "data": { 12:58:38 DEBUG "resultType": "vector", 12:58:38 DEBUG "result": [ 12:58:38 DEBUG { 12:58:38 DEBUG "metric": { 12:58:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG [...] 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 1713527918, 12:58:38 DEBUG "0" 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG } 12:58:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527918" 12:58:38 INFO [http_cmd]: http status code OK 12:58:38 DEBUG --- status code --- 12:58:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:38 DEBUG --- http response --- 12:58:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:38 DEBUG ----- output ----- 12:58:38 DEBUG { 12:58:38 DEBUG "status": "success", 12:58:38 DEBUG "data": { 12:58:38 DEBUG "resultType": "vector", 12:58:38 DEBUG "result": [ 12:58:38 DEBUG { 12:58:38 DEBUG "metric": { 12:58:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG [...] 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG 12:58:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:58:38 DEBUG }, 12:58:38 DEBUG "value": [ 12:58:38 DEBUG 1713527918, 12:58:38 DEBUG "0" 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG ] 12:58:38 DEBUG } 12:58:38 DEBUG } 12:58:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713527920" 12:58:40 INFO [http_cmd]: http status code OK 12:58:40 DEBUG --- status code --- 12:58:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:40 DEBUG --- http response --- 12:58:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:40 DEBUG ----- output ----- 12:58:40 DEBUG { 12:58:40 DEBUG "status": "success", 12:58:40 DEBUG "data": { 12:58:40 DEBUG "resultType": "vector", 12:58:40 DEBUG "result": [ 12:58:40 DEBUG { 12:58:40 DEBUG "metric": { 12:58:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:58:40 DEBUG }, 12:58:40 DEBUG "value": [ 12:58:40 DEBUG 12:58:40 DEBUG 12:58:40 DEBUG 12:58:40 DEBUG [...] 12:58:40 DEBUG 12:58:40 DEBUG 12:58:40 DEBUG 12:58:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:58:40 DEBUG }, 12:58:40 DEBUG "value": [ 12:58:40 DEBUG 1713527920, 12:58:40 DEBUG "3533934592" 12:58:40 DEBUG ] 12:58:40 DEBUG } 12:58:40 DEBUG ] 12:58:40 DEBUG } 12:58:40 DEBUG } 12:58:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527921" 12:58:41 INFO [http_cmd]: http status code OK 12:58:41 DEBUG --- status code --- 12:58:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:41 DEBUG --- http response --- 12:58:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:58:41 DEBUG ----- output ----- 12:58:41 DEBUG { 12:58:41 DEBUG "status": "success", 12:58:41 DEBUG "data": { 12:58:41 DEBUG "resultType": "vector", 12:58:41 DEBUG "result": [ 12:58:41 DEBUG { 12:58:41 DEBUG "metric": { 12:58:41 DEBUG "pod": "ds-cts-0" 12:58:41 DEBUG }, 12:58:41 DEBUG "value": [ 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG [...] 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG "pod": "ds-cts-2" 12:58:41 DEBUG }, 12:58:41 DEBUG "value": [ 12:58:41 DEBUG 1713527921, 12:58:41 DEBUG "29.833333333333332" 12:58:41 DEBUG ] 12:58:41 DEBUG } 12:58:41 DEBUG ] 12:58:41 DEBUG } 12:58:41 DEBUG } 12:58:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713527921" 12:58:41 INFO [http_cmd]: http status code OK 12:58:41 DEBUG --- status code --- 12:58:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:41 DEBUG --- http response --- 12:58:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:41 DEBUG ----- output ----- 12:58:41 DEBUG { 12:58:41 DEBUG "status": "success", 12:58:41 DEBUG "data": { 12:58:41 DEBUG "resultType": "vector", 12:58:41 DEBUG "result": [ 12:58:41 DEBUG { 12:58:41 DEBUG "metric": { 12:58:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:58:41 DEBUG }, 12:58:41 DEBUG "value": [ 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG [...] 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:58:41 DEBUG }, 12:58:41 DEBUG "value": [ 12:58:41 DEBUG 1713527921, 12:58:41 DEBUG "55.19" 12:58:41 DEBUG ] 12:58:41 DEBUG } 12:58:41 DEBUG ] 12:58:41 DEBUG } 12:58:41 DEBUG } 12:58:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527921" 12:58:41 INFO [http_cmd]: http status code OK 12:58:41 DEBUG --- status code --- 12:58:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:41 DEBUG --- http response --- 12:58:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:58:41 DEBUG ----- output ----- 12:58:41 DEBUG { 12:58:41 DEBUG "status": "success", 12:58:41 DEBUG "data": { 12:58:41 DEBUG "resultType": "vector", 12:58:41 DEBUG "result": [ 12:58:41 DEBUG { 12:58:41 DEBUG "metric": { 12:58:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:58:41 DEBUG }, 12:58:41 DEBUG "value": [ 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG [...] 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG "pod": "ds-idrepo-2" 12:58:41 DEBUG }, 12:58:41 DEBUG "value": [ 12:58:41 DEBUG 1713527921, 12:58:41 DEBUG "0.6994414682874206" 12:58:41 DEBUG ] 12:58:41 DEBUG } 12:58:41 DEBUG ] 12:58:41 DEBUG } 12:58:41 DEBUG } 12:58:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527921" 12:58:41 INFO [http_cmd]: http status code OK 12:58:41 DEBUG --- status code --- 12:58:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:41 DEBUG --- http response --- 12:58:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:41 DEBUG ----- output ----- 12:58:41 DEBUG { 12:58:41 DEBUG "status": "success", 12:58:41 DEBUG "data": { 12:58:41 DEBUG "resultType": "vector", 12:58:41 DEBUG "result": [ 12:58:41 DEBUG { 12:58:41 DEBUG "metric": { 12:58:41 DEBUG "pod": "ds-idrepo-0" 12:58:41 DEBUG }, 12:58:41 DEBUG "value": [ 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG [...] 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG 12:58:41 DEBUG "pod": "ds-idrepo-2" 12:58:41 DEBUG }, 12:58:41 DEBUG "value": [ 12:58:41 DEBUG 1713527921, 12:58:41 DEBUG "0" 12:58:41 DEBUG ] 12:58:41 DEBUG } 12:58:41 DEBUG ] 12:58:41 DEBUG } 12:58:41 DEBUG } 12:58:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527922" 12:58:42 INFO [http_cmd]: http status code OK 12:58:42 DEBUG --- status code --- 12:58:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:42 DEBUG --- http response --- 12:58:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:58:42 DEBUG ----- output ----- 12:58:42 DEBUG { 12:58:42 DEBUG "status": "success", 12:58:42 DEBUG "data": { 12:58:42 DEBUG "resultType": "vector", 12:58:42 DEBUG "result": [ 12:58:42 DEBUG { 12:58:42 DEBUG "metric": { 12:58:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG [...] 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG "pod": "ds-idrepo-2" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 1713527922, 12:58:42 DEBUG "27185.533845789174" 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG } 12:58:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527922" 12:58:42 INFO [http_cmd]: http status code OK 12:58:42 DEBUG --- status code --- 12:58:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:42 DEBUG --- http response --- 12:58:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:42 DEBUG ----- output ----- 12:58:42 DEBUG { 12:58:42 DEBUG "status": "success", 12:58:42 DEBUG "data": { 12:58:42 DEBUG "resultType": "vector", 12:58:42 DEBUG "result": [ 12:58:42 DEBUG { 12:58:42 DEBUG "metric": { 12:58:42 DEBUG "instance": "10.142.0.107:9100" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG [...] 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG "instance": "10.142.0.43:9100" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 1713527922, 12:58:42 DEBUG "1591168640000" 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG } 12:58:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527922" 12:58:42 INFO [http_cmd]: http status code OK 12:58:42 DEBUG --- status code --- 12:58:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:42 DEBUG --- http response --- 12:58:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:58:42 DEBUG ----- output ----- 12:58:42 DEBUG { 12:58:42 DEBUG "status": "success", 12:58:42 DEBUG "data": { 12:58:42 DEBUG "resultType": "vector", 12:58:42 DEBUG "result": [ 12:58:42 DEBUG { 12:58:42 DEBUG "metric": { 12:58:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG [...] 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 1713527922, 12:58:42 DEBUG "0" 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG } 12:58:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527922" 12:58:42 INFO [http_cmd]: http status code OK 12:58:42 DEBUG --- status code --- 12:58:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:42 DEBUG --- http response --- 12:58:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:42 DEBUG ----- output ----- 12:58:42 DEBUG { 12:58:42 DEBUG "status": "success", 12:58:42 DEBUG "data": { 12:58:42 DEBUG "resultType": "vector", 12:58:42 DEBUG "result": [ 12:58:42 DEBUG { 12:58:42 DEBUG "metric": { 12:58:42 DEBUG "instance": "10.142.0.107:9100" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG [...] 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG "instance": "10.142.0.43:9100" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 1713527922, 12:58:42 DEBUG "16293807.966666667" 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG } 12:58:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527922" 12:58:42 INFO [http_cmd]: http status code OK 12:58:42 DEBUG --- status code --- 12:58:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:58:42 DEBUG --- http response --- 12:58:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:58:42 DEBUG ----- output ----- 12:58:42 DEBUG { 12:58:42 DEBUG "status": "success", 12:58:42 DEBUG "data": { 12:58:42 DEBUG "resultType": "vector", 12:58:42 DEBUG "result": [ 12:58:42 DEBUG { 12:58:42 DEBUG "metric": { 12:58:42 DEBUG "instance": "10.142.0.107:9100" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG [...] 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG 12:58:42 DEBUG "instance": "10.142.0.43:9100" 12:58:42 DEBUG }, 12:58:42 DEBUG "value": [ 12:58:42 DEBUG 1713527922, 12:58:42 DEBUG "16300726.399999999" 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG ] 12:58:42 DEBUG } 12:58:42 DEBUG } 12:58:56 INFO 12:58:56 INFO [loop_until]: kubectl --namespace=xlou top pods 12:58:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:58:56 INFO [loop_until]: OK (rc = 0) 12:58:56 DEBUG --- stdout --- 12:58:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2404m 9432Mi am-6fd554f8f7-hvvkw 2453m 9008Mi am-6fd554f8f7-pmj4b 2537m 9128Mi ds-cts-0 9980m 13595Mi ds-cts-1 9216m 13295Mi ds-cts-2 9923m 13360Mi ds-idrepo-0 687m 3364Mi ds-idrepo-1 710m 3263Mi ds-idrepo-2 695m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-747d57dcb8-lclbf 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 998m 3731Mi 12:58:56 DEBUG --- stderr --- 12:58:56 DEBUG 12:59:02 INFO 12:59:02 INFO [loop_until]: kubectl --namespace=xlou top node 12:59:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:59:02 INFO [loop_until]: OK (rc = 0) 12:59:02 DEBUG --- stdout --- 12:59:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6134Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 738m 2% 6897Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2416m 8% 11779Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3237Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10096m 33% 16685Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5205Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2541m 8% 12258Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 74m 0% 3130Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2659m 8% 11377Mi 10% gke-xlou-cdm-ds-0f601751-j488 744m 2% 6375Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9209m 30% 17737Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 768m 2% 7016Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 10020m 33% 15504Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 7838Mi 13% 12:59:02 DEBUG --- stderr --- 12:59:02 DEBUG 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527977" 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527977" 12:59:37 INFO [http_cmd]: http status code OK 12:59:37 DEBUG --- status code --- 12:59:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:37 DEBUG --- http response --- 12:59:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:59:37 DEBUG ----- output ----- 12:59:37 DEBUG { 12:59:37 DEBUG "status": "success", 12:59:37 DEBUG "data": { 12:59:37 DEBUG "resultType": "vector", 12:59:37 DEBUG "result": [ 12:59:37 DEBUG { 12:59:37 DEBUG "metric": { 12:59:37 DEBUG "pod": "ds-cts-0" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG [...] 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG "pod": "ds-cts-2" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 1713527977, 12:59:37 DEBUG "0.0004315666666666666" 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG } 12:59:37 INFO [http_cmd]: http status code OK 12:59:37 DEBUG --- status code --- 12:59:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:37 DEBUG --- http response --- 12:59:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713527977" 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527977" 12:59:37 ERROR [http_cmd]: ERROR 12:59:37 ERROR --- status code --- 12:59:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 12:59:37 ERROR --- http response --- 12:59:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 12:59:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 12:59:37 INFO [http_cmd]: http status code OK 12:59:37 DEBUG --- status code --- 12:59:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:37 DEBUG --- http response --- 12:59:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:59:37 DEBUG ----- output ----- 12:59:37 DEBUG { 12:59:37 DEBUG "status": "success", 12:59:37 DEBUG "data": { 12:59:37 DEBUG "resultType": "vector", 12:59:37 DEBUG "result": [ 12:59:37 DEBUG { 12:59:37 DEBUG "metric": { 12:59:37 DEBUG "pod": "ds-cts-0" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG [...] 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG "pod": "ds-cts-2" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 1713527977, 12:59:37 DEBUG "28.5" 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG } 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527977" 12:59:37 INFO [http_cmd]: http status code OK 12:59:37 DEBUG --- status code --- 12:59:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:37 DEBUG --- http response --- 12:59:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 12:59:37 DEBUG ----- output ----- 12:59:37 DEBUG { 12:59:37 DEBUG "status": "success", 12:59:37 DEBUG "data": { 12:59:37 DEBUG "resultType": "vector", 12:59:37 DEBUG "result": [ 12:59:37 DEBUG { 12:59:37 DEBUG "metric": { 12:59:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG [...] 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 1713527977, 12:59:37 DEBUG "729.8324820484971" 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG } 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713527977" 12:59:37 INFO [http_cmd]: http status code OK 12:59:37 DEBUG --- status code --- 12:59:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:37 DEBUG --- http response --- 12:59:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:59:37 DEBUG ----- output ----- 12:59:37 DEBUG { 12:59:37 DEBUG "status": "success", 12:59:37 DEBUG "data": { 12:59:37 DEBUG "resultType": "vector", 12:59:37 DEBUG "result": [ 12:59:37 DEBUG { 12:59:37 DEBUG "metric": { 12:59:37 DEBUG "pod": "ds-idrepo-0" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG [...] 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG "pod": "ds-cts-2" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 1713527977, 12:59:37 DEBUG "0" 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG } 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527977" 12:59:37 INFO [http_cmd]: http status code OK 12:59:37 DEBUG --- status code --- 12:59:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:37 DEBUG --- http response --- 12:59:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:37 DEBUG ----- output ----- 12:59:37 DEBUG { 12:59:37 DEBUG "status": "success", 12:59:37 DEBUG "data": { 12:59:37 DEBUG "resultType": "vector", 12:59:37 DEBUG "result": [ 12:59:37 DEBUG { 12:59:37 DEBUG "metric": { 12:59:37 DEBUG "pod": "ds-idrepo-0" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG [...] 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG "pod": "ds-idrepo-2" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 1713527977, 12:59:37 DEBUG "0" 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG } 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527977" 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527977" 12:59:37 INFO [http_cmd]: http status code OK 12:59:37 INFO [http_cmd]: http status code OK 12:59:37 DEBUG --- status code --- 12:59:37 DEBUG --- status code --- 12:59:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:37 DEBUG --- http response --- 12:59:37 DEBUG --- http response --- 12:59:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:37 DEBUG ----- output ----- 12:59:37 DEBUG { 12:59:37 DEBUG "status": "success", 12:59:37 DEBUG "data": { 12:59:37 DEBUG "resultType": "vector", 12:59:37 DEBUG "result": [ 12:59:37 DEBUG { 12:59:37 DEBUG "metric": { 12:59:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG [...] 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 1713527977, 12:59:37 DEBUG "1780733" 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG } 12:59:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:37 DEBUG ----- output ----- 12:59:37 DEBUG { 12:59:37 DEBUG "status": "success", 12:59:37 DEBUG "data": { 12:59:37 DEBUG "resultType": "vector", 12:59:37 DEBUG "result": [ 12:59:37 DEBUG { 12:59:37 DEBUG "metric": { 12:59:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG [...] 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 1713527977, 12:59:37 DEBUG "2502.9666666666667" 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG } 12:59:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713527977" 12:59:37 INFO [http_cmd]: http status code OK 12:59:37 DEBUG --- status code --- 12:59:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:37 DEBUG --- http response --- 12:59:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:37 DEBUG ----- output ----- 12:59:37 DEBUG { 12:59:37 DEBUG "status": "success", 12:59:37 DEBUG "data": { 12:59:37 DEBUG "resultType": "vector", 12:59:37 DEBUG "result": [ 12:59:37 DEBUG { 12:59:37 DEBUG "metric": { 12:59:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG [...] 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG 12:59:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:59:37 DEBUG }, 12:59:37 DEBUG "value": [ 12:59:37 DEBUG 1713527977, 12:59:37 DEBUG "1.0277126829356422" 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG ] 12:59:37 DEBUG } 12:59:37 DEBUG } 12:59:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527978" 12:59:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527978" 12:59:38 INFO [http_cmd]: http status code OK 12:59:38 INFO [http_cmd]: http status code OK 12:59:38 DEBUG --- status code --- 12:59:38 DEBUG --- status code --- 12:59:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:38 DEBUG --- http response --- 12:59:38 DEBUG --- http response --- 12:59:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 12:59:38 DEBUG ----- output ----- 12:59:38 DEBUG ----- output ----- 12:59:38 DEBUG { 12:59:38 DEBUG { 12:59:38 DEBUG "status": "success", 12:59:38 DEBUG "status": "success", 12:59:38 DEBUG "data": { 12:59:38 DEBUG "data": { 12:59:38 DEBUG "resultType": "vector", 12:59:38 DEBUG "resultType": "vector", 12:59:38 DEBUG "result": [ 12:59:38 DEBUG "result": [ 12:59:38 DEBUG { 12:59:38 DEBUG { 12:59:38 DEBUG "metric": { 12:59:38 DEBUG "metric": { 12:59:38 DEBUG "pod": "ds-idrepo-0" 12:59:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:59:38 DEBUG }, 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG [...] 12:59:38 DEBUG [...] 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG "pod": "ds-idrepo-2" 12:59:38 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 12:59:38 DEBUG }, 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 1713527978, 12:59:38 DEBUG 1713527978, 12:59:38 DEBUG "0" 12:59:38 DEBUG "0" 12:59:38 DEBUG ] 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG } 12:59:38 DEBUG ] 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG } 12:59:38 DEBUG } 12:59:38 DEBUG } 12:59:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527978" 12:59:38 INFO [http_cmd]: http status code OK 12:59:38 DEBUG --- status code --- 12:59:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:38 DEBUG --- http response --- 12:59:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:38 DEBUG ----- output ----- 12:59:38 DEBUG { 12:59:38 DEBUG "status": "success", 12:59:38 DEBUG "data": { 12:59:38 DEBUG "resultType": "vector", 12:59:38 DEBUG "result": [ 12:59:38 DEBUG { 12:59:38 DEBUG "metric": { 12:59:38 DEBUG "instance": "10.142.0.107:9100" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG [...] 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG "instance": "10.142.0.43:9100" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 1713527978, 12:59:38 DEBUG "1.0513333333333301" 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG } 12:59:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527978" 12:59:38 INFO [http_cmd]: http status code OK 12:59:38 DEBUG --- status code --- 12:59:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:38 DEBUG --- http response --- 12:59:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:38 DEBUG ----- output ----- 12:59:38 DEBUG { 12:59:38 DEBUG "status": "success", 12:59:38 DEBUG "data": { 12:59:38 DEBUG "resultType": "vector", 12:59:38 DEBUG "result": [ 12:59:38 DEBUG { 12:59:38 DEBUG "metric": { 12:59:38 DEBUG "instance": "10.142.0.107:9100" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG [...] 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG "instance": "10.142.0.43:9100" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 1713527978, 12:59:38 DEBUG "0.6843333333333883" 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG } 12:59:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527978" 12:59:38 INFO [http_cmd]: http status code OK 12:59:38 DEBUG --- status code --- 12:59:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:38 DEBUG --- http response --- 12:59:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:38 DEBUG ----- output ----- 12:59:38 DEBUG { 12:59:38 DEBUG "status": "success", 12:59:38 DEBUG "data": { 12:59:38 DEBUG "resultType": "vector", 12:59:38 DEBUG "result": [ 12:59:38 DEBUG { 12:59:38 DEBUG "metric": { 12:59:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG [...] 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 1713527978, 12:59:38 DEBUG "0" 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG } 12:59:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527978" 12:59:38 INFO [http_cmd]: http status code OK 12:59:38 DEBUG --- status code --- 12:59:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:38 DEBUG --- http response --- 12:59:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:38 DEBUG ----- output ----- 12:59:38 DEBUG { 12:59:38 DEBUG "status": "success", 12:59:38 DEBUG "data": { 12:59:38 DEBUG "resultType": "vector", 12:59:38 DEBUG "result": [ 12:59:38 DEBUG { 12:59:38 DEBUG "metric": { 12:59:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG [...] 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 1713527978, 12:59:38 DEBUG "0" 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG } 12:59:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527978" 12:59:38 INFO [http_cmd]: http status code OK 12:59:38 DEBUG --- status code --- 12:59:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:38 DEBUG --- http response --- 12:59:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:38 DEBUG ----- output ----- 12:59:38 DEBUG { 12:59:38 DEBUG "status": "success", 12:59:38 DEBUG "data": { 12:59:38 DEBUG "resultType": "vector", 12:59:38 DEBUG "result": [ 12:59:38 DEBUG { 12:59:38 DEBUG "metric": { 12:59:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG [...] 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG 12:59:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:59:38 DEBUG }, 12:59:38 DEBUG "value": [ 12:59:38 DEBUG 1713527978, 12:59:38 DEBUG "0" 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG ] 12:59:38 DEBUG } 12:59:38 DEBUG } 12:59:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713527980" 12:59:40 INFO [http_cmd]: http status code OK 12:59:40 DEBUG --- status code --- 12:59:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:40 DEBUG --- http response --- 12:59:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:40 DEBUG ----- output ----- 12:59:40 DEBUG { 12:59:40 DEBUG "status": "success", 12:59:40 DEBUG "data": { 12:59:40 DEBUG "resultType": "vector", 12:59:40 DEBUG "result": [ 12:59:40 DEBUG { 12:59:40 DEBUG "metric": { 12:59:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 12:59:40 DEBUG }, 12:59:40 DEBUG "value": [ 12:59:40 DEBUG 12:59:40 DEBUG 12:59:40 DEBUG 12:59:40 DEBUG [...] 12:59:40 DEBUG 12:59:40 DEBUG 12:59:40 DEBUG 12:59:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 12:59:40 DEBUG }, 12:59:40 DEBUG "value": [ 12:59:40 DEBUG 1713527980, 12:59:40 DEBUG "4422901760" 12:59:40 DEBUG ] 12:59:40 DEBUG } 12:59:40 DEBUG ] 12:59:40 DEBUG } 12:59:40 DEBUG } 12:59:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527981" 12:59:41 INFO [http_cmd]: http status code OK 12:59:41 DEBUG --- status code --- 12:59:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:41 DEBUG --- http response --- 12:59:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 12:59:41 DEBUG ----- output ----- 12:59:41 DEBUG { 12:59:41 DEBUG "status": "success", 12:59:41 DEBUG "data": { 12:59:41 DEBUG "resultType": "vector", 12:59:41 DEBUG "result": [ 12:59:41 DEBUG { 12:59:41 DEBUG "metric": { 12:59:41 DEBUG "pod": "ds-cts-0" 12:59:41 DEBUG }, 12:59:41 DEBUG "value": [ 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG [...] 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG "pod": "ds-cts-2" 12:59:41 DEBUG }, 12:59:41 DEBUG "value": [ 12:59:41 DEBUG 1713527981, 12:59:41 DEBUG "30" 12:59:41 DEBUG ] 12:59:41 DEBUG } 12:59:41 DEBUG ] 12:59:41 DEBUG } 12:59:41 DEBUG } 12:59:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713527981" 12:59:41 INFO [http_cmd]: http status code OK 12:59:41 DEBUG --- status code --- 12:59:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:41 DEBUG --- http response --- 12:59:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:41 DEBUG ----- output ----- 12:59:41 DEBUG { 12:59:41 DEBUG "status": "success", 12:59:41 DEBUG "data": { 12:59:41 DEBUG "resultType": "vector", 12:59:41 DEBUG "result": [ 12:59:41 DEBUG { 12:59:41 DEBUG "metric": { 12:59:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:59:41 DEBUG }, 12:59:41 DEBUG "value": [ 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG [...] 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:59:41 DEBUG }, 12:59:41 DEBUG "value": [ 12:59:41 DEBUG 1713527981, 12:59:41 DEBUG "55.19" 12:59:41 DEBUG ] 12:59:41 DEBUG } 12:59:41 DEBUG ] 12:59:41 DEBUG } 12:59:41 DEBUG } 12:59:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527981" 12:59:41 INFO [http_cmd]: http status code OK 12:59:41 DEBUG --- status code --- 12:59:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:41 DEBUG --- http response --- 12:59:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:41 DEBUG ----- output ----- 12:59:41 DEBUG { 12:59:41 DEBUG "status": "success", 12:59:41 DEBUG "data": { 12:59:41 DEBUG "resultType": "vector", 12:59:41 DEBUG "result": [ 12:59:41 DEBUG { 12:59:41 DEBUG "metric": { 12:59:41 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:59:41 DEBUG }, 12:59:41 DEBUG "value": [ 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG [...] 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 12:59:41 DEBUG }, 12:59:41 DEBUG "value": [ 12:59:41 DEBUG 1713527981, 12:59:41 DEBUG "0" 12:59:41 DEBUG ] 12:59:41 DEBUG } 12:59:41 DEBUG ] 12:59:41 DEBUG } 12:59:41 DEBUG } 12:59:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527981" 12:59:41 INFO [http_cmd]: http status code OK 12:59:41 DEBUG --- status code --- 12:59:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:41 DEBUG --- http response --- 12:59:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:41 DEBUG ----- output ----- 12:59:41 DEBUG { 12:59:41 DEBUG "status": "success", 12:59:41 DEBUG "data": { 12:59:41 DEBUG "resultType": "vector", 12:59:41 DEBUG "result": [ 12:59:41 DEBUG { 12:59:41 DEBUG "metric": { 12:59:41 DEBUG "pod": "ds-idrepo-0" 12:59:41 DEBUG }, 12:59:41 DEBUG "value": [ 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG [...] 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG 12:59:41 DEBUG "pod": "ds-idrepo-2" 12:59:41 DEBUG }, 12:59:41 DEBUG "value": [ 12:59:41 DEBUG 1713527981, 12:59:41 DEBUG "0" 12:59:41 DEBUG ] 12:59:41 DEBUG } 12:59:41 DEBUG ] 12:59:41 DEBUG } 12:59:41 DEBUG } 12:59:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527982" 12:59:42 INFO [http_cmd]: http status code OK 12:59:42 DEBUG --- status code --- 12:59:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:42 DEBUG --- http response --- 12:59:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 12:59:42 DEBUG ----- output ----- 12:59:42 DEBUG { 12:59:42 DEBUG "status": "success", 12:59:42 DEBUG "data": { 12:59:42 DEBUG "resultType": "vector", 12:59:42 DEBUG "result": [ 12:59:42 DEBUG { 12:59:42 DEBUG "metric": { 12:59:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG [...] 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 1713527982, 12:59:42 DEBUG "0" 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG } 12:59:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527982" 12:59:42 INFO [http_cmd]: http status code OK 12:59:42 DEBUG --- status code --- 12:59:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:42 DEBUG --- http response --- 12:59:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:42 DEBUG ----- output ----- 12:59:42 DEBUG { 12:59:42 DEBUG "status": "success", 12:59:42 DEBUG "data": { 12:59:42 DEBUG "resultType": "vector", 12:59:42 DEBUG "result": [ 12:59:42 DEBUG { 12:59:42 DEBUG "metric": { 12:59:42 DEBUG "instance": "10.142.0.107:9100" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG [...] 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG "instance": "10.142.0.43:9100" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 1713527982, 12:59:42 DEBUG "1592155315200" 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG } 12:59:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713527982" 12:59:42 INFO [http_cmd]: http status code OK 12:59:42 DEBUG --- status code --- 12:59:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:42 DEBUG --- http response --- 12:59:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 12:59:42 DEBUG ----- output ----- 12:59:42 DEBUG { 12:59:42 DEBUG "status": "success", 12:59:42 DEBUG "data": { 12:59:42 DEBUG "resultType": "vector", 12:59:42 DEBUG "result": [ 12:59:42 DEBUG { 12:59:42 DEBUG "metric": { 12:59:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG [...] 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 1713527982, 12:59:42 DEBUG "0" 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG } 12:59:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527982" 12:59:42 INFO [http_cmd]: http status code OK 12:59:42 DEBUG --- status code --- 12:59:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:42 DEBUG --- http response --- 12:59:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:42 DEBUG ----- output ----- 12:59:42 DEBUG { 12:59:42 DEBUG "status": "success", 12:59:42 DEBUG "data": { 12:59:42 DEBUG "resultType": "vector", 12:59:42 DEBUG "result": [ 12:59:42 DEBUG { 12:59:42 DEBUG "metric": { 12:59:42 DEBUG "instance": "10.142.0.107:9100" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG [...] 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG "instance": "10.142.0.43:9100" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 1713527982, 12:59:42 DEBUG "16339262.033333333" 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG } 12:59:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713527982" 12:59:42 INFO [http_cmd]: http status code OK 12:59:42 DEBUG --- status code --- 12:59:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 12:59:42 DEBUG --- http response --- 12:59:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 12:59:42 DEBUG ----- output ----- 12:59:42 DEBUG { 12:59:42 DEBUG "status": "success", 12:59:42 DEBUG "data": { 12:59:42 DEBUG "resultType": "vector", 12:59:42 DEBUG "result": [ 12:59:42 DEBUG { 12:59:42 DEBUG "metric": { 12:59:42 DEBUG "instance": "10.142.0.107:9100" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG [...] 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG 12:59:42 DEBUG "instance": "10.142.0.43:9100" 12:59:42 DEBUG }, 12:59:42 DEBUG "value": [ 12:59:42 DEBUG 1713527982, 12:59:42 DEBUG "16346438.733333334" 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG ] 12:59:42 DEBUG } 12:59:42 DEBUG } 12:59:56 INFO 12:59:56 INFO [loop_until]: kubectl --namespace=xlou top pods 12:59:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 12:59:56 INFO [loop_until]: OK (rc = 0) 12:59:56 DEBUG --- stdout --- 12:59:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2376m 9437Mi am-6fd554f8f7-hvvkw 2436m 9006Mi am-6fd554f8f7-pmj4b 2502m 9133Mi ds-cts-0 9644m 13420Mi ds-cts-1 8898m 13115Mi ds-cts-2 9935m 13074Mi ds-idrepo-0 686m 3365Mi ds-idrepo-1 707m 3262Mi ds-idrepo-2 699m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-747d57dcb8-lclbf 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 437m 3979Mi 12:59:56 DEBUG --- stderr --- 12:59:56 DEBUG 13:00:02 INFO 13:00:02 INFO [loop_until]: kubectl --namespace=xlou top node 13:00:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:00:02 INFO [loop_until]: OK (rc = 0) 13:00:02 DEBUG --- stdout --- 13:00:02 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6131Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 750m 2% 6894Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2412m 8% 11775Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3241Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9862m 33% 16181Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2564m 8% 12260Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3115Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2667m 8% 11372Mi 10% gke-xlou-cdm-ds-0f601751-j488 735m 2% 6372Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8979m 30% 17747Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 774m 2% 7026Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9054m 30% 16032Mi 14% gke-xlou-cdm-frontend-a8771548-i82n 756m 4% 8042Mi 13% 13:00:02 DEBUG --- stderr --- 13:00:02 DEBUG 13:00:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528037" 13:00:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528037" 13:00:37 INFO [http_cmd]: http status code OK 13:00:37 INFO [http_cmd]: http status code OK 13:00:37 DEBUG --- status code --- 13:00:37 DEBUG --- status code --- 13:00:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:37 DEBUG --- http response --- 13:00:37 DEBUG --- http response --- 13:00:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:00:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:00:37 DEBUG ----- output ----- 13:00:37 DEBUG { 13:00:37 DEBUG "status": "success", 13:00:37 DEBUG "data": { 13:00:37 DEBUG "resultType": "vector", 13:00:37 DEBUG "result": [ 13:00:37 DEBUG { 13:00:37 DEBUG "metric": { 13:00:37 DEBUG "pod": "ds-cts-0" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG [...] 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG "pod": "ds-cts-2" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 1713528037, 13:00:37 DEBUG "0.0001666666666666667" 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG } 13:00:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528037" 13:00:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528037" 13:00:37 INFO [http_cmd]: http status code OK 13:00:37 DEBUG --- status code --- 13:00:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:37 DEBUG --- http response --- 13:00:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:00:37 DEBUG ----- output ----- 13:00:37 DEBUG { 13:00:37 DEBUG "status": "success", 13:00:37 DEBUG "data": { 13:00:37 DEBUG "resultType": "vector", 13:00:37 DEBUG "result": [ 13:00:37 DEBUG { 13:00:37 DEBUG "metric": { 13:00:37 DEBUG "pod": "ds-cts-0" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG [...] 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG "pod": "ds-cts-2" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 1713528037, 13:00:37 DEBUG "33.666666666666664" 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG } 13:00:37 ERROR [http_cmd]: ERROR 13:00:37 ERROR --- status code --- 13:00:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:00:37 ERROR --- http response --- 13:00:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:00:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:00:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528037" 13:00:37 INFO [http_cmd]: http status code OK 13:00:37 DEBUG --- status code --- 13:00:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:37 DEBUG --- http response --- 13:00:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:00:37 DEBUG ----- output ----- 13:00:37 DEBUG { 13:00:37 DEBUG "status": "success", 13:00:37 DEBUG "data": { 13:00:37 DEBUG "resultType": "vector", 13:00:37 DEBUG "result": [ 13:00:37 DEBUG { 13:00:37 DEBUG "metric": { 13:00:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG [...] 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 1713528037, 13:00:37 DEBUG "3382.1858270382363" 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG } 13:00:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528037" 13:00:37 INFO [http_cmd]: http status code OK 13:00:37 DEBUG --- status code --- 13:00:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:37 DEBUG --- http response --- 13:00:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:00:37 DEBUG ----- output ----- 13:00:37 DEBUG { 13:00:37 DEBUG "status": "success", 13:00:37 DEBUG "data": { 13:00:37 DEBUG "resultType": "vector", 13:00:37 DEBUG "result": [ 13:00:37 DEBUG { 13:00:37 DEBUG "metric": { 13:00:37 DEBUG "pod": "ds-idrepo-0" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG [...] 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG "pod": "ds-cts-2" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 1713528037, 13:00:37 DEBUG "0" 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG } 13:00:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528037" 13:00:37 INFO [http_cmd]: http status code OK 13:00:37 DEBUG --- status code --- 13:00:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:37 DEBUG --- http response --- 13:00:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:37 DEBUG ----- output ----- 13:00:37 DEBUG { 13:00:37 DEBUG "status": "success", 13:00:37 DEBUG "data": { 13:00:37 DEBUG "resultType": "vector", 13:00:37 DEBUG "result": [ 13:00:37 DEBUG { 13:00:37 DEBUG "metric": { 13:00:37 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG [...] 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG 13:00:37 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:00:37 DEBUG }, 13:00:37 DEBUG "value": [ 13:00:37 DEBUG 1713528037, 13:00:37 DEBUG "1630698.5" 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG ] 13:00:37 DEBUG } 13:00:37 DEBUG } 13:00:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528038" 13:00:38 INFO [http_cmd]: http status code OK 13:00:38 DEBUG --- status code --- 13:00:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:38 DEBUG --- http response --- 13:00:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:38 DEBUG ----- output ----- 13:00:38 DEBUG { 13:00:38 DEBUG "status": "success", 13:00:38 DEBUG "data": { 13:00:38 DEBUG "resultType": "vector", 13:00:38 DEBUG "result": [ 13:00:38 DEBUG { 13:00:38 DEBUG "metric": { 13:00:38 DEBUG "pod": "ds-idrepo-0" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG [...] 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG "pod": "ds-idrepo-2" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 1713528038, 13:00:38 DEBUG "0" 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG } 13:00:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528038" 13:00:38 INFO [http_cmd]: http status code OK 13:00:38 DEBUG --- status code --- 13:00:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:38 DEBUG --- http response --- 13:00:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:38 DEBUG ----- output ----- 13:00:38 DEBUG { 13:00:38 DEBUG "status": "success", 13:00:38 DEBUG "data": { 13:00:38 DEBUG "resultType": "vector", 13:00:38 DEBUG "result": [ 13:00:38 DEBUG { 13:00:38 DEBUG "metric": { 13:00:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG [...] 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 1713528038, 13:00:38 DEBUG "2502.2" 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG } 13:00:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528038" 13:00:38 INFO [http_cmd]: http status code OK 13:00:38 DEBUG --- status code --- 13:00:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:38 DEBUG --- http response --- 13:00:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:38 DEBUG ----- output ----- 13:00:38 DEBUG { 13:00:38 DEBUG "status": "success", 13:00:38 DEBUG "data": { 13:00:38 DEBUG "resultType": "vector", 13:00:38 DEBUG "result": [ 13:00:38 DEBUG { 13:00:38 DEBUG "metric": { 13:00:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG [...] 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 1713528038, 13:00:38 DEBUG "1.0006682326452616" 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG } 13:00:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528038" 13:00:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528038" 13:00:38 INFO [http_cmd]: http status code OK 13:00:38 DEBUG --- status code --- 13:00:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:38 DEBUG --- http response --- 13:00:38 INFO [http_cmd]: http status code OK 13:00:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:38 DEBUG --- status code --- 13:00:38 DEBUG ----- output ----- 13:00:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:38 DEBUG { 13:00:38 DEBUG --- http response --- 13:00:38 DEBUG "status": "success", 13:00:38 DEBUG "data": { 13:00:38 DEBUG "resultType": "vector", 13:00:38 DEBUG "result": [ 13:00:38 DEBUG { 13:00:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:38 DEBUG "metric": { 13:00:38 DEBUG ----- output ----- 13:00:38 DEBUG "pod": "ds-idrepo-0" 13:00:38 DEBUG { 13:00:38 DEBUG }, 13:00:38 DEBUG "status": "success", 13:00:38 DEBUG "value": [ 13:00:38 DEBUG "data": { 13:00:38 DEBUG 13:00:38 DEBUG "resultType": "vector", 13:00:38 DEBUG 13:00:38 DEBUG "result": [ 13:00:38 DEBUG 13:00:38 DEBUG { 13:00:38 DEBUG [...] 13:00:38 DEBUG "metric": { 13:00:38 DEBUG 13:00:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:00:38 DEBUG 13:00:38 DEBUG }, 13:00:38 DEBUG 13:00:38 DEBUG "value": [ 13:00:38 DEBUG "pod": "ds-idrepo-2" 13:00:38 DEBUG 13:00:38 DEBUG }, 13:00:38 DEBUG 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 13:00:38 DEBUG 1713528038, 13:00:38 DEBUG [...] 13:00:38 DEBUG "0" 13:00:38 DEBUG 13:00:38 DEBUG ] 13:00:38 DEBUG 13:00:38 DEBUG } 13:00:38 DEBUG 13:00:38 DEBUG ] 13:00:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:00:38 DEBUG } 13:00:38 DEBUG }, 13:00:38 DEBUG } 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 1713528038, 13:00:38 DEBUG "0" 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG } 13:00:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528038" 13:00:38 INFO [http_cmd]: http status code OK 13:00:38 DEBUG --- status code --- 13:00:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:38 DEBUG --- http response --- 13:00:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:38 DEBUG ----- output ----- 13:00:38 DEBUG { 13:00:38 DEBUG "status": "success", 13:00:38 DEBUG "data": { 13:00:38 DEBUG "resultType": "vector", 13:00:38 DEBUG "result": [ 13:00:38 DEBUG { 13:00:38 DEBUG "metric": { 13:00:38 DEBUG "instance": "10.142.0.107:9100" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG [...] 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG "instance": "10.142.0.43:9100" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 1713528038, 13:00:38 DEBUG "0.8376666666666591" 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG } 13:00:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528038" 13:00:38 INFO [http_cmd]: http status code OK 13:00:38 DEBUG --- status code --- 13:00:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:38 DEBUG --- http response --- 13:00:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:38 DEBUG ----- output ----- 13:00:38 DEBUG { 13:00:38 DEBUG "status": "success", 13:00:38 DEBUG "data": { 13:00:38 DEBUG "resultType": "vector", 13:00:38 DEBUG "result": [ 13:00:38 DEBUG { 13:00:38 DEBUG "metric": { 13:00:38 DEBUG "instance": "10.142.0.107:9100" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG [...] 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG "instance": "10.142.0.43:9100" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 1713528038, 13:00:38 DEBUG "0.6835333333332784" 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG } 13:00:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528038" 13:00:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528038" 13:00:38 INFO [http_cmd]: http status code OK 13:00:38 DEBUG --- status code --- 13:00:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:38 DEBUG --- http response --- 13:00:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:38 DEBUG ----- output ----- 13:00:38 DEBUG { 13:00:38 DEBUG "status": "success", 13:00:38 DEBUG "data": { 13:00:38 DEBUG "resultType": "vector", 13:00:38 DEBUG "result": [ 13:00:38 DEBUG { 13:00:38 DEBUG "metric": { 13:00:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG [...] 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 1713528038, 13:00:38 DEBUG "0" 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG } 13:00:38 INFO [http_cmd]: http status code OK 13:00:38 DEBUG --- status code --- 13:00:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:38 DEBUG --- http response --- 13:00:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:38 DEBUG ----- output ----- 13:00:38 DEBUG { 13:00:38 DEBUG "status": "success", 13:00:38 DEBUG "data": { 13:00:38 DEBUG "resultType": "vector", 13:00:38 DEBUG "result": [ 13:00:38 DEBUG { 13:00:38 DEBUG "metric": { 13:00:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG [...] 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG 13:00:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:00:38 DEBUG }, 13:00:38 DEBUG "value": [ 13:00:38 DEBUG 1713528038, 13:00:38 DEBUG "0" 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG ] 13:00:38 DEBUG } 13:00:38 DEBUG } 13:00:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528039" 13:00:39 INFO [http_cmd]: http status code OK 13:00:39 DEBUG --- status code --- 13:00:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:39 DEBUG --- http response --- 13:00:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:39 DEBUG ----- output ----- 13:00:39 DEBUG { 13:00:39 DEBUG "status": "success", 13:00:39 DEBUG "data": { 13:00:39 DEBUG "resultType": "vector", 13:00:39 DEBUG "result": [ 13:00:39 DEBUG { 13:00:39 DEBUG "metric": { 13:00:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:00:39 DEBUG }, 13:00:39 DEBUG "value": [ 13:00:39 DEBUG 13:00:39 DEBUG 13:00:39 DEBUG 13:00:39 DEBUG [...] 13:00:39 DEBUG 13:00:39 DEBUG 13:00:39 DEBUG 13:00:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:00:39 DEBUG }, 13:00:39 DEBUG "value": [ 13:00:39 DEBUG 1713528039, 13:00:39 DEBUG "0" 13:00:39 DEBUG ] 13:00:39 DEBUG } 13:00:39 DEBUG ] 13:00:39 DEBUG } 13:00:39 DEBUG } 13:00:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528040" 13:00:40 INFO [http_cmd]: http status code OK 13:00:40 DEBUG --- status code --- 13:00:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:40 DEBUG --- http response --- 13:00:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:40 DEBUG ----- output ----- 13:00:40 DEBUG { 13:00:40 DEBUG "status": "success", 13:00:40 DEBUG "data": { 13:00:40 DEBUG "resultType": "vector", 13:00:40 DEBUG "result": [ 13:00:40 DEBUG { 13:00:40 DEBUG "metric": { 13:00:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:00:40 DEBUG }, 13:00:40 DEBUG "value": [ 13:00:40 DEBUG 13:00:40 DEBUG 13:00:40 DEBUG 13:00:40 DEBUG [...] 13:00:40 DEBUG 13:00:40 DEBUG 13:00:40 DEBUG 13:00:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:00:40 DEBUG }, 13:00:40 DEBUG "value": [ 13:00:40 DEBUG 1713528040, 13:00:40 DEBUG "4531122176" 13:00:40 DEBUG ] 13:00:40 DEBUG } 13:00:40 DEBUG ] 13:00:40 DEBUG } 13:00:40 DEBUG } 13:00:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528041" 13:00:41 INFO [http_cmd]: http status code OK 13:00:41 DEBUG --- status code --- 13:00:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:41 DEBUG --- http response --- 13:00:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:00:41 DEBUG ----- output ----- 13:00:41 DEBUG { 13:00:41 DEBUG "status": "success", 13:00:41 DEBUG "data": { 13:00:41 DEBUG "resultType": "vector", 13:00:41 DEBUG "result": [ 13:00:41 DEBUG { 13:00:41 DEBUG "metric": { 13:00:41 DEBUG "pod": "ds-cts-0" 13:00:41 DEBUG }, 13:00:41 DEBUG "value": [ 13:00:41 DEBUG 13:00:41 DEBUG 13:00:41 DEBUG 13:00:41 DEBUG [...] 13:00:41 DEBUG 13:00:41 DEBUG 13:00:41 DEBUG 13:00:41 DEBUG "pod": "ds-cts-2" 13:00:41 DEBUG }, 13:00:41 DEBUG "value": [ 13:00:41 DEBUG 1713528041, 13:00:41 DEBUG "30" 13:00:41 DEBUG ] 13:00:41 DEBUG } 13:00:41 DEBUG ] 13:00:41 DEBUG } 13:00:41 DEBUG } 13:00:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528041" 13:00:41 INFO [http_cmd]: http status code OK 13:00:41 DEBUG --- status code --- 13:00:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:41 DEBUG --- http response --- 13:00:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:41 DEBUG ----- output ----- 13:00:41 DEBUG { 13:00:41 DEBUG "status": "success", 13:00:41 DEBUG "data": { 13:00:41 DEBUG "resultType": "vector", 13:00:41 DEBUG "result": [ 13:00:41 DEBUG { 13:00:41 DEBUG "metric": { 13:00:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:00:41 DEBUG }, 13:00:41 DEBUG "value": [ 13:00:41 DEBUG 13:00:41 DEBUG 13:00:41 DEBUG 13:00:41 DEBUG [...] 13:00:41 DEBUG 13:00:41 DEBUG 13:00:41 DEBUG 13:00:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:00:41 DEBUG }, 13:00:41 DEBUG "value": [ 13:00:41 DEBUG 1713528041, 13:00:41 DEBUG "55.19" 13:00:41 DEBUG ] 13:00:41 DEBUG } 13:00:41 DEBUG ] 13:00:41 DEBUG } 13:00:41 DEBUG } 13:00:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528041" 13:00:42 INFO [http_cmd]: http status code OK 13:00:42 DEBUG --- status code --- 13:00:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:42 DEBUG --- http response --- 13:00:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:00:42 DEBUG ----- output ----- 13:00:42 DEBUG { 13:00:42 DEBUG "status": "success", 13:00:42 DEBUG "data": { 13:00:42 DEBUG "resultType": "vector", 13:00:42 DEBUG "result": [ 13:00:42 DEBUG { 13:00:42 DEBUG "metric": { 13:00:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG [...] 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 1713528041, 13:00:42 DEBUG "0" 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG } 13:00:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528042" 13:00:42 INFO [http_cmd]: http status code OK 13:00:42 DEBUG --- status code --- 13:00:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:42 DEBUG --- http response --- 13:00:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:42 DEBUG ----- output ----- 13:00:42 DEBUG { 13:00:42 DEBUG "status": "success", 13:00:42 DEBUG "data": { 13:00:42 DEBUG "resultType": "vector", 13:00:42 DEBUG "result": [ 13:00:42 DEBUG { 13:00:42 DEBUG "metric": { 13:00:42 DEBUG "pod": "ds-idrepo-0" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG [...] 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG "pod": "ds-idrepo-2" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 1713528042, 13:00:42 DEBUG "0" 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG } 13:00:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528042" 13:00:42 INFO [http_cmd]: http status code OK 13:00:42 DEBUG --- status code --- 13:00:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:42 DEBUG --- http response --- 13:00:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:00:42 DEBUG ----- output ----- 13:00:42 DEBUG { 13:00:42 DEBUG "status": "success", 13:00:42 DEBUG "data": { 13:00:42 DEBUG "resultType": "vector", 13:00:42 DEBUG "result": [ 13:00:42 DEBUG { 13:00:42 DEBUG "metric": { 13:00:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG [...] 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG "pod": "ds-cts-1" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 1713528042, 13:00:42 DEBUG "0" 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG } 13:00:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528042" 13:00:42 INFO [http_cmd]: http status code OK 13:00:42 DEBUG --- status code --- 13:00:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:42 DEBUG --- http response --- 13:00:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:42 DEBUG ----- output ----- 13:00:42 DEBUG { 13:00:42 DEBUG "status": "success", 13:00:42 DEBUG "data": { 13:00:42 DEBUG "resultType": "vector", 13:00:42 DEBUG "result": [ 13:00:42 DEBUG { 13:00:42 DEBUG "metric": { 13:00:42 DEBUG "instance": "10.142.0.107:9100" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG [...] 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG "instance": "10.142.0.43:9100" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 1713528042, 13:00:42 DEBUG "1593548876800" 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG } 13:00:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528042" 13:00:42 INFO [http_cmd]: http status code OK 13:00:42 DEBUG --- status code --- 13:00:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:42 DEBUG --- http response --- 13:00:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:00:42 DEBUG ----- output ----- 13:00:42 DEBUG { 13:00:42 DEBUG "status": "success", 13:00:42 DEBUG "data": { 13:00:42 DEBUG "resultType": "vector", 13:00:42 DEBUG "result": [ 13:00:42 DEBUG { 13:00:42 DEBUG "metric": { 13:00:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG [...] 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 1713528042, 13:00:42 DEBUG "0" 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG } 13:00:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528042" 13:00:42 INFO [http_cmd]: http status code OK 13:00:42 DEBUG --- status code --- 13:00:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:42 DEBUG --- http response --- 13:00:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:42 DEBUG ----- output ----- 13:00:42 DEBUG { 13:00:42 DEBUG "status": "success", 13:00:42 DEBUG "data": { 13:00:42 DEBUG "resultType": "vector", 13:00:42 DEBUG "result": [ 13:00:42 DEBUG { 13:00:42 DEBUG "metric": { 13:00:42 DEBUG "instance": "10.142.0.107:9100" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG [...] 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG "instance": "10.142.0.43:9100" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 1713528042, 13:00:42 DEBUG "16358875.766666668" 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG } 13:00:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528042" 13:00:42 INFO [http_cmd]: http status code OK 13:00:42 DEBUG --- status code --- 13:00:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:00:42 DEBUG --- http response --- 13:00:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:00:42 DEBUG ----- output ----- 13:00:42 DEBUG { 13:00:42 DEBUG "status": "success", 13:00:42 DEBUG "data": { 13:00:42 DEBUG "resultType": "vector", 13:00:42 DEBUG "result": [ 13:00:42 DEBUG { 13:00:42 DEBUG "metric": { 13:00:42 DEBUG "instance": "10.142.0.107:9100" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG [...] 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG 13:00:42 DEBUG "instance": "10.142.0.43:9100" 13:00:42 DEBUG }, 13:00:42 DEBUG "value": [ 13:00:42 DEBUG 1713528042, 13:00:42 DEBUG "16366018.633333333" 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG ] 13:00:42 DEBUG } 13:00:42 DEBUG } 13:00:56 INFO 13:00:56 INFO [loop_until]: kubectl --namespace=xlou top pods 13:00:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:00:56 INFO [loop_until]: OK (rc = 0) 13:00:56 DEBUG --- stdout --- 13:00:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2426m 9435Mi am-6fd554f8f7-hvvkw 2417m 9006Mi am-6fd554f8f7-pmj4b 2542m 9133Mi ds-cts-0 9599m 13384Mi ds-cts-1 9129m 12577Mi ds-cts-2 9730m 13005Mi ds-idrepo-0 698m 3365Mi ds-idrepo-1 654m 3263Mi ds-idrepo-2 710m 3191Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1173Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-747d57dcb8-lclbf 6m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1106m 4394Mi 13:00:56 DEBUG --- stderr --- 13:00:56 DEBUG 13:01:03 INFO 13:01:03 INFO [loop_until]: kubectl --namespace=xlou top node 13:01:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:01:03 INFO [loop_until]: OK (rc = 0) 13:01:03 DEBUG --- stdout --- 13:01:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6130Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 767m 2% 6893Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2422m 8% 11768Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3233Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9499m 31% 16730Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5161Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2489m 8% 12257Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 76m 0% 3095Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2684m 8% 11362Mi 10% gke-xlou-cdm-ds-0f601751-j488 763m 2% 6363Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8633m 28% 17470Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 794m 2% 7024Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9295m 31% 15929Mi 14% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 8714Mi 14% 13:01:03 DEBUG --- stderr --- 13:01:03 DEBUG 13:01:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528097" 13:01:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528097" 13:01:37 INFO [http_cmd]: http status code OK 13:01:37 INFO [http_cmd]: http status code OK 13:01:37 DEBUG --- status code --- 13:01:37 DEBUG --- status code --- 13:01:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:37 DEBUG --- http response --- 13:01:37 DEBUG --- http response --- 13:01:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:01:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:01:37 DEBUG ----- output ----- 13:01:37 DEBUG { 13:01:37 DEBUG "status": "success", 13:01:37 DEBUG "data": { 13:01:37 DEBUG "resultType": "vector", 13:01:37 DEBUG "result": [ 13:01:37 DEBUG { 13:01:37 DEBUG "metric": { 13:01:37 DEBUG "pod": "ds-cts-0" 13:01:37 DEBUG }, 13:01:37 DEBUG "value": [ 13:01:37 DEBUG 13:01:37 DEBUG 13:01:37 DEBUG 13:01:37 DEBUG [...] 13:01:37 DEBUG 13:01:37 DEBUG 13:01:37 DEBUG 13:01:37 DEBUG "pod": "ds-cts-2" 13:01:37 DEBUG }, 13:01:37 DEBUG "value": [ 13:01:37 DEBUG 1713528097, 13:01:37 DEBUG "0.000033333333333333335" 13:01:37 DEBUG ] 13:01:37 DEBUG } 13:01:37 DEBUG ] 13:01:37 DEBUG } 13:01:37 DEBUG } 13:01:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528097" 13:01:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528097" 13:01:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528097" 13:01:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528097" 13:01:37 INFO [http_cmd]: http status code OK 13:01:37 DEBUG --- status code --- 13:01:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:37 DEBUG --- http response --- 13:01:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:01:37 DEBUG ----- output ----- 13:01:37 DEBUG { 13:01:37 DEBUG "status": "success", 13:01:37 DEBUG "data": { 13:01:37 DEBUG "resultType": "vector", 13:01:37 DEBUG "result": [ 13:01:37 DEBUG { 13:01:37 DEBUG "metric": { 13:01:37 DEBUG "pod": "ds-cts-0" 13:01:37 DEBUG }, 13:01:37 DEBUG "value": [ 13:01:37 DEBUG 13:01:37 DEBUG 13:01:37 DEBUG 13:01:37 DEBUG [...] 13:01:37 DEBUG 13:01:37 DEBUG 13:01:37 DEBUG 13:01:37 DEBUG "pod": "ds-cts-2" 13:01:37 DEBUG }, 13:01:37 DEBUG "value": [ 13:01:37 DEBUG 1713528097, 13:01:37 DEBUG "33.166666666666664" 13:01:37 DEBUG ] 13:01:37 DEBUG } 13:01:37 DEBUG ] 13:01:37 DEBUG } 13:01:37 DEBUG } 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528097, 13:01:38 DEBUG "0" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 ERROR [http_cmd]: ERROR 13:01:38 ERROR --- status code --- 13:01:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:01:38 ERROR --- http response --- 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:01:38 DEBUG --- status code --- 13:01:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "pod": "ds-idrepo-0" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "pod": "ds-cts-2" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528097, 13:01:38 DEBUG "0" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528098" 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "1405618" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528098" 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "pod": "ds-idrepo-0" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "pod": "ds-idrepo-2" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "0" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528098" 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "2504.1666666666665" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528098" 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "1.0247627501658514" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528098" 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528098" 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "pod": "ds-idrepo-0" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "0" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "pod": "ds-idrepo-0" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "pod": "ds-idrepo-2" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "0" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528098" 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528098" 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "instance": "10.142.0.107:9100" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "instance": "10.142.0.43:9100" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "0.8006666666666699" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "instance": "10.142.0.107:9100" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "instance": "10.142.0.43:9100" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "0.6458333333333058" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528098" 13:01:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528098" 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "0" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:38 INFO [http_cmd]: http status code OK 13:01:38 DEBUG --- status code --- 13:01:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:38 DEBUG --- http response --- 13:01:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:38 DEBUG ----- output ----- 13:01:38 DEBUG { 13:01:38 DEBUG "status": "success", 13:01:38 DEBUG "data": { 13:01:38 DEBUG "resultType": "vector", 13:01:38 DEBUG "result": [ 13:01:38 DEBUG { 13:01:38 DEBUG "metric": { 13:01:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG [...] 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG 13:01:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:01:38 DEBUG }, 13:01:38 DEBUG "value": [ 13:01:38 DEBUG 1713528098, 13:01:38 DEBUG "0" 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG ] 13:01:38 DEBUG } 13:01:38 DEBUG } 13:01:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528099" 13:01:39 INFO [http_cmd]: http status code OK 13:01:39 DEBUG --- status code --- 13:01:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:39 DEBUG --- http response --- 13:01:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:39 DEBUG ----- output ----- 13:01:39 DEBUG { 13:01:39 DEBUG "status": "success", 13:01:39 DEBUG "data": { 13:01:39 DEBUG "resultType": "vector", 13:01:39 DEBUG "result": [ 13:01:39 DEBUG { 13:01:39 DEBUG "metric": { 13:01:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:01:39 DEBUG }, 13:01:39 DEBUG "value": [ 13:01:39 DEBUG 13:01:39 DEBUG 13:01:39 DEBUG 13:01:39 DEBUG [...] 13:01:39 DEBUG 13:01:39 DEBUG 13:01:39 DEBUG 13:01:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:01:39 DEBUG }, 13:01:39 DEBUG "value": [ 13:01:39 DEBUG 1713528099, 13:01:39 DEBUG "0" 13:01:39 DEBUG ] 13:01:39 DEBUG } 13:01:39 DEBUG ] 13:01:39 DEBUG } 13:01:39 DEBUG } 13:01:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528100" 13:01:40 INFO [http_cmd]: http status code OK 13:01:40 DEBUG --- status code --- 13:01:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:40 DEBUG --- http response --- 13:01:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:40 DEBUG ----- output ----- 13:01:40 DEBUG { 13:01:40 DEBUG "status": "success", 13:01:40 DEBUG "data": { 13:01:40 DEBUG "resultType": "vector", 13:01:40 DEBUG "result": [ 13:01:40 DEBUG { 13:01:40 DEBUG "metric": { 13:01:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:01:40 DEBUG }, 13:01:40 DEBUG "value": [ 13:01:40 DEBUG 13:01:40 DEBUG 13:01:40 DEBUG 13:01:40 DEBUG [...] 13:01:40 DEBUG 13:01:40 DEBUG 13:01:40 DEBUG 13:01:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:01:40 DEBUG }, 13:01:40 DEBUG "value": [ 13:01:40 DEBUG 1713528100, 13:01:40 DEBUG "5332410368" 13:01:40 DEBUG ] 13:01:40 DEBUG } 13:01:40 DEBUG ] 13:01:40 DEBUG } 13:01:40 DEBUG } 13:01:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528101" 13:01:41 INFO [http_cmd]: http status code OK 13:01:41 DEBUG --- status code --- 13:01:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:41 DEBUG --- http response --- 13:01:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:01:41 DEBUG ----- output ----- 13:01:41 DEBUG { 13:01:41 DEBUG "status": "success", 13:01:41 DEBUG "data": { 13:01:41 DEBUG "resultType": "vector", 13:01:41 DEBUG "result": [ 13:01:41 DEBUG { 13:01:41 DEBUG "metric": { 13:01:41 DEBUG "pod": "ds-cts-0" 13:01:41 DEBUG }, 13:01:41 DEBUG "value": [ 13:01:41 DEBUG 13:01:41 DEBUG 13:01:41 DEBUG 13:01:41 DEBUG [...] 13:01:41 DEBUG 13:01:41 DEBUG 13:01:41 DEBUG 13:01:41 DEBUG "pod": "ds-cts-2" 13:01:41 DEBUG }, 13:01:41 DEBUG "value": [ 13:01:41 DEBUG 1713528101, 13:01:41 DEBUG "30" 13:01:41 DEBUG ] 13:01:41 DEBUG } 13:01:41 DEBUG ] 13:01:41 DEBUG } 13:01:41 DEBUG } 13:01:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528101" 13:01:41 INFO [http_cmd]: http status code OK 13:01:41 DEBUG --- status code --- 13:01:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:41 DEBUG --- http response --- 13:01:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:41 DEBUG ----- output ----- 13:01:41 DEBUG { 13:01:41 DEBUG "status": "success", 13:01:41 DEBUG "data": { 13:01:41 DEBUG "resultType": "vector", 13:01:41 DEBUG "result": [ 13:01:41 DEBUG { 13:01:41 DEBUG "metric": { 13:01:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:01:41 DEBUG }, 13:01:41 DEBUG "value": [ 13:01:41 DEBUG 13:01:41 DEBUG 13:01:41 DEBUG 13:01:41 DEBUG [...] 13:01:41 DEBUG 13:01:41 DEBUG 13:01:41 DEBUG 13:01:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:01:41 DEBUG }, 13:01:41 DEBUG "value": [ 13:01:41 DEBUG 1713528101, 13:01:41 DEBUG "55.19" 13:01:41 DEBUG ] 13:01:41 DEBUG } 13:01:41 DEBUG ] 13:01:41 DEBUG } 13:01:41 DEBUG } 13:01:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528102" 13:01:42 INFO [http_cmd]: http status code OK 13:01:42 DEBUG --- status code --- 13:01:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:42 DEBUG --- http response --- 13:01:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:01:42 DEBUG ----- output ----- 13:01:42 DEBUG { 13:01:42 DEBUG "status": "success", 13:01:42 DEBUG "data": { 13:01:42 DEBUG "resultType": "vector", 13:01:42 DEBUG "result": [ 13:01:42 DEBUG { 13:01:42 DEBUG "metric": { 13:01:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG [...] 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG "pod": "ds-idrepo-0" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 1713528102, 13:01:42 DEBUG "0.6818691219996273" 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG } 13:01:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528102" 13:01:42 INFO [http_cmd]: http status code OK 13:01:42 DEBUG --- status code --- 13:01:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:42 DEBUG --- http response --- 13:01:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:42 DEBUG ----- output ----- 13:01:42 DEBUG { 13:01:42 DEBUG "status": "success", 13:01:42 DEBUG "data": { 13:01:42 DEBUG "resultType": "vector", 13:01:42 DEBUG "result": [ 13:01:42 DEBUG { 13:01:42 DEBUG "metric": { 13:01:42 DEBUG "pod": "ds-idrepo-0" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG [...] 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG "pod": "ds-idrepo-2" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 1713528102, 13:01:42 DEBUG "0" 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG } 13:01:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528102" 13:01:42 INFO [http_cmd]: http status code OK 13:01:42 DEBUG --- status code --- 13:01:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:42 DEBUG --- http response --- 13:01:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:01:42 DEBUG ----- output ----- 13:01:42 DEBUG { 13:01:42 DEBUG "status": "success", 13:01:42 DEBUG "data": { 13:01:42 DEBUG "resultType": "vector", 13:01:42 DEBUG "result": [ 13:01:42 DEBUG { 13:01:42 DEBUG "metric": { 13:01:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG [...] 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG "pod": "ds-idrepo-0" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 1713528102, 13:01:42 DEBUG "3442.1425700610175" 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG } 13:01:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528102" 13:01:42 INFO [http_cmd]: http status code OK 13:01:42 DEBUG --- status code --- 13:01:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:42 DEBUG --- http response --- 13:01:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:42 DEBUG ----- output ----- 13:01:42 DEBUG { 13:01:42 DEBUG "status": "success", 13:01:42 DEBUG "data": { 13:01:42 DEBUG "resultType": "vector", 13:01:42 DEBUG "result": [ 13:01:42 DEBUG { 13:01:42 DEBUG "metric": { 13:01:42 DEBUG "instance": "10.142.0.107:9100" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG [...] 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG "instance": "10.142.0.43:9100" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 1713528102, 13:01:42 DEBUG "1594884062208" 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG } 13:01:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528102" 13:01:42 INFO [http_cmd]: http status code OK 13:01:42 DEBUG --- status code --- 13:01:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:42 DEBUG --- http response --- 13:01:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:01:42 DEBUG ----- output ----- 13:01:42 DEBUG { 13:01:42 DEBUG "status": "success", 13:01:42 DEBUG "data": { 13:01:42 DEBUG "resultType": "vector", 13:01:42 DEBUG "result": [ 13:01:42 DEBUG { 13:01:42 DEBUG "metric": { 13:01:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG [...] 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 1713528102, 13:01:42 DEBUG "0" 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG } 13:01:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528102" 13:01:42 INFO [http_cmd]: http status code OK 13:01:42 DEBUG --- status code --- 13:01:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:42 DEBUG --- http response --- 13:01:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:42 DEBUG ----- output ----- 13:01:42 DEBUG { 13:01:42 DEBUG "status": "success", 13:01:42 DEBUG "data": { 13:01:42 DEBUG "resultType": "vector", 13:01:42 DEBUG "result": [ 13:01:42 DEBUG { 13:01:42 DEBUG "metric": { 13:01:42 DEBUG "instance": "10.142.0.107:9100" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG [...] 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG "instance": "10.142.0.43:9100" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 1713528102, 13:01:42 DEBUG "16346598.166666666" 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG } 13:01:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528102" 13:01:42 INFO [http_cmd]: http status code OK 13:01:42 DEBUG --- status code --- 13:01:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:01:42 DEBUG --- http response --- 13:01:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:01:42 DEBUG ----- output ----- 13:01:42 DEBUG { 13:01:42 DEBUG "status": "success", 13:01:42 DEBUG "data": { 13:01:42 DEBUG "resultType": "vector", 13:01:42 DEBUG "result": [ 13:01:42 DEBUG { 13:01:42 DEBUG "metric": { 13:01:42 DEBUG "instance": "10.142.0.107:9100" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG [...] 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG 13:01:42 DEBUG "instance": "10.142.0.43:9100" 13:01:42 DEBUG }, 13:01:42 DEBUG "value": [ 13:01:42 DEBUG 1713528102, 13:01:42 DEBUG "16353551.866666665" 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG ] 13:01:42 DEBUG } 13:01:42 DEBUG } 13:01:56 INFO 13:01:56 INFO [loop_until]: kubectl --namespace=xlou top pods 13:01:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:01:56 INFO [loop_until]: OK (rc = 0) 13:01:56 DEBUG --- stdout --- 13:01:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2486m 9437Mi am-6fd554f8f7-hvvkw 2482m 9006Mi am-6fd554f8f7-pmj4b 2501m 9131Mi ds-cts-0 10052m 12878Mi ds-cts-1 9489m 13221Mi ds-cts-2 10162m 13038Mi ds-idrepo-0 673m 3365Mi ds-idrepo-1 707m 3262Mi ds-idrepo-2 689m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-747d57dcb8-lclbf 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1015m 5417Mi 13:01:56 DEBUG --- stderr --- 13:01:56 DEBUG 13:02:03 INFO 13:02:03 INFO [loop_until]: kubectl --namespace=xlou top node 13:02:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:02:03 INFO [loop_until]: OK (rc = 0) 13:02:03 DEBUG --- stdout --- 13:02:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6130Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 763m 2% 6886Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2459m 8% 11766Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3225Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9719m 32% 16072Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5167Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2668m 8% 11577Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 75m 0% 3092Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2706m 9% 11368Mi 10% gke-xlou-cdm-ds-0f601751-j488 771m 2% 6364Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9151m 30% 18229Mi 16% gke-xlou-cdm-ds-0f601751-nj6q 781m 2% 7026Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9083m 30% 15606Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1040m 6% 9460Mi 16% 13:02:03 DEBUG --- stderr --- 13:02:03 DEBUG 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "ds-cts-0" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "ds-cts-2" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "0.000033333333333333335" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528158" 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "ds-cts-0" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "ds-cts-2" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "9.766666666666666" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "535.2984859559361" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 ERROR [http_cmd]: ERROR 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 ERROR --- status code --- 13:02:38 DEBUG --- status code --- 13:02:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 ERROR --- http response --- 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:02:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "ds-idrepo-0" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "ds-cts-2" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "0" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "1255676.5" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "ds-idrepo-0" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "ds-idrepo-2" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "0" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "2495.3333333333335" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528158" 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "1.0283553973112862" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "0" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "ds-idrepo-0" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "ds-idrepo-2" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "0" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528158" 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528158" 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "instance": "10.142.0.107:9100" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "instance": "10.142.0.43:9100" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "0.5856666666665888" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "instance": "10.142.0.107:9100" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "instance": "10.142.0.43:9100" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "0.6863333333333347" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528158" 13:02:38 INFO [http_cmd]: http status code OK 13:02:38 DEBUG --- status code --- 13:02:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:38 DEBUG --- http response --- 13:02:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:38 DEBUG ----- output ----- 13:02:38 DEBUG { 13:02:38 DEBUG "status": "success", 13:02:38 DEBUG "data": { 13:02:38 DEBUG "resultType": "vector", 13:02:38 DEBUG "result": [ 13:02:38 DEBUG { 13:02:38 DEBUG "metric": { 13:02:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG [...] 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG 13:02:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:02:38 DEBUG }, 13:02:38 DEBUG "value": [ 13:02:38 DEBUG 1713528158, 13:02:38 DEBUG "0" 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG ] 13:02:38 DEBUG } 13:02:38 DEBUG } 13:02:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528159" 13:02:39 INFO [http_cmd]: http status code OK 13:02:39 DEBUG --- status code --- 13:02:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:39 DEBUG --- http response --- 13:02:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:39 DEBUG ----- output ----- 13:02:39 DEBUG { 13:02:39 DEBUG "status": "success", 13:02:39 DEBUG "data": { 13:02:39 DEBUG "resultType": "vector", 13:02:39 DEBUG "result": [ 13:02:39 DEBUG { 13:02:39 DEBUG "metric": { 13:02:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:02:39 DEBUG }, 13:02:39 DEBUG "value": [ 13:02:39 DEBUG 13:02:39 DEBUG 13:02:39 DEBUG 13:02:39 DEBUG [...] 13:02:39 DEBUG 13:02:39 DEBUG 13:02:39 DEBUG 13:02:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:02:39 DEBUG }, 13:02:39 DEBUG "value": [ 13:02:39 DEBUG 1713528159, 13:02:39 DEBUG "0" 13:02:39 DEBUG ] 13:02:39 DEBUG } 13:02:39 DEBUG ] 13:02:39 DEBUG } 13:02:39 DEBUG } 13:02:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528159" 13:02:39 INFO [http_cmd]: http status code OK 13:02:39 DEBUG --- status code --- 13:02:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:39 DEBUG --- http response --- 13:02:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:39 DEBUG ----- output ----- 13:02:39 DEBUG { 13:02:39 DEBUG "status": "success", 13:02:39 DEBUG "data": { 13:02:39 DEBUG "resultType": "vector", 13:02:39 DEBUG "result": [ 13:02:39 DEBUG { 13:02:39 DEBUG "metric": { 13:02:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:02:39 DEBUG }, 13:02:39 DEBUG "value": [ 13:02:39 DEBUG 13:02:39 DEBUG 13:02:39 DEBUG 13:02:39 DEBUG [...] 13:02:39 DEBUG 13:02:39 DEBUG 13:02:39 DEBUG 13:02:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:02:39 DEBUG }, 13:02:39 DEBUG "value": [ 13:02:39 DEBUG 1713528159, 13:02:39 DEBUG "0" 13:02:39 DEBUG ] 13:02:39 DEBUG } 13:02:39 DEBUG ] 13:02:39 DEBUG } 13:02:39 DEBUG } 13:02:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528160" 13:02:40 INFO [http_cmd]: http status code OK 13:02:40 DEBUG --- status code --- 13:02:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:40 DEBUG --- http response --- 13:02:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:40 DEBUG ----- output ----- 13:02:40 DEBUG { 13:02:40 DEBUG "status": "success", 13:02:40 DEBUG "data": { 13:02:40 DEBUG "resultType": "vector", 13:02:40 DEBUG "result": [ 13:02:40 DEBUG { 13:02:40 DEBUG "metric": { 13:02:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:02:40 DEBUG }, 13:02:40 DEBUG "value": [ 13:02:40 DEBUG 13:02:40 DEBUG 13:02:40 DEBUG 13:02:40 DEBUG [...] 13:02:40 DEBUG 13:02:40 DEBUG 13:02:40 DEBUG 13:02:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:02:40 DEBUG }, 13:02:40 DEBUG "value": [ 13:02:40 DEBUG 1713528160, 13:02:40 DEBUG "6157856768" 13:02:40 DEBUG ] 13:02:40 DEBUG } 13:02:40 DEBUG ] 13:02:40 DEBUG } 13:02:40 DEBUG } 13:02:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528161" 13:02:41 INFO [http_cmd]: http status code OK 13:02:41 DEBUG --- status code --- 13:02:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:41 DEBUG --- http response --- 13:02:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:02:41 DEBUG ----- output ----- 13:02:41 DEBUG { 13:02:41 DEBUG "status": "success", 13:02:41 DEBUG "data": { 13:02:41 DEBUG "resultType": "vector", 13:02:41 DEBUG "result": [ 13:02:41 DEBUG { 13:02:41 DEBUG "metric": { 13:02:41 DEBUG "pod": "ds-cts-0" 13:02:41 DEBUG }, 13:02:41 DEBUG "value": [ 13:02:41 DEBUG 13:02:41 DEBUG 13:02:41 DEBUG 13:02:41 DEBUG [...] 13:02:41 DEBUG 13:02:41 DEBUG 13:02:41 DEBUG 13:02:41 DEBUG "pod": "ds-cts-2" 13:02:41 DEBUG }, 13:02:41 DEBUG "value": [ 13:02:41 DEBUG 1713528161, 13:02:41 DEBUG "30" 13:02:41 DEBUG ] 13:02:41 DEBUG } 13:02:41 DEBUG ] 13:02:41 DEBUG } 13:02:41 DEBUG } 13:02:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528161" 13:02:41 INFO [http_cmd]: http status code OK 13:02:41 DEBUG --- status code --- 13:02:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:41 DEBUG --- http response --- 13:02:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:41 DEBUG ----- output ----- 13:02:41 DEBUG { 13:02:41 DEBUG "status": "success", 13:02:41 DEBUG "data": { 13:02:41 DEBUG "resultType": "vector", 13:02:41 DEBUG "result": [ 13:02:41 DEBUG { 13:02:41 DEBUG "metric": { 13:02:41 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:02:41 DEBUG }, 13:02:41 DEBUG "value": [ 13:02:41 DEBUG 13:02:41 DEBUG 13:02:41 DEBUG 13:02:41 DEBUG [...] 13:02:41 DEBUG 13:02:41 DEBUG 13:02:41 DEBUG 13:02:41 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:02:41 DEBUG }, 13:02:41 DEBUG "value": [ 13:02:41 DEBUG 1713528161, 13:02:41 DEBUG "55.19" 13:02:41 DEBUG ] 13:02:41 DEBUG } 13:02:41 DEBUG ] 13:02:41 DEBUG } 13:02:41 DEBUG } 13:02:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528162" 13:02:42 INFO [http_cmd]: http status code OK 13:02:42 DEBUG --- status code --- 13:02:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:42 DEBUG --- http response --- 13:02:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:42 DEBUG ----- output ----- 13:02:42 DEBUG { 13:02:42 DEBUG "status": "success", 13:02:42 DEBUG "data": { 13:02:42 DEBUG "resultType": "vector", 13:02:42 DEBUG "result": [ 13:02:42 DEBUG { 13:02:42 DEBUG "metric": { 13:02:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG [...] 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 1713528162, 13:02:42 DEBUG "0.0016389548397172818" 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG } 13:02:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528162" 13:02:42 INFO [http_cmd]: http status code OK 13:02:42 DEBUG --- status code --- 13:02:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:42 DEBUG --- http response --- 13:02:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:42 DEBUG ----- output ----- 13:02:42 DEBUG { 13:02:42 DEBUG "status": "success", 13:02:42 DEBUG "data": { 13:02:42 DEBUG "resultType": "vector", 13:02:42 DEBUG "result": [ 13:02:42 DEBUG { 13:02:42 DEBUG "metric": { 13:02:42 DEBUG "pod": "ds-idrepo-0" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG [...] 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG "pod": "ds-idrepo-2" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 1713528162, 13:02:42 DEBUG "0" 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG } 13:02:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528162" 13:02:42 INFO [http_cmd]: http status code OK 13:02:42 DEBUG --- status code --- 13:02:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:42 DEBUG --- http response --- 13:02:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:42 DEBUG ----- output ----- 13:02:42 DEBUG { 13:02:42 DEBUG "status": "success", 13:02:42 DEBUG "data": { 13:02:42 DEBUG "resultType": "vector", 13:02:42 DEBUG "result": [ 13:02:42 DEBUG { 13:02:42 DEBUG "metric": { 13:02:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG [...] 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 1713528162, 13:02:42 DEBUG "37376913.30635203" 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG } 13:02:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528162" 13:02:42 INFO [http_cmd]: http status code OK 13:02:42 DEBUG --- status code --- 13:02:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:42 DEBUG --- http response --- 13:02:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:42 DEBUG ----- output ----- 13:02:42 DEBUG { 13:02:42 DEBUG "status": "success", 13:02:42 DEBUG "data": { 13:02:42 DEBUG "resultType": "vector", 13:02:42 DEBUG "result": [ 13:02:42 DEBUG { 13:02:42 DEBUG "metric": { 13:02:42 DEBUG "instance": "10.142.0.107:9100" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG [...] 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG "instance": "10.142.0.43:9100" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 1713528162, 13:02:42 DEBUG "1596211590144" 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG } 13:02:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528162" 13:02:42 INFO [http_cmd]: http status code OK 13:02:42 DEBUG --- status code --- 13:02:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:42 DEBUG --- http response --- 13:02:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:02:42 DEBUG ----- output ----- 13:02:42 DEBUG { 13:02:42 DEBUG "status": "success", 13:02:42 DEBUG "data": { 13:02:42 DEBUG "resultType": "vector", 13:02:42 DEBUG "result": [ 13:02:42 DEBUG { 13:02:42 DEBUG "metric": { 13:02:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG [...] 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 1713528162, 13:02:42 DEBUG "0" 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG } 13:02:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528162" 13:02:42 INFO [http_cmd]: http status code OK 13:02:42 DEBUG --- status code --- 13:02:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:42 DEBUG --- http response --- 13:02:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:42 DEBUG ----- output ----- 13:02:42 DEBUG { 13:02:42 DEBUG "status": "success", 13:02:42 DEBUG "data": { 13:02:42 DEBUG "resultType": "vector", 13:02:42 DEBUG "result": [ 13:02:42 DEBUG { 13:02:42 DEBUG "metric": { 13:02:42 DEBUG "instance": "10.142.0.107:9100" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG [...] 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG "instance": "10.142.0.43:9100" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 1713528162, 13:02:42 DEBUG "16281885.866666667" 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG } 13:02:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528162" 13:02:42 INFO [http_cmd]: http status code OK 13:02:42 DEBUG --- status code --- 13:02:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:02:42 DEBUG --- http response --- 13:02:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:02:42 DEBUG ----- output ----- 13:02:42 DEBUG { 13:02:42 DEBUG "status": "success", 13:02:42 DEBUG "data": { 13:02:42 DEBUG "resultType": "vector", 13:02:42 DEBUG "result": [ 13:02:42 DEBUG { 13:02:42 DEBUG "metric": { 13:02:42 DEBUG "instance": "10.142.0.107:9100" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG [...] 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG 13:02:42 DEBUG "instance": "10.142.0.43:9100" 13:02:42 DEBUG }, 13:02:42 DEBUG "value": [ 13:02:42 DEBUG 1713528162, 13:02:42 DEBUG "16287024.399999999" 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG ] 13:02:42 DEBUG } 13:02:42 DEBUG } 13:02:56 INFO 13:02:56 INFO [loop_until]: kubectl --namespace=xlou top pods 13:02:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:02:56 INFO [loop_until]: OK (rc = 0) 13:02:56 DEBUG --- stdout --- 13:02:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2463m 8763Mi am-6fd554f8f7-hvvkw 2533m 9006Mi am-6fd554f8f7-pmj4b 2700m 9133Mi ds-cts-0 9544m 13069Mi ds-cts-1 8706m 12842Mi ds-cts-2 9990m 12984Mi ds-idrepo-0 678m 3365Mi ds-idrepo-1 708m 3262Mi ds-idrepo-2 709m 3191Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-747d57dcb8-lclbf 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 995m 6068Mi 13:02:56 DEBUG --- stderr --- 13:02:56 DEBUG 13:03:03 INFO 13:03:03 INFO [loop_until]: kubectl --namespace=xlou top node 13:03:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:03:03 INFO [loop_until]: OK (rc = 0) 13:03:03 DEBUG --- stdout --- 13:03:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6135Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 767m 2% 6878Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2481m 8% 11772Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3232Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9929m 33% 16168Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5149Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2557m 8% 11587Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3093Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2742m 9% 11368Mi 10% gke-xlou-cdm-ds-0f601751-j488 773m 2% 6358Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8708m 29% 18137Mi 16% gke-xlou-cdm-ds-0f601751-nj6q 793m 2% 7020Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9453m 31% 15396Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1075m 6% 10338Mi 17% 13:03:03 DEBUG --- stderr --- 13:03:03 DEBUG 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528218" 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 ERROR [http_cmd]: ERROR 13:03:38 ERROR --- status code --- 13:03:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:03:38 ERROR --- http response --- 13:03:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:03:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "pod": "ds-cts-0" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG [...] 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "pod": "ds-cts-2" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "0.0002666666666666667" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG "result": [ 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG { 13:03:38 DEBUG { 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG "pod": "ds-cts-0" 13:03:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG }, 13:03:38 DEBUG }, 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:03:38 DEBUG "value": [ 13:03:38 DEBUG "value": [ 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG { 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "data": { 13:03:38 DEBUG [...] 13:03:38 DEBUG [...] 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "result": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG { 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "pod": "ds-cts-2" 13:03:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:03:38 DEBUG "pod": "ds-idrepo-0" 13:03:38 DEBUG }, 13:03:38 DEBUG }, 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG "value": [ 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG 13:03:38 DEBUG "12" 13:03:38 DEBUG "0" 13:03:38 DEBUG 13:03:38 DEBUG ] 13:03:38 DEBUG ] 13:03:38 DEBUG 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 DEBUG [...] 13:03:38 DEBUG ] 13:03:38 DEBUG ] 13:03:38 DEBUG 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 DEBUG 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 DEBUG 13:03:38 DEBUG "pod": "ds-cts-2" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "0" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "pod": "ds-idrepo-0" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG [...] 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "pod": "ds-idrepo-2" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "0" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG [...] 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "1105592" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG [...] 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "2483.4666666666667" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528218" 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG [...] 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "1.0354855171819959" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528218" 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG [...] 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "0" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "pod": "ds-idrepo-0" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG [...] 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "pod": "ds-idrepo-2" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "0" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528218" 13:03:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528218" 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "instance": "10.142.0.107:9100" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 INFO [http_cmd]: http status code OK 13:03:38 DEBUG [...] 13:03:38 DEBUG --- status code --- 13:03:38 DEBUG 13:03:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:38 DEBUG 13:03:38 DEBUG --- http response --- 13:03:38 DEBUG 13:03:38 DEBUG "instance": "10.142.0.43:9100" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "0.6217333333332941" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:38 DEBUG ----- output ----- 13:03:38 DEBUG { 13:03:38 DEBUG "status": "success", 13:03:38 DEBUG "data": { 13:03:38 DEBUG "resultType": "vector", 13:03:38 DEBUG "result": [ 13:03:38 DEBUG { 13:03:38 DEBUG "metric": { 13:03:38 DEBUG "instance": "10.142.0.107:9100" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG [...] 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG 13:03:38 DEBUG "instance": "10.142.0.43:9100" 13:03:38 DEBUG }, 13:03:38 DEBUG "value": [ 13:03:38 DEBUG 1713528218, 13:03:38 DEBUG "0.6529999999999991" 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG ] 13:03:38 DEBUG } 13:03:38 DEBUG } 13:03:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528219" 13:03:39 INFO [http_cmd]: http status code OK 13:03:39 DEBUG --- status code --- 13:03:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:39 DEBUG --- http response --- 13:03:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:39 DEBUG ----- output ----- 13:03:39 DEBUG { 13:03:39 DEBUG "status": "success", 13:03:39 DEBUG "data": { 13:03:39 DEBUG "resultType": "vector", 13:03:39 DEBUG "result": [ 13:03:39 DEBUG { 13:03:39 DEBUG "metric": { 13:03:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:03:39 DEBUG }, 13:03:39 DEBUG "value": [ 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG [...] 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:03:39 DEBUG }, 13:03:39 DEBUG "value": [ 13:03:39 DEBUG 1713528219, 13:03:39 DEBUG "0" 13:03:39 DEBUG ] 13:03:39 DEBUG } 13:03:39 DEBUG ] 13:03:39 DEBUG } 13:03:39 DEBUG } 13:03:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528219" 13:03:39 INFO [http_cmd]: http status code OK 13:03:39 DEBUG --- status code --- 13:03:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:39 DEBUG --- http response --- 13:03:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:39 DEBUG ----- output ----- 13:03:39 DEBUG { 13:03:39 DEBUG "status": "success", 13:03:39 DEBUG "data": { 13:03:39 DEBUG "resultType": "vector", 13:03:39 DEBUG "result": [ 13:03:39 DEBUG { 13:03:39 DEBUG "metric": { 13:03:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:03:39 DEBUG }, 13:03:39 DEBUG "value": [ 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG [...] 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:03:39 DEBUG }, 13:03:39 DEBUG "value": [ 13:03:39 DEBUG 1713528219, 13:03:39 DEBUG "0" 13:03:39 DEBUG ] 13:03:39 DEBUG } 13:03:39 DEBUG ] 13:03:39 DEBUG } 13:03:39 DEBUG } 13:03:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528219" 13:03:39 INFO [http_cmd]: http status code OK 13:03:39 DEBUG --- status code --- 13:03:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:39 DEBUG --- http response --- 13:03:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:39 DEBUG ----- output ----- 13:03:39 DEBUG { 13:03:39 DEBUG "status": "success", 13:03:39 DEBUG "data": { 13:03:39 DEBUG "resultType": "vector", 13:03:39 DEBUG "result": [ 13:03:39 DEBUG { 13:03:39 DEBUG "metric": { 13:03:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:03:39 DEBUG }, 13:03:39 DEBUG "value": [ 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG [...] 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG 13:03:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:03:39 DEBUG }, 13:03:39 DEBUG "value": [ 13:03:39 DEBUG 1713528219, 13:03:39 DEBUG "0" 13:03:39 DEBUG ] 13:03:39 DEBUG } 13:03:39 DEBUG ] 13:03:39 DEBUG } 13:03:39 DEBUG } 13:03:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528220" 13:03:40 INFO [http_cmd]: http status code OK 13:03:40 DEBUG --- status code --- 13:03:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:40 DEBUG --- http response --- 13:03:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:40 DEBUG ----- output ----- 13:03:40 DEBUG { 13:03:40 DEBUG "status": "success", 13:03:40 DEBUG "data": { 13:03:40 DEBUG "resultType": "vector", 13:03:40 DEBUG "result": [ 13:03:40 DEBUG { 13:03:40 DEBUG "metric": { 13:03:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:03:40 DEBUG }, 13:03:40 DEBUG "value": [ 13:03:40 DEBUG 13:03:40 DEBUG 13:03:40 DEBUG 13:03:40 DEBUG [...] 13:03:40 DEBUG 13:03:40 DEBUG 13:03:40 DEBUG 13:03:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:03:40 DEBUG }, 13:03:40 DEBUG "value": [ 13:03:40 DEBUG 1713528220, 13:03:40 DEBUG "7087050752" 13:03:40 DEBUG ] 13:03:40 DEBUG } 13:03:40 DEBUG ] 13:03:40 DEBUG } 13:03:40 DEBUG } 13:03:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528221" 13:03:41 INFO [http_cmd]: http status code OK 13:03:41 DEBUG --- status code --- 13:03:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:41 DEBUG --- http response --- 13:03:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:03:41 DEBUG ----- output ----- 13:03:41 DEBUG { 13:03:41 DEBUG "status": "success", 13:03:41 DEBUG "data": { 13:03:41 DEBUG "resultType": "vector", 13:03:41 DEBUG "result": [ 13:03:41 DEBUG { 13:03:41 DEBUG "metric": { 13:03:41 DEBUG "pod": "ds-cts-0" 13:03:41 DEBUG }, 13:03:41 DEBUG "value": [ 13:03:41 DEBUG 13:03:41 DEBUG 13:03:41 DEBUG 13:03:41 DEBUG [...] 13:03:41 DEBUG 13:03:41 DEBUG 13:03:41 DEBUG 13:03:41 DEBUG "pod": "ds-cts-2" 13:03:41 DEBUG }, 13:03:41 DEBUG "value": [ 13:03:41 DEBUG 1713528221, 13:03:41 DEBUG "30" 13:03:41 DEBUG ] 13:03:41 DEBUG } 13:03:41 DEBUG ] 13:03:41 DEBUG } 13:03:41 DEBUG } 13:03:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528222" 13:03:42 INFO [http_cmd]: http status code OK 13:03:42 DEBUG --- status code --- 13:03:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:42 DEBUG --- http response --- 13:03:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:42 DEBUG ----- output ----- 13:03:42 DEBUG { 13:03:42 DEBUG "status": "success", 13:03:42 DEBUG "data": { 13:03:42 DEBUG "resultType": "vector", 13:03:42 DEBUG "result": [ 13:03:42 DEBUG { 13:03:42 DEBUG "metric": { 13:03:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG [...] 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 1713528222, 13:03:42 DEBUG "55.19" 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG } 13:03:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528222" 13:03:42 INFO [http_cmd]: http status code OK 13:03:42 DEBUG --- status code --- 13:03:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:42 DEBUG --- http response --- 13:03:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:03:42 DEBUG ----- output ----- 13:03:42 DEBUG { 13:03:42 DEBUG "status": "success", 13:03:42 DEBUG "data": { 13:03:42 DEBUG "resultType": "vector", 13:03:42 DEBUG "result": [ 13:03:42 DEBUG { 13:03:42 DEBUG "metric": { 13:03:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG [...] 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 1713528222, 13:03:42 DEBUG "0" 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG } 13:03:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528222" 13:03:42 INFO [http_cmd]: http status code OK 13:03:42 DEBUG --- status code --- 13:03:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:42 DEBUG --- http response --- 13:03:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:42 DEBUG ----- output ----- 13:03:42 DEBUG { 13:03:42 DEBUG "status": "success", 13:03:42 DEBUG "data": { 13:03:42 DEBUG "resultType": "vector", 13:03:42 DEBUG "result": [ 13:03:42 DEBUG { 13:03:42 DEBUG "metric": { 13:03:42 DEBUG "pod": "ds-idrepo-0" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG [...] 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG "pod": "ds-idrepo-2" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 1713528222, 13:03:42 DEBUG "0" 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG } 13:03:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528222" 13:03:42 INFO [http_cmd]: http status code OK 13:03:42 DEBUG --- status code --- 13:03:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:42 DEBUG --- http response --- 13:03:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:03:42 DEBUG ----- output ----- 13:03:42 DEBUG { 13:03:42 DEBUG "status": "success", 13:03:42 DEBUG "data": { 13:03:42 DEBUG "resultType": "vector", 13:03:42 DEBUG "result": [ 13:03:42 DEBUG { 13:03:42 DEBUG "metric": { 13:03:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG [...] 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 1713528222, 13:03:42 DEBUG "0" 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG } 13:03:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528222" 13:03:42 INFO [http_cmd]: http status code OK 13:03:42 DEBUG --- status code --- 13:03:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:42 DEBUG --- http response --- 13:03:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:42 DEBUG ----- output ----- 13:03:42 DEBUG { 13:03:42 DEBUG "status": "success", 13:03:42 DEBUG "data": { 13:03:42 DEBUG "resultType": "vector", 13:03:42 DEBUG "result": [ 13:03:42 DEBUG { 13:03:42 DEBUG "metric": { 13:03:42 DEBUG "instance": "10.142.0.107:9100" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG [...] 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG "instance": "10.142.0.43:9100" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 1713528222, 13:03:42 DEBUG "1597912738816" 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG } 13:03:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528222" 13:03:42 INFO [http_cmd]: http status code OK 13:03:42 DEBUG --- status code --- 13:03:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:42 DEBUG --- http response --- 13:03:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:03:42 DEBUG ----- output ----- 13:03:42 DEBUG { 13:03:42 DEBUG "status": "success", 13:03:42 DEBUG "data": { 13:03:42 DEBUG "resultType": "vector", 13:03:42 DEBUG "result": [ 13:03:42 DEBUG { 13:03:42 DEBUG "metric": { 13:03:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG [...] 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 1713528222, 13:03:42 DEBUG "0" 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG } 13:03:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528222" 13:03:42 INFO [http_cmd]: http status code OK 13:03:42 DEBUG --- status code --- 13:03:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:42 DEBUG --- http response --- 13:03:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:42 DEBUG ----- output ----- 13:03:42 DEBUG { 13:03:42 DEBUG "status": "success", 13:03:42 DEBUG "data": { 13:03:42 DEBUG "resultType": "vector", 13:03:42 DEBUG "result": [ 13:03:42 DEBUG { 13:03:42 DEBUG "metric": { 13:03:42 DEBUG "instance": "10.142.0.107:9100" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG [...] 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG "instance": "10.142.0.43:9100" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 1713528222, 13:03:42 DEBUG "16231581.533333333" 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG } 13:03:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528222" 13:03:42 INFO [http_cmd]: http status code OK 13:03:42 DEBUG --- status code --- 13:03:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:03:42 DEBUG --- http response --- 13:03:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:03:42 DEBUG ----- output ----- 13:03:42 DEBUG { 13:03:42 DEBUG "status": "success", 13:03:42 DEBUG "data": { 13:03:42 DEBUG "resultType": "vector", 13:03:42 DEBUG "result": [ 13:03:42 DEBUG { 13:03:42 DEBUG "metric": { 13:03:42 DEBUG "instance": "10.142.0.107:9100" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG [...] 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG 13:03:42 DEBUG "instance": "10.142.0.43:9100" 13:03:42 DEBUG }, 13:03:42 DEBUG "value": [ 13:03:42 DEBUG 1713528222, 13:03:42 DEBUG "16238626.666666664" 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG ] 13:03:42 DEBUG } 13:03:42 DEBUG } 13:03:56 INFO 13:03:56 INFO [loop_until]: kubectl --namespace=xlou top pods 13:03:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:03:57 INFO [loop_until]: OK (rc = 0) 13:03:57 DEBUG --- stdout --- 13:03:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2435m 8756Mi am-6fd554f8f7-hvvkw 2475m 9008Mi am-6fd554f8f7-pmj4b 2684m 9133Mi ds-cts-0 9034m 13237Mi ds-cts-1 8409m 13238Mi ds-cts-2 9715m 13137Mi ds-idrepo-0 708m 3365Mi ds-idrepo-1 694m 3262Mi ds-idrepo-2 709m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 4m 1216Mi lodemon-747d57dcb8-lclbf 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1007m 7072Mi 13:03:57 DEBUG --- stderr --- 13:03:57 DEBUG 13:04:03 INFO 13:04:03 INFO [loop_until]: kubectl --namespace=xlou top node 13:04:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:04:03 INFO [loop_until]: OK (rc = 0) 13:04:03 DEBUG --- stdout --- 13:04:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6131Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 770m 2% 6891Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2472m 8% 11772Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3231Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9662m 32% 16362Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5164Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2585m 8% 11595Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 72m 0% 3090Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2745m 9% 11370Mi 10% gke-xlou-cdm-ds-0f601751-j488 763m 2% 6359Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8370m 28% 18112Mi 16% gke-xlou-cdm-ds-0f601751-nj6q 793m 2% 7032Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9848m 32% 15754Mi 14% gke-xlou-cdm-frontend-a8771548-i82n 535m 3% 10823Mi 18% 13:04:03 DEBUG --- stderr --- 13:04:03 DEBUG 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG { 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "data": { 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG "result": [ 13:04:38 DEBUG { 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "pod": "ds-cts-0" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG "pod": "ds-cts-2" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG "0.00013333333333333334" 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528278" 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 ERROR [http_cmd]: ERROR 13:04:38 ERROR --- status code --- 13:04:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:04:38 ERROR --- http response --- 13:04:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:04:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:04:38 DEBUG { 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG { 13:04:38 DEBUG "data": { 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG "data": { 13:04:38 DEBUG "result": [ 13:04:38 DEBUG { 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG { 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "result": [ 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "data": { 13:04:38 DEBUG { 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 DEBUG "pod": "ds-cts-0" 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG "metric": { 13:04:38 DEBUG }, 13:04:38 DEBUG "result": [ 13:04:38 DEBUG "pod": "ds-idrepo-0" 13:04:38 DEBUG "value": [ 13:04:38 DEBUG { 13:04:38 DEBUG }, 13:04:38 DEBUG 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG }, 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG "pod": "ds-cts-2" 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG }, 13:04:38 DEBUG 13:04:38 DEBUG "pod": "ds-cts-2" 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG }, 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG 13:04:38 DEBUG "value": [ 13:04:38 DEBUG "3.6999999999999997" 13:04:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG ] 13:04:38 DEBUG }, 13:04:38 DEBUG "0" 13:04:38 DEBUG } 13:04:38 DEBUG "value": [ 13:04:38 DEBUG ] 13:04:38 DEBUG ] 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 DEBUG "0" 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG { 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "data": { 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG "result": [ 13:04:38 DEBUG { 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "pod": "ds-idrepo-0" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG "pod": "ds-idrepo-2" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG "0" 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG { 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "data": { 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG "result": [ 13:04:38 DEBUG { 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG "955490" 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG { 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "data": { 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG "result": [ 13:04:38 DEBUG { 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG "2499.8" 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528278" 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG { 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "data": { 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG "result": [ 13:04:38 DEBUG { 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG "0.5587438633770314" 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528278" 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG { 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "data": { 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG "result": [ 13:04:38 DEBUG { 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG "0" 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG { 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "data": { 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG "result": [ 13:04:38 DEBUG { 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "pod": "ds-idrepo-0" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG "pod": "ds-idrepo-2" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG "0" 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528278" 13:04:38 INFO [http_cmd]: http status code OK 13:04:38 DEBUG --- status code --- 13:04:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:38 DEBUG --- http response --- 13:04:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:38 DEBUG ----- output ----- 13:04:38 DEBUG { 13:04:38 DEBUG "status": "success", 13:04:38 DEBUG "data": { 13:04:38 DEBUG "resultType": "vector", 13:04:38 DEBUG "result": [ 13:04:38 DEBUG { 13:04:38 DEBUG "metric": { 13:04:38 DEBUG "instance": "10.142.0.107:9100" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG [...] 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG 13:04:38 DEBUG "instance": "10.142.0.43:9100" 13:04:38 DEBUG }, 13:04:38 DEBUG "value": [ 13:04:38 DEBUG 1713528278, 13:04:38 DEBUG "0.6356666666666646" 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG ] 13:04:38 DEBUG } 13:04:38 DEBUG } 13:04:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528279" 13:04:39 INFO [http_cmd]: http status code OK 13:04:39 DEBUG --- status code --- 13:04:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:39 DEBUG --- http response --- 13:04:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:39 DEBUG ----- output ----- 13:04:39 DEBUG { 13:04:39 DEBUG "status": "success", 13:04:39 DEBUG "data": { 13:04:39 DEBUG "resultType": "vector", 13:04:39 DEBUG "result": [ 13:04:39 DEBUG { 13:04:39 DEBUG "metric": { 13:04:39 DEBUG "instance": "10.142.0.107:9100" 13:04:39 DEBUG }, 13:04:39 DEBUG "value": [ 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG [...] 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG "instance": "10.142.0.43:9100" 13:04:39 DEBUG }, 13:04:39 DEBUG "value": [ 13:04:39 DEBUG 1713528279, 13:04:39 DEBUG "0.6355333333333096" 13:04:39 DEBUG ] 13:04:39 DEBUG } 13:04:39 DEBUG ] 13:04:39 DEBUG } 13:04:39 DEBUG } 13:04:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528279" 13:04:39 INFO [http_cmd]: http status code OK 13:04:39 DEBUG --- status code --- 13:04:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:39 DEBUG --- http response --- 13:04:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:39 DEBUG ----- output ----- 13:04:39 DEBUG { 13:04:39 DEBUG "status": "success", 13:04:39 DEBUG "data": { 13:04:39 DEBUG "resultType": "vector", 13:04:39 DEBUG "result": [ 13:04:39 DEBUG { 13:04:39 DEBUG "metric": { 13:04:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:04:39 DEBUG }, 13:04:39 DEBUG "value": [ 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG [...] 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:04:39 DEBUG }, 13:04:39 DEBUG "value": [ 13:04:39 DEBUG 1713528279, 13:04:39 DEBUG "0" 13:04:39 DEBUG ] 13:04:39 DEBUG } 13:04:39 DEBUG ] 13:04:39 DEBUG } 13:04:39 DEBUG } 13:04:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528279" 13:04:39 INFO [http_cmd]: http status code OK 13:04:39 DEBUG --- status code --- 13:04:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:39 DEBUG --- http response --- 13:04:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:39 DEBUG ----- output ----- 13:04:39 DEBUG { 13:04:39 DEBUG "status": "success", 13:04:39 DEBUG "data": { 13:04:39 DEBUG "resultType": "vector", 13:04:39 DEBUG "result": [ 13:04:39 DEBUG { 13:04:39 DEBUG "metric": { 13:04:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:04:39 DEBUG }, 13:04:39 DEBUG "value": [ 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG [...] 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:04:39 DEBUG }, 13:04:39 DEBUG "value": [ 13:04:39 DEBUG 1713528279, 13:04:39 DEBUG "0" 13:04:39 DEBUG ] 13:04:39 DEBUG } 13:04:39 DEBUG ] 13:04:39 DEBUG } 13:04:39 DEBUG } 13:04:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528279" 13:04:39 INFO [http_cmd]: http status code OK 13:04:39 DEBUG --- status code --- 13:04:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:39 DEBUG --- http response --- 13:04:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:39 DEBUG ----- output ----- 13:04:39 DEBUG { 13:04:39 DEBUG "status": "success", 13:04:39 DEBUG "data": { 13:04:39 DEBUG "resultType": "vector", 13:04:39 DEBUG "result": [ 13:04:39 DEBUG { 13:04:39 DEBUG "metric": { 13:04:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:04:39 DEBUG }, 13:04:39 DEBUG "value": [ 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG [...] 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG 13:04:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:04:39 DEBUG }, 13:04:39 DEBUG "value": [ 13:04:39 DEBUG 1713528279, 13:04:39 DEBUG "0" 13:04:39 DEBUG ] 13:04:39 DEBUG } 13:04:39 DEBUG ] 13:04:39 DEBUG } 13:04:39 DEBUG } 13:04:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528280" 13:04:40 INFO [http_cmd]: http status code OK 13:04:40 DEBUG --- status code --- 13:04:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:40 DEBUG --- http response --- 13:04:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:40 DEBUG ----- output ----- 13:04:40 DEBUG { 13:04:40 DEBUG "status": "success", 13:04:40 DEBUG "data": { 13:04:40 DEBUG "resultType": "vector", 13:04:40 DEBUG "result": [ 13:04:40 DEBUG { 13:04:40 DEBUG "metric": { 13:04:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:04:40 DEBUG }, 13:04:40 DEBUG "value": [ 13:04:40 DEBUG 13:04:40 DEBUG 13:04:40 DEBUG 13:04:40 DEBUG [...] 13:04:40 DEBUG 13:04:40 DEBUG 13:04:40 DEBUG 13:04:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:04:40 DEBUG }, 13:04:40 DEBUG "value": [ 13:04:40 DEBUG 1713528280, 13:04:40 DEBUG "7502110720" 13:04:40 DEBUG ] 13:04:40 DEBUG } 13:04:40 DEBUG ] 13:04:40 DEBUG } 13:04:40 DEBUG } 13:04:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528281" 13:04:41 INFO [http_cmd]: http status code OK 13:04:41 DEBUG --- status code --- 13:04:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:41 DEBUG --- http response --- 13:04:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:04:41 DEBUG ----- output ----- 13:04:41 DEBUG { 13:04:41 DEBUG "status": "success", 13:04:41 DEBUG "data": { 13:04:41 DEBUG "resultType": "vector", 13:04:41 DEBUG "result": [ 13:04:41 DEBUG { 13:04:41 DEBUG "metric": { 13:04:41 DEBUG "pod": "ds-cts-0" 13:04:41 DEBUG }, 13:04:41 DEBUG "value": [ 13:04:41 DEBUG 13:04:41 DEBUG 13:04:41 DEBUG 13:04:41 DEBUG [...] 13:04:41 DEBUG 13:04:41 DEBUG 13:04:41 DEBUG 13:04:41 DEBUG "pod": "ds-cts-2" 13:04:41 DEBUG }, 13:04:41 DEBUG "value": [ 13:04:41 DEBUG 1713528281, 13:04:41 DEBUG "30" 13:04:41 DEBUG ] 13:04:41 DEBUG } 13:04:41 DEBUG ] 13:04:41 DEBUG } 13:04:41 DEBUG } 13:04:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528282" 13:04:42 INFO [http_cmd]: http status code OK 13:04:42 DEBUG --- status code --- 13:04:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:42 DEBUG --- http response --- 13:04:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:42 DEBUG ----- output ----- 13:04:42 DEBUG { 13:04:42 DEBUG "status": "success", 13:04:42 DEBUG "data": { 13:04:42 DEBUG "resultType": "vector", 13:04:42 DEBUG "result": [ 13:04:42 DEBUG { 13:04:42 DEBUG "metric": { 13:04:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG [...] 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 1713528282, 13:04:42 DEBUG "55.19" 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG } 13:04:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528282" 13:04:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528282" 13:04:42 INFO [http_cmd]: http status code OK 13:04:42 DEBUG --- status code --- 13:04:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:42 DEBUG --- http response --- 13:04:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:04:42 DEBUG ----- output ----- 13:04:42 DEBUG { 13:04:42 DEBUG "status": "success", 13:04:42 DEBUG "data": { 13:04:42 DEBUG "resultType": "vector", 13:04:42 DEBUG "result": [ 13:04:42 DEBUG { 13:04:42 DEBUG "metric": { 13:04:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG [...] 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 1713528282, 13:04:42 DEBUG "0.0017578248532963113" 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG } 13:04:42 INFO [http_cmd]: http status code OK 13:04:42 DEBUG --- status code --- 13:04:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:42 DEBUG --- http response --- 13:04:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:42 DEBUG ----- output ----- 13:04:42 DEBUG { 13:04:42 DEBUG "status": "success", 13:04:42 DEBUG "data": { 13:04:42 DEBUG "resultType": "vector", 13:04:42 DEBUG "result": [ 13:04:42 DEBUG { 13:04:42 DEBUG "metric": { 13:04:42 DEBUG "pod": "ds-idrepo-0" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG [...] 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG "pod": "ds-idrepo-2" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 1713528282, 13:04:42 DEBUG "0" 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG } 13:04:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528282" 13:04:42 INFO [http_cmd]: http status code OK 13:04:42 DEBUG --- status code --- 13:04:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:42 DEBUG --- http response --- 13:04:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:04:42 DEBUG ----- output ----- 13:04:42 DEBUG { 13:04:42 DEBUG "status": "success", 13:04:42 DEBUG "data": { 13:04:42 DEBUG "resultType": "vector", 13:04:42 DEBUG "result": [ 13:04:42 DEBUG { 13:04:42 DEBUG "metric": { 13:04:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG [...] 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 1713528282, 13:04:42 DEBUG "37583848.18263232" 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG } 13:04:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528282" 13:04:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528282" 13:04:42 INFO [http_cmd]: http status code OK 13:04:42 DEBUG --- status code --- 13:04:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:42 DEBUG --- http response --- 13:04:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:42 DEBUG ----- output ----- 13:04:42 DEBUG { 13:04:42 DEBUG "status": "success", 13:04:42 DEBUG "data": { 13:04:42 DEBUG "resultType": "vector", 13:04:42 DEBUG "result": [ 13:04:42 DEBUG { 13:04:42 DEBUG "metric": { 13:04:42 DEBUG "instance": "10.142.0.107:9100" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG [...] 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG "instance": "10.142.0.43:9100" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 1713528282, 13:04:42 DEBUG "1599614848000" 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG } 13:04:42 INFO [http_cmd]: http status code OK 13:04:42 DEBUG --- status code --- 13:04:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:42 DEBUG --- http response --- 13:04:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:04:42 DEBUG ----- output ----- 13:04:42 DEBUG { 13:04:42 DEBUG "status": "success", 13:04:42 DEBUG "data": { 13:04:42 DEBUG "resultType": "vector", 13:04:42 DEBUG "result": [ 13:04:42 DEBUG { 13:04:42 DEBUG "metric": { 13:04:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG [...] 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 1713528282, 13:04:42 DEBUG "0" 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG } 13:04:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528282" 13:04:42 INFO [http_cmd]: http status code OK 13:04:42 DEBUG --- status code --- 13:04:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:42 DEBUG --- http response --- 13:04:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:42 DEBUG ----- output ----- 13:04:42 DEBUG { 13:04:42 DEBUG "status": "success", 13:04:42 DEBUG "data": { 13:04:42 DEBUG "resultType": "vector", 13:04:42 DEBUG "result": [ 13:04:42 DEBUG { 13:04:42 DEBUG "metric": { 13:04:42 DEBUG "instance": "10.142.0.107:9100" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG [...] 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG "instance": "10.142.0.43:9100" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 1713528282, 13:04:42 DEBUG "16350759.700000001" 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG } 13:04:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528282" 13:04:42 INFO [http_cmd]: http status code OK 13:04:42 DEBUG --- status code --- 13:04:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:04:42 DEBUG --- http response --- 13:04:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:04:42 DEBUG ----- output ----- 13:04:42 DEBUG { 13:04:42 DEBUG "status": "success", 13:04:42 DEBUG "data": { 13:04:42 DEBUG "resultType": "vector", 13:04:42 DEBUG "result": [ 13:04:42 DEBUG { 13:04:42 DEBUG "metric": { 13:04:42 DEBUG "instance": "10.142.0.107:9100" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG [...] 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG 13:04:42 DEBUG "instance": "10.142.0.43:9100" 13:04:42 DEBUG }, 13:04:42 DEBUG "value": [ 13:04:42 DEBUG 1713528282, 13:04:42 DEBUG "16358808.799999999" 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG ] 13:04:42 DEBUG } 13:04:42 DEBUG } 13:04:57 INFO 13:04:57 INFO [loop_until]: kubectl --namespace=xlou top pods 13:04:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:04:57 INFO [loop_until]: OK (rc = 0) 13:04:57 DEBUG --- stdout --- 13:04:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2487m 8761Mi am-6fd554f8f7-hvvkw 2547m 9009Mi am-6fd554f8f7-pmj4b 2637m 8656Mi ds-cts-0 9108m 13038Mi ds-cts-1 8606m 13239Mi ds-cts-2 8526m 13060Mi ds-idrepo-0 692m 3364Mi ds-idrepo-1 710m 3262Mi ds-idrepo-2 689m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-747d57dcb8-lclbf 8m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1040m 7114Mi 13:04:57 DEBUG --- stderr --- 13:04:57 DEBUG 13:05:03 INFO 13:05:03 INFO [loop_until]: kubectl --namespace=xlou top node 13:05:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:05:03 INFO [loop_until]: OK (rc = 0) 13:05:03 DEBUG --- stdout --- 13:05:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6129Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 765m 2% 6895Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2405m 8% 11778Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3235Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8987m 30% 16064Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 136m 0% 5161Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2618m 8% 11591Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 81m 0% 3091Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2747m 9% 10892Mi 9% gke-xlou-cdm-ds-0f601751-j488 760m 2% 6367Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8498m 28% 18011Mi 16% gke-xlou-cdm-ds-0f601751-nj6q 788m 2% 7024Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8830m 29% 15469Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1051m 6% 10915Mi 18% 13:05:03 DEBUG --- stderr --- 13:05:03 DEBUG 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528338" 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "pod": "ds-cts-0" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "pod": "ds-cts-2" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG "0.0001" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 ERROR [http_cmd]: ERROR 13:05:38 ERROR --- status code --- 13:05:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:05:38 ERROR --- http response --- 13:05:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:05:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "pod": "ds-cts-0" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "pod": "ds-cts-2" 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG }, 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG "value": [ 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG "9.8" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:38 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG "0" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "pod": "ds-idrepo-0" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "pod": "ds-cts-2" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG "0" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "pod": "ds-idrepo-0" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "pod": "ds-idrepo-2" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG "0" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG "805275" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG "2501.3333333333335" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528338" 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG "1.0775921772429249" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528338" 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG "0" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:38 INFO [http_cmd]: http status code OK 13:05:38 DEBUG --- status code --- 13:05:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:38 DEBUG --- http response --- 13:05:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:38 DEBUG ----- output ----- 13:05:38 DEBUG { 13:05:38 DEBUG "status": "success", 13:05:38 DEBUG "data": { 13:05:38 DEBUG "resultType": "vector", 13:05:38 DEBUG "result": [ 13:05:38 DEBUG { 13:05:38 DEBUG "metric": { 13:05:38 DEBUG "pod": "ds-idrepo-0" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG [...] 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG 13:05:38 DEBUG "pod": "ds-idrepo-2" 13:05:38 DEBUG }, 13:05:38 DEBUG "value": [ 13:05:38 DEBUG 1713528338, 13:05:38 DEBUG "0" 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG ] 13:05:38 DEBUG } 13:05:38 DEBUG } 13:05:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528339" 13:05:39 INFO [http_cmd]: http status code OK 13:05:39 DEBUG --- status code --- 13:05:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:39 DEBUG --- http response --- 13:05:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:39 DEBUG ----- output ----- 13:05:39 DEBUG { 13:05:39 DEBUG "status": "success", 13:05:39 DEBUG "data": { 13:05:39 DEBUG "resultType": "vector", 13:05:39 DEBUG "result": [ 13:05:39 DEBUG { 13:05:39 DEBUG "metric": { 13:05:39 DEBUG "instance": "10.142.0.107:9100" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG [...] 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG "instance": "10.142.0.43:9100" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 1713528339, 13:05:39 DEBUG "0.1830000000000022" 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG } 13:05:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528339" 13:05:39 INFO [http_cmd]: http status code OK 13:05:39 DEBUG --- status code --- 13:05:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:39 DEBUG --- http response --- 13:05:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:39 DEBUG ----- output ----- 13:05:39 DEBUG { 13:05:39 DEBUG "status": "success", 13:05:39 DEBUG "data": { 13:05:39 DEBUG "resultType": "vector", 13:05:39 DEBUG "result": [ 13:05:39 DEBUG { 13:05:39 DEBUG "metric": { 13:05:39 DEBUG "instance": "10.142.0.107:9100" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG [...] 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG "instance": "10.142.0.43:9100" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 1713528339, 13:05:39 DEBUG "0.32663333333333544" 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG } 13:05:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528339" 13:05:39 INFO [http_cmd]: http status code OK 13:05:39 DEBUG --- status code --- 13:05:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:39 DEBUG --- http response --- 13:05:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:39 DEBUG ----- output ----- 13:05:39 DEBUG { 13:05:39 DEBUG "status": "success", 13:05:39 DEBUG "data": { 13:05:39 DEBUG "resultType": "vector", 13:05:39 DEBUG "result": [ 13:05:39 DEBUG { 13:05:39 DEBUG "metric": { 13:05:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG [...] 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 1713528339, 13:05:39 DEBUG "0" 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG } 13:05:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528339" 13:05:39 INFO [http_cmd]: http status code OK 13:05:39 DEBUG --- status code --- 13:05:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:39 DEBUG --- http response --- 13:05:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:39 DEBUG ----- output ----- 13:05:39 DEBUG { 13:05:39 DEBUG "status": "success", 13:05:39 DEBUG "data": { 13:05:39 DEBUG "resultType": "vector", 13:05:39 DEBUG "result": [ 13:05:39 DEBUG { 13:05:39 DEBUG "metric": { 13:05:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG [...] 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 1713528339, 13:05:39 DEBUG "0" 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG } 13:05:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528339" 13:05:39 INFO [http_cmd]: http status code OK 13:05:39 DEBUG --- status code --- 13:05:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:39 DEBUG --- http response --- 13:05:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:39 DEBUG ----- output ----- 13:05:39 DEBUG { 13:05:39 DEBUG "status": "success", 13:05:39 DEBUG "data": { 13:05:39 DEBUG "resultType": "vector", 13:05:39 DEBUG "result": [ 13:05:39 DEBUG { 13:05:39 DEBUG "metric": { 13:05:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG [...] 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG 13:05:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:05:39 DEBUG }, 13:05:39 DEBUG "value": [ 13:05:39 DEBUG 1713528339, 13:05:39 DEBUG "0" 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG ] 13:05:39 DEBUG } 13:05:39 DEBUG } 13:05:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528340" 13:05:40 INFO [http_cmd]: http status code OK 13:05:40 DEBUG --- status code --- 13:05:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:40 DEBUG --- http response --- 13:05:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:40 DEBUG ----- output ----- 13:05:40 DEBUG { 13:05:40 DEBUG "status": "success", 13:05:40 DEBUG "data": { 13:05:40 DEBUG "resultType": "vector", 13:05:40 DEBUG "result": [ 13:05:40 DEBUG { 13:05:40 DEBUG "metric": { 13:05:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:05:40 DEBUG }, 13:05:40 DEBUG "value": [ 13:05:40 DEBUG 13:05:40 DEBUG 13:05:40 DEBUG 13:05:40 DEBUG [...] 13:05:40 DEBUG 13:05:40 DEBUG 13:05:40 DEBUG 13:05:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:05:40 DEBUG }, 13:05:40 DEBUG "value": [ 13:05:40 DEBUG 1713528340, 13:05:40 DEBUG "7556251648" 13:05:40 DEBUG ] 13:05:40 DEBUG } 13:05:40 DEBUG ] 13:05:40 DEBUG } 13:05:40 DEBUG } 13:05:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528341" 13:05:41 INFO [http_cmd]: http status code OK 13:05:41 DEBUG --- status code --- 13:05:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:41 DEBUG --- http response --- 13:05:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:05:41 DEBUG ----- output ----- 13:05:41 DEBUG { 13:05:41 DEBUG "status": "success", 13:05:41 DEBUG "data": { 13:05:41 DEBUG "resultType": "vector", 13:05:41 DEBUG "result": [ 13:05:41 DEBUG { 13:05:41 DEBUG "metric": { 13:05:41 DEBUG "pod": "ds-cts-0" 13:05:41 DEBUG }, 13:05:41 DEBUG "value": [ 13:05:41 DEBUG 13:05:41 DEBUG 13:05:41 DEBUG 13:05:41 DEBUG [...] 13:05:41 DEBUG 13:05:41 DEBUG 13:05:41 DEBUG 13:05:41 DEBUG "pod": "ds-cts-2" 13:05:41 DEBUG }, 13:05:41 DEBUG "value": [ 13:05:41 DEBUG 1713528341, 13:05:41 DEBUG "29.966666666666665" 13:05:41 DEBUG ] 13:05:41 DEBUG } 13:05:41 DEBUG ] 13:05:41 DEBUG } 13:05:41 DEBUG } 13:05:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528342" 13:05:42 INFO [http_cmd]: http status code OK 13:05:42 DEBUG --- status code --- 13:05:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:42 DEBUG --- http response --- 13:05:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:42 DEBUG ----- output ----- 13:05:42 DEBUG { 13:05:42 DEBUG "status": "success", 13:05:42 DEBUG "data": { 13:05:42 DEBUG "resultType": "vector", 13:05:42 DEBUG "result": [ 13:05:42 DEBUG { 13:05:42 DEBUG "metric": { 13:05:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG [...] 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 1713528342, 13:05:42 DEBUG "55.19" 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG } 13:05:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528342" 13:05:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528342" 13:05:42 INFO [http_cmd]: http status code OK 13:05:42 DEBUG --- status code --- 13:05:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:42 DEBUG --- http response --- 13:05:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:05:42 DEBUG ----- output ----- 13:05:42 DEBUG { 13:05:42 DEBUG "status": "success", 13:05:42 DEBUG "data": { 13:05:42 DEBUG "resultType": "vector", 13:05:42 DEBUG "result": [ 13:05:42 DEBUG { 13:05:42 DEBUG "metric": { 13:05:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG [...] 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 1713528342, 13:05:42 DEBUG "0.002926222695084245" 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG } 13:05:42 INFO [http_cmd]: http status code OK 13:05:42 DEBUG --- status code --- 13:05:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:42 DEBUG --- http response --- 13:05:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:42 DEBUG ----- output ----- 13:05:42 DEBUG { 13:05:42 DEBUG "status": "success", 13:05:42 DEBUG "data": { 13:05:42 DEBUG "resultType": "vector", 13:05:42 DEBUG "result": [ 13:05:42 DEBUG { 13:05:42 DEBUG "metric": { 13:05:42 DEBUG "pod": "ds-idrepo-0" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG [...] 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG "pod": "ds-idrepo-2" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 1713528342, 13:05:42 DEBUG "0" 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG } 13:05:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528342" 13:05:42 INFO [http_cmd]: http status code OK 13:05:42 DEBUG --- status code --- 13:05:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:42 DEBUG --- http response --- 13:05:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:05:42 DEBUG ----- output ----- 13:05:42 DEBUG { 13:05:42 DEBUG "status": "success", 13:05:42 DEBUG "data": { 13:05:42 DEBUG "resultType": "vector", 13:05:42 DEBUG "result": [ 13:05:42 DEBUG { 13:05:42 DEBUG "metric": { 13:05:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG [...] 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 1713528342, 13:05:42 DEBUG "4357.881437643382" 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG } 13:05:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528342" 13:05:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528342" 13:05:42 INFO [http_cmd]: http status code OK 13:05:42 DEBUG --- status code --- 13:05:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:42 DEBUG --- http response --- 13:05:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:42 DEBUG ----- output ----- 13:05:42 DEBUG { 13:05:42 DEBUG "status": "success", 13:05:42 DEBUG "data": { 13:05:42 DEBUG "resultType": "vector", 13:05:42 DEBUG "result": [ 13:05:42 DEBUG { 13:05:42 DEBUG "metric": { 13:05:42 DEBUG "instance": "10.142.0.107:9100" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG [...] 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG "instance": "10.142.0.43:9100" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 1713528342, 13:05:42 DEBUG "1601541178368" 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG } 13:05:42 INFO [http_cmd]: http status code OK 13:05:42 DEBUG --- status code --- 13:05:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:42 DEBUG --- http response --- 13:05:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:05:42 DEBUG ----- output ----- 13:05:42 DEBUG { 13:05:42 DEBUG "status": "success", 13:05:42 DEBUG "data": { 13:05:42 DEBUG "resultType": "vector", 13:05:42 DEBUG "result": [ 13:05:42 DEBUG { 13:05:42 DEBUG "metric": { 13:05:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG [...] 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 1713528342, 13:05:42 DEBUG "0" 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG } 13:05:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528342" 13:05:42 INFO [http_cmd]: http status code OK 13:05:42 DEBUG --- status code --- 13:05:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:42 DEBUG --- http response --- 13:05:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:42 DEBUG ----- output ----- 13:05:42 DEBUG { 13:05:42 DEBUG "status": "success", 13:05:42 DEBUG "data": { 13:05:42 DEBUG "resultType": "vector", 13:05:42 DEBUG "result": [ 13:05:42 DEBUG { 13:05:42 DEBUG "metric": { 13:05:42 DEBUG "instance": "10.142.0.107:9100" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG [...] 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG "instance": "10.142.0.43:9100" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 1713528342, 13:05:42 DEBUG "16363770.566666668" 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG } 13:05:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528342" 13:05:42 INFO [http_cmd]: http status code OK 13:05:42 DEBUG --- status code --- 13:05:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:05:42 DEBUG --- http response --- 13:05:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:05:42 DEBUG ----- output ----- 13:05:42 DEBUG { 13:05:42 DEBUG "status": "success", 13:05:42 DEBUG "data": { 13:05:42 DEBUG "resultType": "vector", 13:05:42 DEBUG "result": [ 13:05:42 DEBUG { 13:05:42 DEBUG "metric": { 13:05:42 DEBUG "instance": "10.142.0.107:9100" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG [...] 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG 13:05:42 DEBUG "instance": "10.142.0.43:9100" 13:05:42 DEBUG }, 13:05:42 DEBUG "value": [ 13:05:42 DEBUG 1713528342, 13:05:42 DEBUG "16371066.233333332" 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG ] 13:05:42 DEBUG } 13:05:42 DEBUG } 13:05:57 INFO 13:05:57 INFO [loop_until]: kubectl --namespace=xlou top pods 13:05:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:05:57 INFO [loop_until]: OK (rc = 0) 13:05:57 DEBUG --- stdout --- 13:05:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2422m 8764Mi am-6fd554f8f7-hvvkw 2479m 9007Mi am-6fd554f8f7-pmj4b 2660m 8653Mi ds-cts-0 8749m 12896Mi ds-cts-1 8086m 12712Mi ds-cts-2 8071m 12933Mi ds-idrepo-0 697m 3365Mi ds-idrepo-1 701m 3263Mi ds-idrepo-2 713m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 4m 1215Mi lodemon-747d57dcb8-lclbf 6m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1013m 7141Mi 13:05:57 DEBUG --- stderr --- 13:05:57 DEBUG 13:06:03 INFO 13:06:03 INFO [loop_until]: kubectl --namespace=xlou top node 13:06:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:06:03 INFO [loop_until]: OK (rc = 0) 13:06:03 DEBUG --- stdout --- 13:06:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6142Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 774m 2% 6879Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2437m 8% 11768Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3233Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8271m 27% 15932Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 5165Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2585m 8% 11593Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3096Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2753m 9% 10887Mi 9% gke-xlou-cdm-ds-0f601751-j488 767m 2% 6371Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8069m 27% 17520Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 788m 2% 7021Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8382m 28% 15289Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1080m 6% 10945Mi 18% 13:06:03 DEBUG --- stderr --- 13:06:03 DEBUG 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528398" 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528398" 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528398" 13:06:38 ERROR [http_cmd]: ERROR 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 ERROR --- status code --- 13:06:38 DEBUG --- status code --- 13:06:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 ERROR --- http response --- 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:06:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:06:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:06:38 DEBUG ----- output ----- 13:06:38 DEBUG { 13:06:38 DEBUG "status": "success", 13:06:38 DEBUG "data": { 13:06:38 DEBUG "resultType": "vector", 13:06:38 DEBUG "result": [ 13:06:38 DEBUG { 13:06:38 DEBUG "metric": { 13:06:38 DEBUG "pod": "ds-cts-0" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG [...] 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG "pod": "ds-cts-2" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 1713528398, 13:06:38 DEBUG "0" 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG } 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 DEBUG --- status code --- 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:06:38 DEBUG ----- output ----- 13:06:38 DEBUG { 13:06:38 DEBUG "status": "success", 13:06:38 DEBUG "data": { 13:06:38 DEBUG "resultType": "vector", 13:06:38 DEBUG "result": [ 13:06:38 DEBUG { 13:06:38 DEBUG "metric": { 13:06:38 DEBUG "pod": "ds-cts-0" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG [...] 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG "pod": "ds-cts-2" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 1713528398, 13:06:38 DEBUG "0" 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG } 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528398" 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528398" 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528398" 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 DEBUG --- status code --- 13:06:38 DEBUG --- status code --- 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:06:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:06:38 DEBUG ----- output ----- 13:06:38 DEBUG { 13:06:38 DEBUG "status": "success", 13:06:38 DEBUG "data": { 13:06:38 DEBUG "resultType": "vector", 13:06:38 DEBUG "result": [ 13:06:38 DEBUG { 13:06:38 DEBUG "metric": { 13:06:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG [...] 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 1713528398, 13:06:38 DEBUG "0" 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG } 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528398" 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 DEBUG --- status code --- 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:38 DEBUG ----- output ----- 13:06:38 DEBUG { 13:06:38 DEBUG "status": "success", 13:06:38 DEBUG "data": { 13:06:38 DEBUG "resultType": "vector", 13:06:38 DEBUG "result": [ 13:06:38 DEBUG { 13:06:38 DEBUG "metric": { 13:06:38 DEBUG "pod": "ds-idrepo-0" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG [...] 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG "pod": "ds-idrepo-2" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 1713528398, 13:06:38 DEBUG "0" 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG } 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 DEBUG --- status code --- 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:38 DEBUG ----- output ----- 13:06:38 DEBUG { 13:06:38 DEBUG "status": "success", 13:06:38 DEBUG "data": { 13:06:38 DEBUG "resultType": "vector", 13:06:38 DEBUG "result": [ 13:06:38 DEBUG { 13:06:38 DEBUG "metric": { 13:06:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG [...] 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 1713528398, 13:06:38 DEBUG "676208" 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG } 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528398" 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 DEBUG --- status code --- 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:06:38 DEBUG ----- output ----- 13:06:38 DEBUG { 13:06:38 DEBUG "status": "success", 13:06:38 DEBUG "data": { 13:06:38 DEBUG "resultType": "vector", 13:06:38 DEBUG "result": [ 13:06:38 DEBUG { 13:06:38 DEBUG "metric": { 13:06:38 DEBUG "pod": "ds-idrepo-0" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG [...] 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG "pod": "ds-cts-2" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 1713528398, 13:06:38 DEBUG "0" 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG } 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528398" 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 DEBUG --- status code --- 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:38 DEBUG ----- output ----- 13:06:38 DEBUG { 13:06:38 DEBUG "status": "success", 13:06:38 DEBUG "data": { 13:06:38 DEBUG "resultType": "vector", 13:06:38 DEBUG "result": [ 13:06:38 DEBUG { 13:06:38 DEBUG "metric": { 13:06:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG [...] 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 1713528398, 13:06:38 DEBUG "1.0314774310285695" 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG } 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528398" 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 DEBUG --- status code --- 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:38 DEBUG ----- output ----- 13:06:38 DEBUG { 13:06:38 DEBUG "status": "success", 13:06:38 DEBUG "data": { 13:06:38 DEBUG "resultType": "vector", 13:06:38 DEBUG "result": [ 13:06:38 DEBUG { 13:06:38 DEBUG "metric": { 13:06:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG [...] 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 1713528398, 13:06:38 DEBUG "1474.2333333333333" 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG } 13:06:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528398" 13:06:38 INFO [http_cmd]: http status code OK 13:06:38 DEBUG --- status code --- 13:06:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:38 DEBUG --- http response --- 13:06:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:38 DEBUG ----- output ----- 13:06:38 DEBUG { 13:06:38 DEBUG "status": "success", 13:06:38 DEBUG "data": { 13:06:38 DEBUG "resultType": "vector", 13:06:38 DEBUG "result": [ 13:06:38 DEBUG { 13:06:38 DEBUG "metric": { 13:06:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG [...] 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG 13:06:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:06:38 DEBUG }, 13:06:38 DEBUG "value": [ 13:06:38 DEBUG 1713528398, 13:06:38 DEBUG "0" 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG ] 13:06:38 DEBUG } 13:06:38 DEBUG } 13:06:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528399" 13:06:39 INFO [http_cmd]: http status code OK 13:06:39 DEBUG --- status code --- 13:06:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:39 DEBUG --- http response --- 13:06:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:39 DEBUG ----- output ----- 13:06:39 DEBUG { 13:06:39 DEBUG "status": "success", 13:06:39 DEBUG "data": { 13:06:39 DEBUG "resultType": "vector", 13:06:39 DEBUG "result": [ 13:06:39 DEBUG { 13:06:39 DEBUG "metric": { 13:06:39 DEBUG "pod": "ds-idrepo-0" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG [...] 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG "pod": "ds-idrepo-2" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 1713528399, 13:06:39 DEBUG "0" 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG } 13:06:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528399" 13:06:39 INFO [http_cmd]: http status code OK 13:06:39 DEBUG --- status code --- 13:06:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:39 DEBUG --- http response --- 13:06:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:39 DEBUG ----- output ----- 13:06:39 DEBUG { 13:06:39 DEBUG "status": "success", 13:06:39 DEBUG "data": { 13:06:39 DEBUG "resultType": "vector", 13:06:39 DEBUG "result": [ 13:06:39 DEBUG { 13:06:39 DEBUG "metric": { 13:06:39 DEBUG "instance": "10.142.0.107:9100" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG [...] 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG "instance": "10.142.0.43:9100" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 1713528399, 13:06:39 DEBUG "0.07633333333333213" 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG } 13:06:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528399" 13:06:39 INFO [http_cmd]: http status code OK 13:06:39 DEBUG --- status code --- 13:06:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:39 DEBUG --- http response --- 13:06:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:39 DEBUG ----- output ----- 13:06:39 DEBUG { 13:06:39 DEBUG "status": "success", 13:06:39 DEBUG "data": { 13:06:39 DEBUG "resultType": "vector", 13:06:39 DEBUG "result": [ 13:06:39 DEBUG { 13:06:39 DEBUG "metric": { 13:06:39 DEBUG "instance": "10.142.0.107:9100" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG [...] 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG "instance": "10.142.0.43:9100" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 1713528399, 13:06:39 DEBUG "0.1215333333332192" 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG } 13:06:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528399" 13:06:39 INFO [http_cmd]: http status code OK 13:06:39 DEBUG --- status code --- 13:06:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:39 DEBUG --- http response --- 13:06:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:39 DEBUG ----- output ----- 13:06:39 DEBUG { 13:06:39 DEBUG "status": "success", 13:06:39 DEBUG "data": { 13:06:39 DEBUG "resultType": "vector", 13:06:39 DEBUG "result": [ 13:06:39 DEBUG { 13:06:39 DEBUG "metric": { 13:06:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG [...] 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 1713528399, 13:06:39 DEBUG "0" 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG } 13:06:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528399" 13:06:39 INFO [http_cmd]: http status code OK 13:06:39 DEBUG --- status code --- 13:06:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:39 DEBUG --- http response --- 13:06:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:39 DEBUG ----- output ----- 13:06:39 DEBUG { 13:06:39 DEBUG "status": "success", 13:06:39 DEBUG "data": { 13:06:39 DEBUG "resultType": "vector", 13:06:39 DEBUG "result": [ 13:06:39 DEBUG { 13:06:39 DEBUG "metric": { 13:06:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG [...] 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 1713528399, 13:06:39 DEBUG "0" 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG } 13:06:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528399" 13:06:39 INFO [http_cmd]: http status code OK 13:06:39 DEBUG --- status code --- 13:06:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:39 DEBUG --- http response --- 13:06:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:39 DEBUG ----- output ----- 13:06:39 DEBUG { 13:06:39 DEBUG "status": "success", 13:06:39 DEBUG "data": { 13:06:39 DEBUG "resultType": "vector", 13:06:39 DEBUG "result": [ 13:06:39 DEBUG { 13:06:39 DEBUG "metric": { 13:06:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG [...] 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG 13:06:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:06:39 DEBUG }, 13:06:39 DEBUG "value": [ 13:06:39 DEBUG 1713528399, 13:06:39 DEBUG "0" 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG ] 13:06:39 DEBUG } 13:06:39 DEBUG } 13:06:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528400" 13:06:40 INFO [http_cmd]: http status code OK 13:06:40 DEBUG --- status code --- 13:06:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:40 DEBUG --- http response --- 13:06:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:40 DEBUG ----- output ----- 13:06:40 DEBUG { 13:06:40 DEBUG "status": "success", 13:06:40 DEBUG "data": { 13:06:40 DEBUG "resultType": "vector", 13:06:40 DEBUG "result": [ 13:06:40 DEBUG { 13:06:40 DEBUG "metric": { 13:06:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:06:40 DEBUG }, 13:06:40 DEBUG "value": [ 13:06:40 DEBUG 13:06:40 DEBUG 13:06:40 DEBUG 13:06:40 DEBUG [...] 13:06:40 DEBUG 13:06:40 DEBUG 13:06:40 DEBUG 13:06:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:06:40 DEBUG }, 13:06:40 DEBUG "value": [ 13:06:40 DEBUG 1713528400, 13:06:40 DEBUG "7583932416" 13:06:40 DEBUG ] 13:06:40 DEBUG } 13:06:40 DEBUG ] 13:06:40 DEBUG } 13:06:40 DEBUG } 13:06:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528401" 13:06:41 INFO [http_cmd]: http status code OK 13:06:41 DEBUG --- status code --- 13:06:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:41 DEBUG --- http response --- 13:06:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:06:41 DEBUG ----- output ----- 13:06:41 DEBUG { 13:06:41 DEBUG "status": "success", 13:06:41 DEBUG "data": { 13:06:41 DEBUG "resultType": "vector", 13:06:41 DEBUG "result": [ 13:06:41 DEBUG { 13:06:41 DEBUG "metric": { 13:06:41 DEBUG "pod": "ds-cts-0" 13:06:41 DEBUG }, 13:06:41 DEBUG "value": [ 13:06:41 DEBUG 13:06:41 DEBUG 13:06:41 DEBUG 13:06:41 DEBUG [...] 13:06:41 DEBUG 13:06:41 DEBUG 13:06:41 DEBUG 13:06:41 DEBUG "pod": "ds-cts-2" 13:06:41 DEBUG }, 13:06:41 DEBUG "value": [ 13:06:41 DEBUG 1713528401, 13:06:41 DEBUG "30" 13:06:41 DEBUG ] 13:06:41 DEBUG } 13:06:41 DEBUG ] 13:06:41 DEBUG } 13:06:41 DEBUG } 13:06:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528402" 13:06:42 INFO [http_cmd]: http status code OK 13:06:42 DEBUG --- status code --- 13:06:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:42 DEBUG --- http response --- 13:06:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:42 DEBUG ----- output ----- 13:06:42 DEBUG { 13:06:42 DEBUG "status": "success", 13:06:42 DEBUG "data": { 13:06:42 DEBUG "resultType": "vector", 13:06:42 DEBUG "result": [ 13:06:42 DEBUG { 13:06:42 DEBUG "metric": { 13:06:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG [...] 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 1713528402, 13:06:42 DEBUG "55.19" 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG } 13:06:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528402" 13:06:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528402" 13:06:42 INFO [http_cmd]: http status code OK 13:06:42 DEBUG --- status code --- 13:06:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:42 DEBUG --- http response --- 13:06:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:06:42 DEBUG ----- output ----- 13:06:42 DEBUG { 13:06:42 DEBUG "status": "success", 13:06:42 DEBUG "data": { 13:06:42 DEBUG "resultType": "vector", 13:06:42 DEBUG "result": [ 13:06:42 DEBUG { 13:06:42 DEBUG "metric": { 13:06:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG [...] 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 1713528402, 13:06:42 DEBUG "0.0032856814449922075" 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG } 13:06:42 INFO [http_cmd]: http status code OK 13:06:42 DEBUG --- status code --- 13:06:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:42 DEBUG --- http response --- 13:06:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:42 DEBUG ----- output ----- 13:06:42 DEBUG { 13:06:42 DEBUG "status": "success", 13:06:42 DEBUG "data": { 13:06:42 DEBUG "resultType": "vector", 13:06:42 DEBUG "result": [ 13:06:42 DEBUG { 13:06:42 DEBUG "metric": { 13:06:42 DEBUG "pod": "ds-idrepo-0" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG [...] 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG "pod": "ds-idrepo-2" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 1713528402, 13:06:42 DEBUG "0" 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG } 13:06:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528402" 13:06:42 INFO [http_cmd]: http status code OK 13:06:42 DEBUG --- status code --- 13:06:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:42 DEBUG --- http response --- 13:06:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:06:42 DEBUG ----- output ----- 13:06:42 DEBUG { 13:06:42 DEBUG "status": "success", 13:06:42 DEBUG "data": { 13:06:42 DEBUG "resultType": "vector", 13:06:42 DEBUG "result": [ 13:06:42 DEBUG { 13:06:42 DEBUG "metric": { 13:06:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG [...] 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG "pod": "idm-b85f9d978-wnhxb" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 1713528402, 13:06:42 DEBUG "42054553.4318555" 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG } 13:06:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528402" 13:06:42 INFO [http_cmd]: http status code OK 13:06:42 DEBUG --- status code --- 13:06:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:42 DEBUG --- http response --- 13:06:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:42 DEBUG ----- output ----- 13:06:42 DEBUG { 13:06:42 DEBUG "status": "success", 13:06:42 DEBUG "data": { 13:06:42 DEBUG "resultType": "vector", 13:06:42 DEBUG "result": [ 13:06:42 DEBUG { 13:06:42 DEBUG "metric": { 13:06:42 DEBUG "instance": "10.142.0.107:9100" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG [...] 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG "instance": "10.142.0.43:9100" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 1713528402, 13:06:42 DEBUG "1602637982720" 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG } 13:06:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528402" 13:06:42 INFO [http_cmd]: http status code OK 13:06:42 DEBUG --- status code --- 13:06:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:42 DEBUG --- http response --- 13:06:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:06:42 DEBUG ----- output ----- 13:06:42 DEBUG { 13:06:42 DEBUG "status": "success", 13:06:42 DEBUG "data": { 13:06:42 DEBUG "resultType": "vector", 13:06:42 DEBUG "result": [ 13:06:42 DEBUG { 13:06:42 DEBUG "metric": { 13:06:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG [...] 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG 13:06:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:06:42 DEBUG }, 13:06:42 DEBUG "value": [ 13:06:42 DEBUG 1713528402, 13:06:42 DEBUG "0" 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG ] 13:06:42 DEBUG } 13:06:42 DEBUG } 13:06:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528403" 13:06:43 INFO [http_cmd]: http status code OK 13:06:43 DEBUG --- status code --- 13:06:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:43 DEBUG --- http response --- 13:06:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:43 DEBUG ----- output ----- 13:06:43 DEBUG { 13:06:43 DEBUG "status": "success", 13:06:43 DEBUG "data": { 13:06:43 DEBUG "resultType": "vector", 13:06:43 DEBUG "result": [ 13:06:43 DEBUG { 13:06:43 DEBUG "metric": { 13:06:43 DEBUG "instance": "10.142.0.107:9100" 13:06:43 DEBUG }, 13:06:43 DEBUG "value": [ 13:06:43 DEBUG 13:06:43 DEBUG 13:06:43 DEBUG 13:06:43 DEBUG [...] 13:06:43 DEBUG 13:06:43 DEBUG 13:06:43 DEBUG 13:06:43 DEBUG "instance": "10.142.0.43:9100" 13:06:43 DEBUG }, 13:06:43 DEBUG "value": [ 13:06:43 DEBUG 1713528403, 13:06:43 DEBUG "9691980.666666668" 13:06:43 DEBUG ] 13:06:43 DEBUG } 13:06:43 DEBUG ] 13:06:43 DEBUG } 13:06:43 DEBUG } 13:06:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528403" 13:06:43 INFO [http_cmd]: http status code OK 13:06:43 DEBUG --- status code --- 13:06:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:06:43 DEBUG --- http response --- 13:06:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:06:43 DEBUG ----- output ----- 13:06:43 DEBUG { 13:06:43 DEBUG "status": "success", 13:06:43 DEBUG "data": { 13:06:43 DEBUG "resultType": "vector", 13:06:43 DEBUG "result": [ 13:06:43 DEBUG { 13:06:43 DEBUG "metric": { 13:06:43 DEBUG "instance": "10.142.0.107:9100" 13:06:43 DEBUG }, 13:06:43 DEBUG "value": [ 13:06:43 DEBUG 13:06:43 DEBUG 13:06:43 DEBUG 13:06:43 DEBUG [...] 13:06:43 DEBUG 13:06:43 DEBUG 13:06:43 DEBUG 13:06:43 DEBUG "instance": "10.142.0.43:9100" 13:06:43 DEBUG }, 13:06:43 DEBUG "value": [ 13:06:43 DEBUG 1713528403, 13:06:43 DEBUG "9699448.233333332" 13:06:43 DEBUG ] 13:06:43 DEBUG } 13:06:43 DEBUG ] 13:06:43 DEBUG } 13:06:43 DEBUG } 13:06:57 INFO 13:06:57 INFO [loop_until]: kubectl --namespace=xlou top pods 13:06:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:06:57 INFO [loop_until]: OK (rc = 0) 13:06:57 DEBUG --- stdout --- 13:06:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 1455m 8764Mi am-6fd554f8f7-hvvkw 1470m 9006Mi am-6fd554f8f7-pmj4b 1488m 8649Mi ds-cts-0 4645m 12837Mi ds-cts-1 3919m 12525Mi ds-cts-2 4599m 12809Mi ds-idrepo-0 415m 3366Mi ds-idrepo-1 413m 3263Mi ds-idrepo-2 422m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-747d57dcb8-lclbf 6m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1013m 7172Mi 13:06:57 DEBUG --- stderr --- 13:06:57 DEBUG 13:07:03 INFO 13:07:03 INFO [loop_until]: kubectl --namespace=xlou top node 13:07:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:07:03 INFO [loop_until]: OK (rc = 0) 13:07:03 DEBUG --- stdout --- 13:07:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6143Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 481m 1% 6888Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1398m 4% 11770Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3236Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 4739m 15% 15869Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5173Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1565m 5% 11586Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 3086Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1603m 5% 10887Mi 9% gke-xlou-cdm-ds-0f601751-j488 452m 1% 6366Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 4184m 14% 17315Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 494m 1% 7025Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 4347m 14% 15268Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1080m 6% 10981Mi 18% 13:07:03 DEBUG --- stderr --- 13:07:03 DEBUG 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528458" 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528458" 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528458" 13:07:38 ERROR [http_cmd]: ERROR 13:07:38 INFO [http_cmd]: http status code OK 13:07:38 ERROR --- status code --- 13:07:38 DEBUG --- status code --- 13:07:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:07:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:38 ERROR --- http response --- 13:07:38 DEBUG --- http response --- 13:07:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:07:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:07:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:07:38 DEBUG ----- output ----- 13:07:38 DEBUG { 13:07:38 DEBUG "status": "success", 13:07:38 DEBUG "data": { 13:07:38 DEBUG "resultType": "vector", 13:07:38 DEBUG "result": [ 13:07:38 DEBUG { 13:07:38 DEBUG "metric": { 13:07:38 DEBUG "pod": "ds-cts-0" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG [...] 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG "pod": "ds-cts-2" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 1713528458, 13:07:38 DEBUG "6.4" 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG } 13:07:38 INFO [http_cmd]: http status code OK 13:07:38 DEBUG --- status code --- 13:07:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:38 DEBUG --- http response --- 13:07:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:07:38 DEBUG ----- output ----- 13:07:38 DEBUG { 13:07:38 DEBUG "status": "success", 13:07:38 DEBUG "data": { 13:07:38 DEBUG "resultType": "vector", 13:07:38 DEBUG "result": [ 13:07:38 DEBUG { 13:07:38 DEBUG "metric": { 13:07:38 DEBUG "pod": "ds-cts-0" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG [...] 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG "pod": "ds-cts-2" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 1713528458, 13:07:38 DEBUG "0.000033333333333333335" 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG } 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528458" 13:07:38 INFO [http_cmd]: http status code OK 13:07:38 DEBUG --- status code --- 13:07:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:38 DEBUG --- http response --- 13:07:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:07:38 DEBUG ----- output ----- 13:07:38 DEBUG { 13:07:38 DEBUG "status": "success", 13:07:38 DEBUG "data": { 13:07:38 DEBUG "resultType": "vector", 13:07:38 DEBUG "result": [ 13:07:38 DEBUG { 13:07:38 DEBUG "metric": { 13:07:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG [...] 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG "pod": "lodemon-747d57dcb8-lclbf" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 1713528458, 13:07:38 DEBUG "0" 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG } 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528458" 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528458" 13:07:38 INFO [http_cmd]: http status code OK 13:07:38 DEBUG --- status code --- 13:07:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:38 DEBUG --- http response --- 13:07:38 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:07:38 INFO [http_cmd]: http status code OK 13:07:38 DEBUG --- status code --- 13:07:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:38 DEBUG --- http response --- 13:07:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:07:38 DEBUG ----- output ----- 13:07:38 DEBUG { 13:07:38 DEBUG "status": "success", 13:07:38 DEBUG "data": { 13:07:38 DEBUG "resultType": "vector", 13:07:38 DEBUG "result": [ 13:07:38 DEBUG { 13:07:38 DEBUG "metric": { 13:07:38 DEBUG "pod": "ds-idrepo-0" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG [...] 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG "pod": "ds-cts-2" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 1713528458, 13:07:38 DEBUG "0" 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG } 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528458" 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528458" 13:07:38 INFO [http_cmd]: http status code OK 13:07:38 DEBUG --- status code --- 13:07:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:38 DEBUG --- http response --- 13:07:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:38 DEBUG ----- output ----- 13:07:38 DEBUG { 13:07:38 DEBUG "status": "success", 13:07:38 DEBUG "data": { 13:07:38 DEBUG "resultType": "vector", 13:07:38 DEBUG "result": [ 13:07:38 DEBUG { 13:07:38 DEBUG "metric": { 13:07:38 DEBUG "pod": "ds-idrepo-0" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG [...] 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG "pod": "ds-idrepo-2" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 1713528458, 13:07:38 DEBUG "0" 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG } 13:07:38 INFO [http_cmd]: http status code OK 13:07:38 DEBUG --- status code --- 13:07:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:38 DEBUG --- http response --- 13:07:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:38 DEBUG ----- output ----- 13:07:38 DEBUG { 13:07:38 DEBUG "status": "success", 13:07:38 DEBUG "data": { 13:07:38 DEBUG "resultType": "vector", 13:07:38 DEBUG "result": [ 13:07:38 DEBUG { 13:07:38 DEBUG "metric": { 13:07:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG [...] 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 1713528458, 13:07:38 DEBUG "592132" 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG } 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528458" 13:07:38 INFO [http_cmd]: http status code OK 13:07:38 DEBUG --- status code --- 13:07:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:38 DEBUG --- http response --- 13:07:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:38 DEBUG ----- output ----- 13:07:38 DEBUG { 13:07:38 DEBUG "status": "success", 13:07:38 DEBUG "data": { 13:07:38 DEBUG "resultType": "vector", 13:07:38 DEBUG "result": [ 13:07:38 DEBUG { 13:07:38 DEBUG "metric": { 13:07:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG [...] 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 1713528458, 13:07:38 DEBUG "1.0258214096723477" 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG } 13:07:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528458" 13:07:38 INFO [http_cmd]: http status code OK 13:07:38 DEBUG --- status code --- 13:07:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:38 DEBUG --- http response --- 13:07:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:38 DEBUG ----- output ----- 13:07:38 DEBUG { 13:07:38 DEBUG "status": "success", 13:07:38 DEBUG "data": { 13:07:38 DEBUG "resultType": "vector", 13:07:38 DEBUG "result": [ 13:07:38 DEBUG { 13:07:38 DEBUG "metric": { 13:07:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG [...] 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG 13:07:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:07:38 DEBUG }, 13:07:38 DEBUG "value": [ 13:07:38 DEBUG 1713528458, 13:07:38 DEBUG "1388.4" 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG ] 13:07:38 DEBUG } 13:07:38 DEBUG } 13:07:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528459" 13:07:39 INFO [http_cmd]: http status code OK 13:07:39 DEBUG --- status code --- 13:07:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:39 DEBUG --- http response --- 13:07:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 13:07:39 DEBUG ----- output ----- 13:07:39 DEBUG { 13:07:39 DEBUG "status": "success", 13:07:39 DEBUG "data": { 13:07:39 DEBUG "resultType": "vector", 13:07:39 DEBUG "result": [ 13:07:39 DEBUG { 13:07:39 DEBUG "metric": { 13:07:39 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG [...] 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG "pod": "ds-cts-2" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 1713528459, 13:07:39 DEBUG "58308570.81186089" 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG } 13:07:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528459" 13:07:39 INFO [http_cmd]: http status code OK 13:07:39 DEBUG --- status code --- 13:07:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:39 DEBUG --- http response --- 13:07:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:39 DEBUG ----- output ----- 13:07:39 DEBUG { 13:07:39 DEBUG "status": "success", 13:07:39 DEBUG "data": { 13:07:39 DEBUG "resultType": "vector", 13:07:39 DEBUG "result": [ 13:07:39 DEBUG { 13:07:39 DEBUG "metric": { 13:07:39 DEBUG "pod": "ds-idrepo-0" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG [...] 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG "pod": "ds-idrepo-2" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 1713528459, 13:07:39 DEBUG "0" 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG } 13:07:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528459" 13:07:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528459" 13:07:39 INFO [http_cmd]: http status code OK 13:07:39 DEBUG --- status code --- 13:07:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:39 DEBUG --- http response --- 13:07:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:39 DEBUG ----- output ----- 13:07:39 DEBUG { 13:07:39 DEBUG "status": "success", 13:07:39 DEBUG "data": { 13:07:39 DEBUG "resultType": "vector", 13:07:39 DEBUG "result": [ 13:07:39 DEBUG { 13:07:39 DEBUG "metric": { 13:07:39 DEBUG "instance": "10.142.0.107:9100" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG [...] 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG "instance": "10.142.0.43:9100" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 1713528459, 13:07:39 DEBUG "0.08533333333332961" 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG } 13:07:39 INFO [http_cmd]: http status code OK 13:07:39 DEBUG --- status code --- 13:07:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:39 DEBUG --- http response --- 13:07:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:39 DEBUG ----- output ----- 13:07:39 DEBUG { 13:07:39 DEBUG "status": "success", 13:07:39 DEBUG "data": { 13:07:39 DEBUG "resultType": "vector", 13:07:39 DEBUG "result": [ 13:07:39 DEBUG { 13:07:39 DEBUG "metric": { 13:07:39 DEBUG "instance": "10.142.0.107:9100" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG [...] 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG "instance": "10.142.0.43:9100" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 1713528459, 13:07:39 DEBUG "0.13656666666661105" 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG } 13:07:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528459" 13:07:39 INFO [http_cmd]: http status code OK 13:07:39 DEBUG --- status code --- 13:07:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:39 DEBUG --- http response --- 13:07:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:39 DEBUG ----- output ----- 13:07:39 DEBUG { 13:07:39 DEBUG "status": "success", 13:07:39 DEBUG "data": { 13:07:39 DEBUG "resultType": "vector", 13:07:39 DEBUG "result": [ 13:07:39 DEBUG { 13:07:39 DEBUG "metric": { 13:07:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG [...] 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 1713528459, 13:07:39 DEBUG "0" 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG } 13:07:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528459" 13:07:39 INFO [http_cmd]: http status code OK 13:07:39 DEBUG --- status code --- 13:07:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:39 DEBUG --- http response --- 13:07:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:39 DEBUG ----- output ----- 13:07:39 DEBUG { 13:07:39 DEBUG "status": "success", 13:07:39 DEBUG "data": { 13:07:39 DEBUG "resultType": "vector", 13:07:39 DEBUG "result": [ 13:07:39 DEBUG { 13:07:39 DEBUG "metric": { 13:07:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG [...] 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 1713528459, 13:07:39 DEBUG "0" 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG } 13:07:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528459" 13:07:39 INFO [http_cmd]: http status code OK 13:07:39 DEBUG --- status code --- 13:07:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:39 DEBUG --- http response --- 13:07:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:39 DEBUG ----- output ----- 13:07:39 DEBUG { 13:07:39 DEBUG "status": "success", 13:07:39 DEBUG "data": { 13:07:39 DEBUG "resultType": "vector", 13:07:39 DEBUG "result": [ 13:07:39 DEBUG { 13:07:39 DEBUG "metric": { 13:07:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG [...] 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG 13:07:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:07:39 DEBUG }, 13:07:39 DEBUG "value": [ 13:07:39 DEBUG 1713528459, 13:07:39 DEBUG "0" 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG ] 13:07:39 DEBUG } 13:07:39 DEBUG } 13:07:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528460" 13:07:40 INFO [http_cmd]: http status code OK 13:07:40 DEBUG --- status code --- 13:07:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:40 DEBUG --- http response --- 13:07:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:40 DEBUG ----- output ----- 13:07:40 DEBUG { 13:07:40 DEBUG "status": "success", 13:07:40 DEBUG "data": { 13:07:40 DEBUG "resultType": "vector", 13:07:40 DEBUG "result": [ 13:07:40 DEBUG { 13:07:40 DEBUG "metric": { 13:07:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:07:40 DEBUG }, 13:07:40 DEBUG "value": [ 13:07:40 DEBUG 13:07:40 DEBUG 13:07:40 DEBUG 13:07:40 DEBUG [...] 13:07:40 DEBUG 13:07:40 DEBUG 13:07:40 DEBUG 13:07:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:07:40 DEBUG }, 13:07:40 DEBUG "value": [ 13:07:40 DEBUG 1713528460, 13:07:40 DEBUG "7618621440" 13:07:40 DEBUG ] 13:07:40 DEBUG } 13:07:40 DEBUG ] 13:07:40 DEBUG } 13:07:40 DEBUG } 13:07:41 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528461" 13:07:41 INFO [http_cmd]: http status code OK 13:07:41 DEBUG --- status code --- 13:07:41 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:41 DEBUG --- http response --- 13:07:41 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:07:41 DEBUG ----- output ----- 13:07:41 DEBUG { 13:07:41 DEBUG "status": "success", 13:07:41 DEBUG "data": { 13:07:41 DEBUG "resultType": "vector", 13:07:41 DEBUG "result": [ 13:07:41 DEBUG { 13:07:41 DEBUG "metric": { 13:07:41 DEBUG "pod": "ds-cts-0" 13:07:41 DEBUG }, 13:07:41 DEBUG "value": [ 13:07:41 DEBUG 13:07:41 DEBUG 13:07:41 DEBUG 13:07:41 DEBUG [...] 13:07:41 DEBUG 13:07:41 DEBUG 13:07:41 DEBUG 13:07:41 DEBUG "pod": "ds-cts-2" 13:07:41 DEBUG }, 13:07:41 DEBUG "value": [ 13:07:41 DEBUG 1713528461, 13:07:41 DEBUG "30" 13:07:41 DEBUG ] 13:07:41 DEBUG } 13:07:41 DEBUG ] 13:07:41 DEBUG } 13:07:41 DEBUG } 13:07:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528462" 13:07:42 INFO [http_cmd]: http status code OK 13:07:42 DEBUG --- status code --- 13:07:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:42 DEBUG --- http response --- 13:07:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:42 DEBUG ----- output ----- 13:07:42 DEBUG { 13:07:42 DEBUG "status": "success", 13:07:42 DEBUG "data": { 13:07:42 DEBUG "resultType": "vector", 13:07:42 DEBUG "result": [ 13:07:42 DEBUG { 13:07:42 DEBUG "metric": { 13:07:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG [...] 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 1713528462, 13:07:42 DEBUG "55.19" 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG } 13:07:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528462" 13:07:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528462" 13:07:42 INFO [http_cmd]: http status code OK 13:07:42 DEBUG --- status code --- 13:07:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:42 DEBUG --- http response --- 13:07:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:07:42 DEBUG ----- output ----- 13:07:42 DEBUG { 13:07:42 DEBUG "status": "success", 13:07:42 DEBUG "data": { 13:07:42 DEBUG "resultType": "vector", 13:07:42 DEBUG "result": [ 13:07:42 DEBUG { 13:07:42 DEBUG "metric": { 13:07:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG [...] 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG "pod": "ds-cts-2" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 1713528462, 13:07:42 DEBUG "4.439514178560607" 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG } 13:07:42 INFO [http_cmd]: http status code OK 13:07:42 DEBUG --- status code --- 13:07:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:42 DEBUG --- http response --- 13:07:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:42 DEBUG ----- output ----- 13:07:42 DEBUG { 13:07:42 DEBUG "status": "success", 13:07:42 DEBUG "data": { 13:07:42 DEBUG "resultType": "vector", 13:07:42 DEBUG "result": [ 13:07:42 DEBUG { 13:07:42 DEBUG "metric": { 13:07:42 DEBUG "pod": "ds-idrepo-0" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG [...] 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG "pod": "ds-idrepo-2" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 1713528462, 13:07:42 DEBUG "0" 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG } 13:07:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528462" 13:07:42 INFO [http_cmd]: http status code OK 13:07:42 DEBUG --- status code --- 13:07:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:42 DEBUG --- http response --- 13:07:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 13:07:42 DEBUG ----- output ----- 13:07:42 DEBUG { 13:07:42 DEBUG "status": "success", 13:07:42 DEBUG "data": { 13:07:42 DEBUG "resultType": "vector", 13:07:42 DEBUG "result": [ 13:07:42 DEBUG { 13:07:42 DEBUG "metric": { 13:07:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG [...] 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG "pod": "ds-cts-2" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 1713528462, 13:07:42 DEBUG "361351677.3062084" 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG } 13:07:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528462" 13:07:42 INFO [http_cmd]: http status code OK 13:07:42 DEBUG --- status code --- 13:07:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:42 DEBUG --- http response --- 13:07:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:42 DEBUG ----- output ----- 13:07:42 DEBUG { 13:07:42 DEBUG "status": "success", 13:07:42 DEBUG "data": { 13:07:42 DEBUG "resultType": "vector", 13:07:42 DEBUG "result": [ 13:07:42 DEBUG { 13:07:42 DEBUG "metric": { 13:07:42 DEBUG "instance": "10.142.0.107:9100" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG [...] 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG "instance": "10.142.0.43:9100" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 1713528462, 13:07:42 DEBUG "1603507545088" 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG } 13:07:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528462" 13:07:42 INFO [http_cmd]: http status code OK 13:07:42 DEBUG --- status code --- 13:07:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:42 DEBUG --- http response --- 13:07:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:07:42 DEBUG ----- output ----- 13:07:42 DEBUG { 13:07:42 DEBUG "status": "success", 13:07:42 DEBUG "data": { 13:07:42 DEBUG "resultType": "vector", 13:07:42 DEBUG "result": [ 13:07:42 DEBUG { 13:07:42 DEBUG "metric": { 13:07:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG [...] 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG 13:07:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:07:42 DEBUG }, 13:07:42 DEBUG "value": [ 13:07:42 DEBUG 1713528462, 13:07:42 DEBUG "0" 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG ] 13:07:42 DEBUG } 13:07:42 DEBUG } 13:07:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528463" 13:07:43 INFO [http_cmd]: http status code OK 13:07:43 DEBUG --- status code --- 13:07:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:43 DEBUG --- http response --- 13:07:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:43 DEBUG ----- output ----- 13:07:43 DEBUG { 13:07:43 DEBUG "status": "success", 13:07:43 DEBUG "data": { 13:07:43 DEBUG "resultType": "vector", 13:07:43 DEBUG "result": [ 13:07:43 DEBUG { 13:07:43 DEBUG "metric": { 13:07:43 DEBUG "instance": "10.142.0.107:9100" 13:07:43 DEBUG }, 13:07:43 DEBUG "value": [ 13:07:43 DEBUG 13:07:43 DEBUG 13:07:43 DEBUG 13:07:43 DEBUG [...] 13:07:43 DEBUG 13:07:43 DEBUG 13:07:43 DEBUG 13:07:43 DEBUG "instance": "10.142.0.43:9100" 13:07:43 DEBUG }, 13:07:43 DEBUG "value": [ 13:07:43 DEBUG 1713528463, 13:07:43 DEBUG "9192003.433333334" 13:07:43 DEBUG ] 13:07:43 DEBUG } 13:07:43 DEBUG ] 13:07:43 DEBUG } 13:07:43 DEBUG } 13:07:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528463" 13:07:43 INFO [http_cmd]: http status code OK 13:07:43 DEBUG --- status code --- 13:07:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:07:43 DEBUG --- http response --- 13:07:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:07:43 DEBUG ----- output ----- 13:07:43 DEBUG { 13:07:43 DEBUG "status": "success", 13:07:43 DEBUG "data": { 13:07:43 DEBUG "resultType": "vector", 13:07:43 DEBUG "result": [ 13:07:43 DEBUG { 13:07:43 DEBUG "metric": { 13:07:43 DEBUG "instance": "10.142.0.107:9100" 13:07:43 DEBUG }, 13:07:43 DEBUG "value": [ 13:07:43 DEBUG 13:07:43 DEBUG 13:07:43 DEBUG 13:07:43 DEBUG [...] 13:07:43 DEBUG 13:07:43 DEBUG 13:07:43 DEBUG 13:07:43 DEBUG "instance": "10.142.0.43:9100" 13:07:43 DEBUG }, 13:07:43 DEBUG "value": [ 13:07:43 DEBUG 1713528463, 13:07:43 DEBUG "9197524.033333333" 13:07:43 DEBUG ] 13:07:43 DEBUG } 13:07:43 DEBUG ] 13:07:43 DEBUG } 13:07:43 DEBUG } 13:07:57 INFO 13:07:57 INFO [loop_until]: kubectl --namespace=xlou top pods 13:07:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:07:57 INFO [loop_until]: OK (rc = 0) 13:07:57 DEBUG --- stdout --- 13:07:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 1448m 8757Mi am-6fd554f8f7-hvvkw 1520m 9005Mi am-6fd554f8f7-pmj4b 1469m 8647Mi ds-cts-0 4569m 12932Mi ds-cts-1 3964m 12483Mi ds-cts-2 4291m 12805Mi ds-idrepo-0 420m 3366Mi ds-idrepo-1 392m 3262Mi ds-idrepo-2 429m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-747d57dcb8-lclbf 6m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 626m 7135Mi 13:07:57 DEBUG --- stderr --- 13:07:57 DEBUG 13:08:03 INFO 13:08:03 INFO [loop_until]: kubectl --namespace=xlou top node 13:08:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:08:04 INFO [loop_until]: OK (rc = 0) 13:08:04 DEBUG --- stdout --- 13:08:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6135Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 468m 1% 6892Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1468m 4% 11768Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 76m 0% 3242Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 4505m 15% 15968Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5183Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1518m 5% 11576Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 75m 0% 3086Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1581m 5% 10878Mi 9% gke-xlou-cdm-ds-0f601751-j488 487m 1% 6364Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 4145m 13% 17266Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 505m 1% 7037Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 4402m 14% 15269Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 622m 3% 10937Mi 18% 13:08:04 DEBUG --- stderr --- 13:08:04 DEBUG 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528518" 13:08:38 INFO [http_cmd]: http status code OK 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528518" 13:08:38 DEBUG --- status code --- 13:08:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:38 DEBUG --- http response --- 13:08:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:08:38 DEBUG ----- output ----- 13:08:38 DEBUG { 13:08:38 DEBUG "status": "success", 13:08:38 DEBUG "data": { 13:08:38 DEBUG "resultType": "vector", 13:08:38 DEBUG "result": [ 13:08:38 DEBUG { 13:08:38 DEBUG "metric": { 13:08:38 DEBUG "pod": "ds-cts-0" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG [...] 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG "pod": "ds-cts-2" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 1713528518, 13:08:38 DEBUG "0" 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG } 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528518" 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528518" 13:08:38 INFO [http_cmd]: http status code OK 13:08:38 DEBUG --- status code --- 13:08:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:38 DEBUG --- http response --- 13:08:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:08:38 DEBUG ----- output ----- 13:08:38 DEBUG { 13:08:38 DEBUG "status": "success", 13:08:38 DEBUG "data": { 13:08:38 DEBUG "resultType": "vector", 13:08:38 DEBUG "result": [ 13:08:38 DEBUG { 13:08:38 DEBUG "metric": { 13:08:38 DEBUG "pod": "ds-cts-0" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG [...] 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG "pod": "ds-cts-2" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 1713528518, 13:08:38 DEBUG "0" 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG } 13:08:38 ERROR [http_cmd]: ERROR 13:08:38 ERROR --- status code --- 13:08:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:08:38 ERROR --- http response --- 13:08:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:08:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:08:38 INFO [http_cmd]: http status code OK 13:08:38 DEBUG --- status code --- 13:08:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:38 DEBUG --- http response --- 13:08:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:08:38 DEBUG ----- output ----- 13:08:38 DEBUG { 13:08:38 DEBUG "status": "success", 13:08:38 DEBUG "data": { 13:08:38 DEBUG "resultType": "vector", 13:08:38 DEBUG "result": [ 13:08:38 DEBUG { 13:08:38 DEBUG "metric": { 13:08:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG [...] 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG "pod": "ds-cts-1" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 1713528518, 13:08:38 DEBUG "13990449.62049859" 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG } 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528518" 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528518" 13:08:38 INFO [http_cmd]: http status code OK 13:08:38 INFO [http_cmd]: http status code OK 13:08:38 DEBUG --- status code --- 13:08:38 DEBUG --- status code --- 13:08:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:38 DEBUG --- http response --- 13:08:38 DEBUG --- http response --- 13:08:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:08:38 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:08:38 DEBUG ----- output ----- 13:08:38 DEBUG { 13:08:38 DEBUG "status": "success", 13:08:38 DEBUG "data": { 13:08:38 DEBUG "resultType": "vector", 13:08:38 DEBUG "result": [ 13:08:38 DEBUG { 13:08:38 DEBUG "metric": { 13:08:38 DEBUG "pod": "ds-idrepo-0" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG [...] 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG "pod": "ds-cts-2" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 1713528518, 13:08:38 DEBUG "0" 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG } 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528518" 13:08:38 INFO [http_cmd]: http status code OK 13:08:38 DEBUG --- status code --- 13:08:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:38 DEBUG --- http response --- 13:08:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:38 DEBUG ----- output ----- 13:08:38 DEBUG { 13:08:38 DEBUG "status": "success", 13:08:38 DEBUG "data": { 13:08:38 DEBUG "resultType": "vector", 13:08:38 DEBUG "result": [ 13:08:38 DEBUG { 13:08:38 DEBUG "metric": { 13:08:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG [...] 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 1713528518, 13:08:38 DEBUG "509651.5" 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG } 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528518" 13:08:38 INFO [http_cmd]: http status code OK 13:08:38 DEBUG --- status code --- 13:08:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:38 DEBUG --- http response --- 13:08:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:38 DEBUG ----- output ----- 13:08:38 DEBUG { 13:08:38 DEBUG "status": "success", 13:08:38 DEBUG "data": { 13:08:38 DEBUG "resultType": "vector", 13:08:38 DEBUG "result": [ 13:08:38 DEBUG { 13:08:38 DEBUG "metric": { 13:08:38 DEBUG "pod": "ds-idrepo-0" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG [...] 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG "pod": "ds-idrepo-2" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 1713528518, 13:08:38 DEBUG "0" 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG } 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528518" 13:08:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528518" 13:08:38 INFO [http_cmd]: http status code OK 13:08:38 DEBUG --- status code --- 13:08:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:38 DEBUG --- http response --- 13:08:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:38 DEBUG ----- output ----- 13:08:38 DEBUG { 13:08:38 DEBUG "status": "success", 13:08:38 DEBUG "data": { 13:08:38 DEBUG "resultType": "vector", 13:08:38 DEBUG "result": [ 13:08:38 DEBUG { 13:08:38 DEBUG "metric": { 13:08:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG [...] 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 1713528518, 13:08:38 DEBUG "1385.1" 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG } 13:08:38 INFO [http_cmd]: http status code OK 13:08:38 DEBUG --- status code --- 13:08:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:38 DEBUG --- http response --- 13:08:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:38 DEBUG ----- output ----- 13:08:38 DEBUG { 13:08:38 DEBUG "status": "success", 13:08:38 DEBUG "data": { 13:08:38 DEBUG "resultType": "vector", 13:08:38 DEBUG "result": [ 13:08:38 DEBUG { 13:08:38 DEBUG "metric": { 13:08:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG [...] 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG 13:08:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:08:38 DEBUG }, 13:08:38 DEBUG "value": [ 13:08:38 DEBUG 1713528518, 13:08:38 DEBUG "0.6655703577646416" 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG ] 13:08:38 DEBUG } 13:08:38 DEBUG } 13:08:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528519" 13:08:39 INFO [http_cmd]: http status code OK 13:08:39 DEBUG --- status code --- 13:08:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:39 DEBUG --- http response --- 13:08:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:39 DEBUG ----- output ----- 13:08:39 DEBUG { 13:08:39 DEBUG "status": "success", 13:08:39 DEBUG "data": { 13:08:39 DEBUG "resultType": "vector", 13:08:39 DEBUG "result": [ 13:08:39 DEBUG { 13:08:39 DEBUG "metric": { 13:08:39 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG [...] 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG "pod": "ds-cts-1" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 1713528519, 13:08:39 DEBUG "26989497.043207996" 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG } 13:08:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528519" 13:08:39 INFO [http_cmd]: http status code OK 13:08:39 DEBUG --- status code --- 13:08:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:39 DEBUG --- http response --- 13:08:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:39 DEBUG ----- output ----- 13:08:39 DEBUG { 13:08:39 DEBUG "status": "success", 13:08:39 DEBUG "data": { 13:08:39 DEBUG "resultType": "vector", 13:08:39 DEBUG "result": [ 13:08:39 DEBUG { 13:08:39 DEBUG "metric": { 13:08:39 DEBUG "pod": "ds-idrepo-0" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG [...] 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG "pod": "ds-idrepo-2" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 1713528519, 13:08:39 DEBUG "0" 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG } 13:08:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528519" 13:08:39 INFO [http_cmd]: http status code OK 13:08:39 DEBUG --- status code --- 13:08:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:39 DEBUG --- http response --- 13:08:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:39 DEBUG ----- output ----- 13:08:39 DEBUG { 13:08:39 DEBUG "status": "success", 13:08:39 DEBUG "data": { 13:08:39 DEBUG "resultType": "vector", 13:08:39 DEBUG "result": [ 13:08:39 DEBUG { 13:08:39 DEBUG "metric": { 13:08:39 DEBUG "instance": "10.142.0.107:9100" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG [...] 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG "instance": "10.142.0.43:9100" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 1713528519, 13:08:39 DEBUG "0.07433333333333203" 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG } 13:08:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528519" 13:08:39 INFO [http_cmd]: http status code OK 13:08:39 DEBUG --- status code --- 13:08:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:39 DEBUG --- http response --- 13:08:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:39 DEBUG ----- output ----- 13:08:39 DEBUG { 13:08:39 DEBUG "status": "success", 13:08:39 DEBUG "data": { 13:08:39 DEBUG "resultType": "vector", 13:08:39 DEBUG "result": [ 13:08:39 DEBUG { 13:08:39 DEBUG "metric": { 13:08:39 DEBUG "instance": "10.142.0.107:9100" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG [...] 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG "instance": "10.142.0.43:9100" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 1713528519, 13:08:39 DEBUG "0.12660000000000196" 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG } 13:08:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528519" 13:08:39 INFO [http_cmd]: http status code OK 13:08:39 DEBUG --- status code --- 13:08:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:39 DEBUG --- http response --- 13:08:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:39 DEBUG ----- output ----- 13:08:39 DEBUG { 13:08:39 DEBUG "status": "success", 13:08:39 DEBUG "data": { 13:08:39 DEBUG "resultType": "vector", 13:08:39 DEBUG "result": [ 13:08:39 DEBUG { 13:08:39 DEBUG "metric": { 13:08:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG [...] 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 1713528519, 13:08:39 DEBUG "0" 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG } 13:08:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528519" 13:08:39 INFO [http_cmd]: http status code OK 13:08:39 DEBUG --- status code --- 13:08:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:39 DEBUG --- http response --- 13:08:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:39 DEBUG ----- output ----- 13:08:39 DEBUG { 13:08:39 DEBUG "status": "success", 13:08:39 DEBUG "data": { 13:08:39 DEBUG "resultType": "vector", 13:08:39 DEBUG "result": [ 13:08:39 DEBUG { 13:08:39 DEBUG "metric": { 13:08:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG [...] 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 1713528519, 13:08:39 DEBUG "0" 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG } 13:08:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528519" 13:08:39 INFO [http_cmd]: http status code OK 13:08:39 DEBUG --- status code --- 13:08:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:39 DEBUG --- http response --- 13:08:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:39 DEBUG ----- output ----- 13:08:39 DEBUG { 13:08:39 DEBUG "status": "success", 13:08:39 DEBUG "data": { 13:08:39 DEBUG "resultType": "vector", 13:08:39 DEBUG "result": [ 13:08:39 DEBUG { 13:08:39 DEBUG "metric": { 13:08:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG [...] 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG 13:08:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:08:39 DEBUG }, 13:08:39 DEBUG "value": [ 13:08:39 DEBUG 1713528519, 13:08:39 DEBUG "0" 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG ] 13:08:39 DEBUG } 13:08:39 DEBUG } 13:08:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528520" 13:08:40 INFO [http_cmd]: http status code OK 13:08:40 DEBUG --- status code --- 13:08:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:40 DEBUG --- http response --- 13:08:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:40 DEBUG ----- output ----- 13:08:40 DEBUG { 13:08:40 DEBUG "status": "success", 13:08:40 DEBUG "data": { 13:08:40 DEBUG "resultType": "vector", 13:08:40 DEBUG "result": [ 13:08:40 DEBUG { 13:08:40 DEBUG "metric": { 13:08:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:08:40 DEBUG }, 13:08:40 DEBUG "value": [ 13:08:40 DEBUG 13:08:40 DEBUG 13:08:40 DEBUG 13:08:40 DEBUG [...] 13:08:40 DEBUG 13:08:40 DEBUG 13:08:40 DEBUG 13:08:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:08:40 DEBUG }, 13:08:40 DEBUG "value": [ 13:08:40 DEBUG 1713528520, 13:08:40 DEBUG "7580389376" 13:08:40 DEBUG ] 13:08:40 DEBUG } 13:08:40 DEBUG ] 13:08:40 DEBUG } 13:08:40 DEBUG } 13:08:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528522" 13:08:42 INFO [http_cmd]: http status code OK 13:08:42 DEBUG --- status code --- 13:08:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:42 DEBUG --- http response --- 13:08:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:08:42 DEBUG ----- output ----- 13:08:42 DEBUG { 13:08:42 DEBUG "status": "success", 13:08:42 DEBUG "data": { 13:08:42 DEBUG "resultType": "vector", 13:08:42 DEBUG "result": [ 13:08:42 DEBUG { 13:08:42 DEBUG "metric": { 13:08:42 DEBUG "pod": "ds-cts-0" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG [...] 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG "pod": "ds-cts-2" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 1713528522, 13:08:42 DEBUG "30" 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG } 13:08:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528522" 13:08:42 INFO [http_cmd]: http status code OK 13:08:42 DEBUG --- status code --- 13:08:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:42 DEBUG --- http response --- 13:08:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:42 DEBUG ----- output ----- 13:08:42 DEBUG { 13:08:42 DEBUG "status": "success", 13:08:42 DEBUG "data": { 13:08:42 DEBUG "resultType": "vector", 13:08:42 DEBUG "result": [ 13:08:42 DEBUG { 13:08:42 DEBUG "metric": { 13:08:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG [...] 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 1713528522, 13:08:42 DEBUG "55.19" 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG } 13:08:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528522" 13:08:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528522" 13:08:42 INFO [http_cmd]: http status code OK 13:08:42 DEBUG --- status code --- 13:08:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:42 DEBUG --- http response --- 13:08:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:42 DEBUG ----- output ----- 13:08:42 DEBUG { 13:08:42 DEBUG "status": "success", 13:08:42 DEBUG "data": { 13:08:42 DEBUG "resultType": "vector", 13:08:42 DEBUG "result": [ 13:08:42 DEBUG { 13:08:42 DEBUG "metric": { 13:08:42 DEBUG "pod": "ds-idrepo-0" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG [...] 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG "pod": "ds-idrepo-2" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 1713528522, 13:08:42 DEBUG "0" 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG } 13:08:42 INFO [http_cmd]: http status code OK 13:08:42 DEBUG --- status code --- 13:08:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:42 DEBUG --- http response --- 13:08:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:08:42 DEBUG ----- output ----- 13:08:42 DEBUG { 13:08:42 DEBUG "status": "success", 13:08:42 DEBUG "data": { 13:08:42 DEBUG "resultType": "vector", 13:08:42 DEBUG "result": [ 13:08:42 DEBUG { 13:08:42 DEBUG "metric": { 13:08:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG [...] 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG "pod": "ds-cts-2" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 1713528522, 13:08:42 DEBUG "0" 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG } 13:08:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528522" 13:08:42 INFO [http_cmd]: http status code OK 13:08:42 DEBUG --- status code --- 13:08:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:42 DEBUG --- http response --- 13:08:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:42 DEBUG ----- output ----- 13:08:42 DEBUG { 13:08:42 DEBUG "status": "success", 13:08:42 DEBUG "data": { 13:08:42 DEBUG "resultType": "vector", 13:08:42 DEBUG "result": [ 13:08:42 DEBUG { 13:08:42 DEBUG "metric": { 13:08:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG [...] 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG "pod": "ds-cts-2" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 1713528522, 13:08:42 DEBUG "0" 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG } 13:08:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528522" 13:08:42 INFO [http_cmd]: http status code OK 13:08:42 DEBUG --- status code --- 13:08:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:42 DEBUG --- http response --- 13:08:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:42 DEBUG ----- output ----- 13:08:42 DEBUG { 13:08:42 DEBUG "status": "success", 13:08:42 DEBUG "data": { 13:08:42 DEBUG "resultType": "vector", 13:08:42 DEBUG "result": [ 13:08:42 DEBUG { 13:08:42 DEBUG "metric": { 13:08:42 DEBUG "instance": "10.142.0.107:9100" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG [...] 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG "instance": "10.142.0.43:9100" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 1713528522, 13:08:42 DEBUG "1604559580160" 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG } 13:08:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528522" 13:08:42 INFO [http_cmd]: http status code OK 13:08:42 DEBUG --- status code --- 13:08:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:42 DEBUG --- http response --- 13:08:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:08:42 DEBUG ----- output ----- 13:08:42 DEBUG { 13:08:42 DEBUG "status": "success", 13:08:42 DEBUG "data": { 13:08:42 DEBUG "resultType": "vector", 13:08:42 DEBUG "result": [ 13:08:42 DEBUG { 13:08:42 DEBUG "metric": { 13:08:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG [...] 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG 13:08:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:08:42 DEBUG }, 13:08:42 DEBUG "value": [ 13:08:42 DEBUG 1713528522, 13:08:42 DEBUG "0" 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG ] 13:08:42 DEBUG } 13:08:42 DEBUG } 13:08:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528523" 13:08:43 INFO [http_cmd]: http status code OK 13:08:43 DEBUG --- status code --- 13:08:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:43 DEBUG --- http response --- 13:08:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:43 DEBUG ----- output ----- 13:08:43 DEBUG { 13:08:43 DEBUG "status": "success", 13:08:43 DEBUG "data": { 13:08:43 DEBUG "resultType": "vector", 13:08:43 DEBUG "result": [ 13:08:43 DEBUG { 13:08:43 DEBUG "metric": { 13:08:43 DEBUG "instance": "10.142.0.107:9100" 13:08:43 DEBUG }, 13:08:43 DEBUG "value": [ 13:08:43 DEBUG 13:08:43 DEBUG 13:08:43 DEBUG 13:08:43 DEBUG [...] 13:08:43 DEBUG 13:08:43 DEBUG 13:08:43 DEBUG 13:08:43 DEBUG "instance": "10.142.0.43:9100" 13:08:43 DEBUG }, 13:08:43 DEBUG "value": [ 13:08:43 DEBUG 1713528523, 13:08:43 DEBUG "9133225.133333335" 13:08:43 DEBUG ] 13:08:43 DEBUG } 13:08:43 DEBUG ] 13:08:43 DEBUG } 13:08:43 DEBUG } 13:08:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528523" 13:08:43 INFO [http_cmd]: http status code OK 13:08:43 DEBUG --- status code --- 13:08:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:08:43 DEBUG --- http response --- 13:08:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:08:43 DEBUG ----- output ----- 13:08:43 DEBUG { 13:08:43 DEBUG "status": "success", 13:08:43 DEBUG "data": { 13:08:43 DEBUG "resultType": "vector", 13:08:43 DEBUG "result": [ 13:08:43 DEBUG { 13:08:43 DEBUG "metric": { 13:08:43 DEBUG "instance": "10.142.0.107:9100" 13:08:43 DEBUG }, 13:08:43 DEBUG "value": [ 13:08:43 DEBUG 13:08:43 DEBUG 13:08:43 DEBUG 13:08:43 DEBUG [...] 13:08:43 DEBUG 13:08:43 DEBUG 13:08:43 DEBUG 13:08:43 DEBUG "instance": "10.142.0.43:9100" 13:08:43 DEBUG }, 13:08:43 DEBUG "value": [ 13:08:43 DEBUG 1713528523, 13:08:43 DEBUG "9140300.7" 13:08:43 DEBUG ] 13:08:43 DEBUG } 13:08:43 DEBUG ] 13:08:43 DEBUG } 13:08:43 DEBUG } 13:08:57 INFO 13:08:57 INFO [loop_until]: kubectl --namespace=xlou top pods 13:08:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:08:57 INFO [loop_until]: OK (rc = 0) 13:08:57 DEBUG --- stdout --- 13:08:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 1388m 8755Mi am-6fd554f8f7-hvvkw 1348m 9005Mi am-6fd554f8f7-pmj4b 1500m 8649Mi ds-cts-0 4333m 12596Mi ds-cts-1 3858m 12482Mi ds-cts-2 4006m 12810Mi ds-idrepo-0 388m 3365Mi ds-idrepo-1 397m 3263Mi ds-idrepo-2 400m 3192Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-747d57dcb8-lclbf 5m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1018m 7191Mi 13:08:57 DEBUG --- stderr --- 13:08:57 DEBUG 13:09:04 INFO 13:09:04 INFO [loop_until]: kubectl --namespace=xlou top node 13:09:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:09:04 INFO [loop_until]: OK (rc = 0) 13:09:04 DEBUG --- stdout --- 13:09:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 6136Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 461m 1% 6882Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1407m 4% 11762Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3249Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 4397m 14% 15634Mi 14% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 5158Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1471m 4% 11578Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3080Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1511m 5% 10884Mi 9% gke-xlou-cdm-ds-0f601751-j488 459m 1% 6372Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 3837m 12% 17279Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 479m 1% 7027Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 4225m 14% 14933Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1082m 6% 10994Mi 18% 13:09:04 DEBUG --- stderr --- 13:09:04 DEBUG 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528578" 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528578" 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528578" 13:09:38 ERROR [http_cmd]: ERROR 13:09:38 INFO [http_cmd]: http status code OK 13:09:38 ERROR --- status code --- 13:09:38 DEBUG --- status code --- 13:09:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:09:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:38 ERROR --- http response --- 13:09:38 DEBUG --- http response --- 13:09:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:09:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:09:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:09:38 DEBUG ----- output ----- 13:09:38 DEBUG { 13:09:38 DEBUG "status": "success", 13:09:38 DEBUG "data": { 13:09:38 DEBUG "resultType": "vector", 13:09:38 DEBUG "result": [ 13:09:38 DEBUG { 13:09:38 DEBUG "metric": { 13:09:38 DEBUG "pod": "ds-cts-0" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG [...] 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG "pod": "ds-cts-2" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 1713528578, 13:09:38 DEBUG "0" 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG } 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528578" 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528578" 13:09:38 INFO [http_cmd]: http status code OK 13:09:38 DEBUG --- status code --- 13:09:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:38 DEBUG --- http response --- 13:09:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:09:38 DEBUG ----- output ----- 13:09:38 DEBUG { 13:09:38 DEBUG "status": "success", 13:09:38 DEBUG "data": { 13:09:38 DEBUG "resultType": "vector", 13:09:38 DEBUG "result": [ 13:09:38 DEBUG { 13:09:38 DEBUG "metric": { 13:09:38 DEBUG "pod": "ds-cts-0" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG [...] 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG "pod": "ds-cts-2" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 1713528578, 13:09:38 DEBUG "0.00018262666666666666" 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG } 13:09:38 INFO [http_cmd]: http status code OK 13:09:38 DEBUG --- status code --- 13:09:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:38 DEBUG --- http response --- 13:09:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:09:38 DEBUG ----- output ----- 13:09:38 DEBUG { 13:09:38 DEBUG "status": "success", 13:09:38 DEBUG "data": { 13:09:38 DEBUG "resultType": "vector", 13:09:38 DEBUG "result": [ 13:09:38 DEBUG { 13:09:38 DEBUG "metric": { 13:09:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG [...] 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG "pod": "ds-idrepo-1" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 1713528578, 13:09:38 DEBUG "2335.9685690093143" 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG } 13:09:38 INFO [http_cmd]: http status code OK 13:09:38 DEBUG --- status code --- 13:09:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:38 DEBUG --- http response --- 13:09:38 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528578" 13:09:38 INFO [http_cmd]: http status code OK 13:09:38 DEBUG --- status code --- 13:09:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:38 DEBUG --- http response --- 13:09:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:09:38 DEBUG ----- output ----- 13:09:38 DEBUG { 13:09:38 DEBUG "status": "success", 13:09:38 DEBUG "data": { 13:09:38 DEBUG "resultType": "vector", 13:09:38 DEBUG "result": [ 13:09:38 DEBUG { 13:09:38 DEBUG "metric": { 13:09:38 DEBUG "pod": "ds-idrepo-0" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG [...] 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG "pod": "ds-cts-2" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 1713528578, 13:09:38 DEBUG "0" 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG } 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528578" 13:09:38 INFO [http_cmd]: http status code OK 13:09:38 DEBUG --- status code --- 13:09:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:38 DEBUG --- http response --- 13:09:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:38 DEBUG ----- output ----- 13:09:38 DEBUG { 13:09:38 DEBUG "status": "success", 13:09:38 DEBUG "data": { 13:09:38 DEBUG "resultType": "vector", 13:09:38 DEBUG "result": [ 13:09:38 DEBUG { 13:09:38 DEBUG "metric": { 13:09:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG [...] 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 1713528578, 13:09:38 DEBUG "430064" 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG } 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528578" 13:09:38 INFO [http_cmd]: http status code OK 13:09:38 DEBUG --- status code --- 13:09:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:38 DEBUG --- http response --- 13:09:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:38 DEBUG ----- output ----- 13:09:38 DEBUG { 13:09:38 DEBUG "status": "success", 13:09:38 DEBUG "data": { 13:09:38 DEBUG "resultType": "vector", 13:09:38 DEBUG "result": [ 13:09:38 DEBUG { 13:09:38 DEBUG "metric": { 13:09:38 DEBUG "pod": "ds-idrepo-0" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG [...] 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG "pod": "ds-idrepo-2" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 1713528578, 13:09:38 DEBUG "0" 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG } 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528578" 13:09:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528578" 13:09:38 INFO [http_cmd]: http status code OK 13:09:38 DEBUG --- status code --- 13:09:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:38 DEBUG --- http response --- 13:09:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:38 DEBUG ----- output ----- 13:09:38 DEBUG { 13:09:38 DEBUG "status": "success", 13:09:38 DEBUG "data": { 13:09:38 DEBUG "resultType": "vector", 13:09:38 DEBUG "result": [ 13:09:38 DEBUG { 13:09:38 DEBUG "metric": { 13:09:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG [...] 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 1713528578, 13:09:38 DEBUG "1294.0333333333333" 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG } 13:09:38 INFO [http_cmd]: http status code OK 13:09:38 DEBUG --- status code --- 13:09:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:38 DEBUG --- http response --- 13:09:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:38 DEBUG ----- output ----- 13:09:38 DEBUG { 13:09:38 DEBUG "status": "success", 13:09:38 DEBUG "data": { 13:09:38 DEBUG "resultType": "vector", 13:09:38 DEBUG "result": [ 13:09:38 DEBUG { 13:09:38 DEBUG "metric": { 13:09:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG [...] 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG 13:09:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:09:38 DEBUG }, 13:09:38 DEBUG "value": [ 13:09:38 DEBUG 1713528578, 13:09:38 DEBUG "1.0349015934842227" 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG ] 13:09:38 DEBUG } 13:09:38 DEBUG } 13:09:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528579" 13:09:39 INFO [http_cmd]: http status code OK 13:09:39 DEBUG --- status code --- 13:09:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:39 DEBUG --- http response --- 13:09:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:39 DEBUG ----- output ----- 13:09:39 DEBUG { 13:09:39 DEBUG "status": "success", 13:09:39 DEBUG "data": { 13:09:39 DEBUG "resultType": "vector", 13:09:39 DEBUG "result": [ 13:09:39 DEBUG { 13:09:39 DEBUG "metric": { 13:09:39 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG [...] 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG "pod": "ds-idrepo-1" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 1713528579, 13:09:39 DEBUG "0" 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG } 13:09:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528579" 13:09:39 INFO [http_cmd]: http status code OK 13:09:39 DEBUG --- status code --- 13:09:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:39 DEBUG --- http response --- 13:09:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:39 DEBUG ----- output ----- 13:09:39 DEBUG { 13:09:39 DEBUG "status": "success", 13:09:39 DEBUG "data": { 13:09:39 DEBUG "resultType": "vector", 13:09:39 DEBUG "result": [ 13:09:39 DEBUG { 13:09:39 DEBUG "metric": { 13:09:39 DEBUG "pod": "ds-idrepo-0" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG [...] 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG "pod": "ds-idrepo-2" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 1713528579, 13:09:39 DEBUG "0" 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG } 13:09:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528579" 13:09:39 INFO [http_cmd]: http status code OK 13:09:39 DEBUG --- status code --- 13:09:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:39 DEBUG --- http response --- 13:09:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:39 DEBUG ----- output ----- 13:09:39 DEBUG { 13:09:39 DEBUG "status": "success", 13:09:39 DEBUG "data": { 13:09:39 DEBUG "resultType": "vector", 13:09:39 DEBUG "result": [ 13:09:39 DEBUG { 13:09:39 DEBUG "metric": { 13:09:39 DEBUG "instance": "10.142.0.107:9100" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG [...] 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG "instance": "10.142.0.43:9100" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 1713528579, 13:09:39 DEBUG "0.10033333333333681" 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG } 13:09:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528579" 13:09:39 INFO [http_cmd]: http status code OK 13:09:39 DEBUG --- status code --- 13:09:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:39 DEBUG --- http response --- 13:09:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:39 DEBUG ----- output ----- 13:09:39 DEBUG { 13:09:39 DEBUG "status": "success", 13:09:39 DEBUG "data": { 13:09:39 DEBUG "resultType": "vector", 13:09:39 DEBUG "result": [ 13:09:39 DEBUG { 13:09:39 DEBUG "metric": { 13:09:39 DEBUG "instance": "10.142.0.107:9100" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG [...] 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG "instance": "10.142.0.43:9100" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 1713528579, 13:09:39 DEBUG "0.1644666666667225" 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG } 13:09:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528579" 13:09:39 INFO [http_cmd]: http status code OK 13:09:39 DEBUG --- status code --- 13:09:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:39 DEBUG --- http response --- 13:09:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:39 DEBUG ----- output ----- 13:09:39 DEBUG { 13:09:39 DEBUG "status": "success", 13:09:39 DEBUG "data": { 13:09:39 DEBUG "resultType": "vector", 13:09:39 DEBUG "result": [ 13:09:39 DEBUG { 13:09:39 DEBUG "metric": { 13:09:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG [...] 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 1713528579, 13:09:39 DEBUG "0" 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG } 13:09:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528579" 13:09:39 INFO [http_cmd]: http status code OK 13:09:39 DEBUG --- status code --- 13:09:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:39 DEBUG --- http response --- 13:09:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:39 DEBUG ----- output ----- 13:09:39 DEBUG { 13:09:39 DEBUG "status": "success", 13:09:39 DEBUG "data": { 13:09:39 DEBUG "resultType": "vector", 13:09:39 DEBUG "result": [ 13:09:39 DEBUG { 13:09:39 DEBUG "metric": { 13:09:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG [...] 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 1713528579, 13:09:39 DEBUG "0" 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG } 13:09:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528579" 13:09:39 INFO [http_cmd]: http status code OK 13:09:39 DEBUG --- status code --- 13:09:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:39 DEBUG --- http response --- 13:09:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:39 DEBUG ----- output ----- 13:09:39 DEBUG { 13:09:39 DEBUG "status": "success", 13:09:39 DEBUG "data": { 13:09:39 DEBUG "resultType": "vector", 13:09:39 DEBUG "result": [ 13:09:39 DEBUG { 13:09:39 DEBUG "metric": { 13:09:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG [...] 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG 13:09:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:09:39 DEBUG }, 13:09:39 DEBUG "value": [ 13:09:39 DEBUG 1713528579, 13:09:39 DEBUG "0" 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG ] 13:09:39 DEBUG } 13:09:39 DEBUG } 13:09:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528580" 13:09:40 INFO [http_cmd]: http status code OK 13:09:40 DEBUG --- status code --- 13:09:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:40 DEBUG --- http response --- 13:09:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:40 DEBUG ----- output ----- 13:09:40 DEBUG { 13:09:40 DEBUG "status": "success", 13:09:40 DEBUG "data": { 13:09:40 DEBUG "resultType": "vector", 13:09:40 DEBUG "result": [ 13:09:40 DEBUG { 13:09:40 DEBUG "metric": { 13:09:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:09:40 DEBUG }, 13:09:40 DEBUG "value": [ 13:09:40 DEBUG 13:09:40 DEBUG 13:09:40 DEBUG 13:09:40 DEBUG [...] 13:09:40 DEBUG 13:09:40 DEBUG 13:09:40 DEBUG 13:09:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:09:40 DEBUG }, 13:09:40 DEBUG "value": [ 13:09:40 DEBUG 1713528580, 13:09:40 DEBUG "7638577152" 13:09:40 DEBUG ] 13:09:40 DEBUG } 13:09:40 DEBUG ] 13:09:40 DEBUG } 13:09:40 DEBUG } 13:09:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528582" 13:09:42 INFO [http_cmd]: http status code OK 13:09:42 DEBUG --- status code --- 13:09:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:42 DEBUG --- http response --- 13:09:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:09:42 DEBUG ----- output ----- 13:09:42 DEBUG { 13:09:42 DEBUG "status": "success", 13:09:42 DEBUG "data": { 13:09:42 DEBUG "resultType": "vector", 13:09:42 DEBUG "result": [ 13:09:42 DEBUG { 13:09:42 DEBUG "metric": { 13:09:42 DEBUG "pod": "ds-cts-0" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG [...] 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG "pod": "ds-cts-2" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 1713528582, 13:09:42 DEBUG "30" 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG } 13:09:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528582" 13:09:42 INFO [http_cmd]: http status code OK 13:09:42 DEBUG --- status code --- 13:09:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:42 DEBUG --- http response --- 13:09:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:42 DEBUG ----- output ----- 13:09:42 DEBUG { 13:09:42 DEBUG "status": "success", 13:09:42 DEBUG "data": { 13:09:42 DEBUG "resultType": "vector", 13:09:42 DEBUG "result": [ 13:09:42 DEBUG { 13:09:42 DEBUG "metric": { 13:09:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG [...] 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 1713528582, 13:09:42 DEBUG "55.19" 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG } 13:09:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528582" 13:09:42 INFO [http_cmd]: http status code OK 13:09:42 DEBUG --- status code --- 13:09:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:42 DEBUG --- http response --- 13:09:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:09:42 DEBUG ----- output ----- 13:09:42 DEBUG { 13:09:42 DEBUG "status": "success", 13:09:42 DEBUG "data": { 13:09:42 DEBUG "resultType": "vector", 13:09:42 DEBUG "result": [ 13:09:42 DEBUG { 13:09:42 DEBUG "metric": { 13:09:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG [...] 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 1713528582, 13:09:42 DEBUG "1.4405935925501354" 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG } 13:09:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528582" 13:09:42 INFO [http_cmd]: http status code OK 13:09:42 DEBUG --- status code --- 13:09:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:42 DEBUG --- http response --- 13:09:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:42 DEBUG ----- output ----- 13:09:42 DEBUG { 13:09:42 DEBUG "status": "success", 13:09:42 DEBUG "data": { 13:09:42 DEBUG "resultType": "vector", 13:09:42 DEBUG "result": [ 13:09:42 DEBUG { 13:09:42 DEBUG "metric": { 13:09:42 DEBUG "pod": "ds-idrepo-0" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG [...] 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG "pod": "ds-idrepo-2" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 1713528582, 13:09:42 DEBUG "0" 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG } 13:09:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528582" 13:09:42 INFO [http_cmd]: http status code OK 13:09:42 DEBUG --- status code --- 13:09:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:42 DEBUG --- http response --- 13:09:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:42 DEBUG ----- output ----- 13:09:42 DEBUG { 13:09:42 DEBUG "status": "success", 13:09:42 DEBUG "data": { 13:09:42 DEBUG "resultType": "vector", 13:09:42 DEBUG "result": [ 13:09:42 DEBUG { 13:09:42 DEBUG "metric": { 13:09:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG [...] 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 1713528582, 13:09:42 DEBUG "133074.81149449252" 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG } 13:09:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528582" 13:09:42 INFO [http_cmd]: http status code OK 13:09:42 DEBUG --- status code --- 13:09:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:42 DEBUG --- http response --- 13:09:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:42 DEBUG ----- output ----- 13:09:42 DEBUG { 13:09:42 DEBUG "status": "success", 13:09:42 DEBUG "data": { 13:09:42 DEBUG "resultType": "vector", 13:09:42 DEBUG "result": [ 13:09:42 DEBUG { 13:09:42 DEBUG "metric": { 13:09:42 DEBUG "instance": "10.142.0.107:9100" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG [...] 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG "instance": "10.142.0.43:9100" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 1713528582, 13:09:42 DEBUG "1605575758848" 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG } 13:09:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528582" 13:09:42 INFO [http_cmd]: http status code OK 13:09:42 DEBUG --- status code --- 13:09:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:42 DEBUG --- http response --- 13:09:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:09:42 DEBUG ----- output ----- 13:09:42 DEBUG { 13:09:42 DEBUG "status": "success", 13:09:42 DEBUG "data": { 13:09:42 DEBUG "resultType": "vector", 13:09:42 DEBUG "result": [ 13:09:42 DEBUG { 13:09:42 DEBUG "metric": { 13:09:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG [...] 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG 13:09:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:09:42 DEBUG }, 13:09:42 DEBUG "value": [ 13:09:42 DEBUG 1713528582, 13:09:42 DEBUG "0" 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG ] 13:09:42 DEBUG } 13:09:42 DEBUG } 13:09:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528583" 13:09:43 INFO [http_cmd]: http status code OK 13:09:43 DEBUG --- status code --- 13:09:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:43 DEBUG --- http response --- 13:09:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:43 DEBUG ----- output ----- 13:09:43 DEBUG { 13:09:43 DEBUG "status": "success", 13:09:43 DEBUG "data": { 13:09:43 DEBUG "resultType": "vector", 13:09:43 DEBUG "result": [ 13:09:43 DEBUG { 13:09:43 DEBUG "metric": { 13:09:43 DEBUG "instance": "10.142.0.107:9100" 13:09:43 DEBUG }, 13:09:43 DEBUG "value": [ 13:09:43 DEBUG 13:09:43 DEBUG 13:09:43 DEBUG 13:09:43 DEBUG [...] 13:09:43 DEBUG 13:09:43 DEBUG 13:09:43 DEBUG 13:09:43 DEBUG "instance": "10.142.0.43:9100" 13:09:43 DEBUG }, 13:09:43 DEBUG "value": [ 13:09:43 DEBUG 1713528583, 13:09:43 DEBUG "8484714.966666667" 13:09:43 DEBUG ] 13:09:43 DEBUG } 13:09:43 DEBUG ] 13:09:43 DEBUG } 13:09:43 DEBUG } 13:09:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528583" 13:09:43 INFO [http_cmd]: http status code OK 13:09:43 DEBUG --- status code --- 13:09:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:09:43 DEBUG --- http response --- 13:09:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:09:43 DEBUG ----- output ----- 13:09:43 DEBUG { 13:09:43 DEBUG "status": "success", 13:09:43 DEBUG "data": { 13:09:43 DEBUG "resultType": "vector", 13:09:43 DEBUG "result": [ 13:09:43 DEBUG { 13:09:43 DEBUG "metric": { 13:09:43 DEBUG "instance": "10.142.0.107:9100" 13:09:43 DEBUG }, 13:09:43 DEBUG "value": [ 13:09:43 DEBUG 13:09:43 DEBUG 13:09:43 DEBUG 13:09:43 DEBUG [...] 13:09:43 DEBUG 13:09:43 DEBUG 13:09:43 DEBUG 13:09:43 DEBUG "instance": "10.142.0.43:9100" 13:09:43 DEBUG }, 13:09:43 DEBUG "value": [ 13:09:43 DEBUG 1713528583, 13:09:43 DEBUG "8491904" 13:09:43 DEBUG ] 13:09:43 DEBUG } 13:09:43 DEBUG ] 13:09:43 DEBUG } 13:09:43 DEBUG } 13:09:57 INFO 13:09:57 INFO [loop_until]: kubectl --namespace=xlou top pods 13:09:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:09:57 INFO [loop_until]: OK (rc = 0) 13:09:57 DEBUG --- stdout --- 13:09:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 1156m 8753Mi am-6fd554f8f7-hvvkw 1280m 9004Mi am-6fd554f8f7-pmj4b 994m 8653Mi ds-cts-0 3784m 12384Mi ds-cts-1 3864m 12375Mi ds-cts-2 2966m 12461Mi ds-idrepo-0 307m 3364Mi ds-idrepo-1 311m 3262Mi ds-idrepo-2 289m 3191Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-747d57dcb8-lclbf 6m 70Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1038m 7337Mi 13:09:57 DEBUG --- stderr --- 13:09:57 DEBUG 13:10:04 INFO 13:10:04 INFO [loop_until]: kubectl --namespace=xlou top node 13:10:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 13:10:04 INFO [loop_until]: OK (rc = 0) 13:10:04 DEBUG --- stdout --- 13:10:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6138Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 438m 1% 6887Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 896m 3% 11751Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3239Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 3500m 11% 15417Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 5155Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1329m 4% 11572Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 81m 0% 3087Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1225m 4% 10890Mi 9% gke-xlou-cdm-ds-0f601751-j488 328m 1% 6377Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 2863m 9% 17165Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 411m 1% 7022Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3295m 11% 14920Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1100m 6% 10669Mi 18% 13:10:04 DEBUG --- stderr --- 13:10:04 DEBUG 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528638" 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528638" 13:10:38 INFO [http_cmd]: http status code OK 13:10:38 DEBUG --- status code --- 13:10:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:38 DEBUG --- http response --- 13:10:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:10:38 DEBUG ----- output ----- 13:10:38 DEBUG { 13:10:38 DEBUG "status": "success", 13:10:38 DEBUG "data": { 13:10:38 DEBUG "resultType": "vector", 13:10:38 DEBUG "result": [ 13:10:38 DEBUG { 13:10:38 DEBUG "metric": { 13:10:38 DEBUG "pod": "ds-cts-0" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG [...] 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG "pod": "ds-cts-2" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 1713528638, 13:10:38 DEBUG "0" 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG } 13:10:38 INFO [http_cmd]: http status code OK 13:10:38 DEBUG --- status code --- 13:10:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:38 DEBUG --- http response --- 13:10:38 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713528638" 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528638" 13:10:38 ERROR [http_cmd]: ERROR 13:10:38 ERROR --- status code --- 13:10:38 ERROR http status code is 400 (expected [200, 401, 403, 404]) 13:10:38 ERROR --- http response --- 13:10:38 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 13:10:38 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528638" 13:10:38 INFO [http_cmd]: http status code OK 13:10:38 DEBUG --- status code --- 13:10:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:38 DEBUG --- http response --- 13:10:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:10:38 DEBUG ----- output ----- 13:10:38 DEBUG { 13:10:38 DEBUG "status": "success", 13:10:38 DEBUG "data": { 13:10:38 DEBUG "resultType": "vector", 13:10:38 DEBUG "result": [ 13:10:38 DEBUG { 13:10:38 DEBUG "metric": { 13:10:38 DEBUG "pod": "ds-cts-0" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG [...] 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG "pod": "ds-cts-2" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 1713528638, 13:10:38 DEBUG "0" 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG } 13:10:38 INFO [http_cmd]: http status code OK 13:10:38 DEBUG --- status code --- 13:10:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:38 DEBUG --- http response --- 13:10:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 13:10:38 DEBUG ----- output ----- 13:10:38 DEBUG { 13:10:38 DEBUG "status": "success", 13:10:38 DEBUG "data": { 13:10:38 DEBUG "resultType": "vector", 13:10:38 DEBUG "result": [ 13:10:38 DEBUG { 13:10:38 DEBUG "metric": { 13:10:38 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG [...] 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG "pod": "ds-idrepo-2" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 1713528638, 13:10:38 DEBUG "2586.7823026019473" 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG } 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528638" 13:10:38 INFO [http_cmd]: http status code OK 13:10:38 DEBUG --- status code --- 13:10:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:38 DEBUG --- http response --- 13:10:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:38 DEBUG ----- output ----- 13:10:38 DEBUG { 13:10:38 DEBUG "status": "success", 13:10:38 DEBUG "data": { 13:10:38 DEBUG "resultType": "vector", 13:10:38 DEBUG "result": [ 13:10:38 DEBUG { 13:10:38 DEBUG "metric": { 13:10:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG [...] 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 1713528638, 13:10:38 DEBUG "399401" 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG } 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713528638" 13:10:38 INFO [http_cmd]: http status code OK 13:10:38 DEBUG --- status code --- 13:10:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:38 DEBUG --- http response --- 13:10:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:10:38 DEBUG ----- output ----- 13:10:38 DEBUG { 13:10:38 DEBUG "status": "success", 13:10:38 DEBUG "data": { 13:10:38 DEBUG "resultType": "vector", 13:10:38 DEBUG "result": [ 13:10:38 DEBUG { 13:10:38 DEBUG "metric": { 13:10:38 DEBUG "pod": "ds-idrepo-0" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG [...] 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG "pod": "ds-cts-2" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 1713528638, 13:10:38 DEBUG "0" 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG } 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528638" 13:10:38 INFO [http_cmd]: http status code OK 13:10:38 DEBUG --- status code --- 13:10:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:38 DEBUG --- http response --- 13:10:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:38 DEBUG ----- output ----- 13:10:38 DEBUG { 13:10:38 DEBUG "status": "success", 13:10:38 DEBUG "data": { 13:10:38 DEBUG "resultType": "vector", 13:10:38 DEBUG "result": [ 13:10:38 DEBUG { 13:10:38 DEBUG "metric": { 13:10:38 DEBUG "pod": "ds-idrepo-0" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG [...] 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG "pod": "ds-idrepo-2" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 1713528638, 13:10:38 DEBUG "0" 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG } 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713528638" 13:10:38 INFO [http_cmd]: http status code OK 13:10:38 DEBUG --- status code --- 13:10:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:38 DEBUG --- http response --- 13:10:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:38 DEBUG ----- output ----- 13:10:38 DEBUG { 13:10:38 DEBUG "status": "success", 13:10:38 DEBUG "data": { 13:10:38 DEBUG "resultType": "vector", 13:10:38 DEBUG "result": [ 13:10:38 DEBUG { 13:10:38 DEBUG "metric": { 13:10:38 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG [...] 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 1713528638, 13:10:38 DEBUG "0.7991733334519772" 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG } 13:10:38 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528638" 13:10:38 INFO [http_cmd]: http status code OK 13:10:38 DEBUG --- status code --- 13:10:38 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:38 DEBUG --- http response --- 13:10:38 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:38 DEBUG ----- output ----- 13:10:38 DEBUG { 13:10:38 DEBUG "status": "success", 13:10:38 DEBUG "data": { 13:10:38 DEBUG "resultType": "vector", 13:10:38 DEBUG "result": [ 13:10:38 DEBUG { 13:10:38 DEBUG "metric": { 13:10:38 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG [...] 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG 13:10:38 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:10:38 DEBUG }, 13:10:38 DEBUG "value": [ 13:10:38 DEBUG 1713528638, 13:10:38 DEBUG "0" 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG ] 13:10:38 DEBUG } 13:10:38 DEBUG } 13:10:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528639" 13:10:39 INFO [http_cmd]: http status code OK 13:10:39 DEBUG --- status code --- 13:10:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:39 DEBUG --- http response --- 13:10:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:39 DEBUG ----- output ----- 13:10:39 DEBUG { 13:10:39 DEBUG "status": "success", 13:10:39 DEBUG "data": { 13:10:39 DEBUG "resultType": "vector", 13:10:39 DEBUG "result": [ 13:10:39 DEBUG { 13:10:39 DEBUG "metric": { 13:10:39 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG [...] 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG "pod": "ds-idrepo-2" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 1713528639, 13:10:39 DEBUG "0" 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG } 13:10:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528639" 13:10:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528639" 13:10:39 INFO [http_cmd]: http status code OK 13:10:39 DEBUG --- status code --- 13:10:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:39 DEBUG --- http response --- 13:10:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:39 DEBUG ----- output ----- 13:10:39 DEBUG { 13:10:39 DEBUG "status": "success", 13:10:39 DEBUG "data": { 13:10:39 DEBUG "resultType": "vector", 13:10:39 DEBUG "result": [ 13:10:39 DEBUG { 13:10:39 DEBUG "metric": { 13:10:39 DEBUG "pod": "ds-idrepo-0" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG [...] 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG "pod": "ds-idrepo-2" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 1713528639, 13:10:39 DEBUG "0" 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG } 13:10:39 INFO [http_cmd]: http status code OK 13:10:39 DEBUG --- status code --- 13:10:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:39 DEBUG --- http response --- 13:10:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:39 DEBUG ----- output ----- 13:10:39 DEBUG { 13:10:39 DEBUG "status": "success", 13:10:39 DEBUG "data": { 13:10:39 DEBUG "resultType": "vector", 13:10:39 DEBUG "result": [ 13:10:39 DEBUG { 13:10:39 DEBUG "metric": { 13:10:39 DEBUG "instance": "10.142.0.107:9100" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG [...] 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG "instance": "10.142.0.43:9100" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 1713528639, 13:10:39 DEBUG "0.006199999999955708" 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG } 13:10:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528639" 13:10:39 INFO [http_cmd]: http status code OK 13:10:39 DEBUG --- status code --- 13:10:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:39 DEBUG --- http response --- 13:10:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:39 DEBUG ----- output ----- 13:10:39 DEBUG { 13:10:39 DEBUG "status": "success", 13:10:39 DEBUG "data": { 13:10:39 DEBUG "resultType": "vector", 13:10:39 DEBUG "result": [ 13:10:39 DEBUG { 13:10:39 DEBUG "metric": { 13:10:39 DEBUG "instance": "10.142.0.107:9100" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG [...] 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG "instance": "10.142.0.43:9100" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 1713528639, 13:10:39 DEBUG "0.003999999999998257" 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG } 13:10:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528639" 13:10:39 INFO [http_cmd]: http status code OK 13:10:39 DEBUG --- status code --- 13:10:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:39 DEBUG --- http response --- 13:10:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:39 DEBUG ----- output ----- 13:10:39 DEBUG { 13:10:39 DEBUG "status": "success", 13:10:39 DEBUG "data": { 13:10:39 DEBUG "resultType": "vector", 13:10:39 DEBUG "result": [ 13:10:39 DEBUG { 13:10:39 DEBUG "metric": { 13:10:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG [...] 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 1713528639, 13:10:39 DEBUG "0" 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG } 13:10:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528639" 13:10:39 INFO [http_cmd]: http status code OK 13:10:39 DEBUG --- status code --- 13:10:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:39 DEBUG --- http response --- 13:10:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:39 DEBUG ----- output ----- 13:10:39 DEBUG { 13:10:39 DEBUG "status": "success", 13:10:39 DEBUG "data": { 13:10:39 DEBUG "resultType": "vector", 13:10:39 DEBUG "result": [ 13:10:39 DEBUG { 13:10:39 DEBUG "metric": { 13:10:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG [...] 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 1713528639, 13:10:39 DEBUG "0" 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG } 13:10:39 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528639" 13:10:39 INFO [http_cmd]: http status code OK 13:10:39 DEBUG --- status code --- 13:10:39 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:39 DEBUG --- http response --- 13:10:39 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:39 DEBUG ----- output ----- 13:10:39 DEBUG { 13:10:39 DEBUG "status": "success", 13:10:39 DEBUG "data": { 13:10:39 DEBUG "resultType": "vector", 13:10:39 DEBUG "result": [ 13:10:39 DEBUG { 13:10:39 DEBUG "metric": { 13:10:39 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG [...] 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG 13:10:39 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:10:39 DEBUG }, 13:10:39 DEBUG "value": [ 13:10:39 DEBUG 1713528639, 13:10:39 DEBUG "0" 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG ] 13:10:39 DEBUG } 13:10:39 DEBUG } 13:10:40 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713528640" 13:10:40 INFO [http_cmd]: http status code OK 13:10:40 DEBUG --- status code --- 13:10:40 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:40 DEBUG --- http response --- 13:10:40 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:40 DEBUG ----- output ----- 13:10:40 DEBUG { 13:10:40 DEBUG "status": "success", 13:10:40 DEBUG "data": { 13:10:40 DEBUG "resultType": "vector", 13:10:40 DEBUG "result": [ 13:10:40 DEBUG { 13:10:40 DEBUG "metric": { 13:10:40 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 13:10:40 DEBUG }, 13:10:40 DEBUG "value": [ 13:10:40 DEBUG 13:10:40 DEBUG 13:10:40 DEBUG 13:10:40 DEBUG [...] 13:10:40 DEBUG 13:10:40 DEBUG 13:10:40 DEBUG 13:10:40 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 13:10:40 DEBUG }, 13:10:40 DEBUG "value": [ 13:10:40 DEBUG 1713528640, 13:10:40 DEBUG "7295262720" 13:10:40 DEBUG ] 13:10:40 DEBUG } 13:10:40 DEBUG ] 13:10:40 DEBUG } 13:10:40 DEBUG } 13:10:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528642" 13:10:42 INFO [http_cmd]: http status code OK 13:10:42 DEBUG --- status code --- 13:10:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:42 DEBUG --- http response --- 13:10:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 13:10:42 DEBUG ----- output ----- 13:10:42 DEBUG { 13:10:42 DEBUG "status": "success", 13:10:42 DEBUG "data": { 13:10:42 DEBUG "resultType": "vector", 13:10:42 DEBUG "result": [ 13:10:42 DEBUG { 13:10:42 DEBUG "metric": { 13:10:42 DEBUG "pod": "ds-cts-0" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG [...] 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG "pod": "ds-cts-2" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 1713528642, 13:10:42 DEBUG "30" 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG } 13:10:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713528642" 13:10:42 INFO [http_cmd]: http status code OK 13:10:42 DEBUG --- status code --- 13:10:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:42 DEBUG --- http response --- 13:10:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:42 DEBUG ----- output ----- 13:10:42 DEBUG { 13:10:42 DEBUG "status": "success", 13:10:42 DEBUG "data": { 13:10:42 DEBUG "resultType": "vector", 13:10:42 DEBUG "result": [ 13:10:42 DEBUG { 13:10:42 DEBUG "metric": { 13:10:42 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG [...] 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 1713528642, 13:10:42 DEBUG "55.19" 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG } 13:10:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528642" 13:10:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528642" 13:10:42 INFO [http_cmd]: http status code OK 13:10:42 DEBUG --- status code --- 13:10:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:42 DEBUG --- http response --- 13:10:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:42 DEBUG ----- output ----- 13:10:42 DEBUG { 13:10:42 DEBUG "status": "success", 13:10:42 DEBUG "data": { 13:10:42 DEBUG "resultType": "vector", 13:10:42 DEBUG "result": [ 13:10:42 DEBUG { 13:10:42 DEBUG "metric": { 13:10:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG [...] 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG "pod": "ds-idrepo-2" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 1713528642, 13:10:42 DEBUG "0.055170845245241476" 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG } 13:10:42 INFO [http_cmd]: http status code OK 13:10:42 DEBUG --- status code --- 13:10:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:42 DEBUG --- http response --- 13:10:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:42 DEBUG ----- output ----- 13:10:42 DEBUG { 13:10:42 DEBUG "status": "success", 13:10:42 DEBUG "data": { 13:10:42 DEBUG "resultType": "vector", 13:10:42 DEBUG "result": [ 13:10:42 DEBUG { 13:10:42 DEBUG "metric": { 13:10:42 DEBUG "pod": "ds-idrepo-0" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG [...] 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG "pod": "ds-idrepo-2" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 1713528642, 13:10:42 DEBUG "0" 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG } 13:10:42 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528642" 13:10:42 INFO [http_cmd]: http status code OK 13:10:42 DEBUG --- status code --- 13:10:42 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:42 DEBUG --- http response --- 13:10:42 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 13:10:42 DEBUG ----- output ----- 13:10:42 DEBUG { 13:10:42 DEBUG "status": "success", 13:10:42 DEBUG "data": { 13:10:42 DEBUG "resultType": "vector", 13:10:42 DEBUG "result": [ 13:10:42 DEBUG { 13:10:42 DEBUG "metric": { 13:10:42 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG [...] 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG 13:10:42 DEBUG "pod": "ds-idrepo-2" 13:10:42 DEBUG }, 13:10:42 DEBUG "value": [ 13:10:42 DEBUG 1713528642, 13:10:42 DEBUG "103895824.58771437" 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG ] 13:10:42 DEBUG } 13:10:42 DEBUG } 13:10:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713528643" 13:10:43 INFO [http_cmd]: http status code OK 13:10:43 DEBUG --- status code --- 13:10:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:43 DEBUG --- http response --- 13:10:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 13:10:43 DEBUG ----- output ----- 13:10:43 DEBUG { 13:10:43 DEBUG "status": "success", 13:10:43 DEBUG "data": { 13:10:43 DEBUG "resultType": "vector", 13:10:43 DEBUG "result": [ 13:10:43 DEBUG { 13:10:43 DEBUG "metric": { 13:10:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 13:10:43 DEBUG }, 13:10:43 DEBUG "value": [ 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG [...] 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 13:10:43 DEBUG }, 13:10:43 DEBUG "value": [ 13:10:43 DEBUG 1713528643, 13:10:43 DEBUG "0" 13:10:43 DEBUG ] 13:10:43 DEBUG } 13:10:43 DEBUG ] 13:10:43 DEBUG } 13:10:43 DEBUG } 13:10:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528643" 13:10:43 INFO [http_cmd]: http status code OK 13:10:43 DEBUG --- status code --- 13:10:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:43 DEBUG --- http response --- 13:10:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:43 DEBUG ----- output ----- 13:10:43 DEBUG { 13:10:43 DEBUG "status": "success", 13:10:43 DEBUG "data": { 13:10:43 DEBUG "resultType": "vector", 13:10:43 DEBUG "result": [ 13:10:43 DEBUG { 13:10:43 DEBUG "metric": { 13:10:43 DEBUG "instance": "10.142.0.107:9100" 13:10:43 DEBUG }, 13:10:43 DEBUG "value": [ 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG [...] 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG "instance": "10.142.0.43:9100" 13:10:43 DEBUG }, 13:10:43 DEBUG "value": [ 13:10:43 DEBUG 1713528643, 13:10:43 DEBUG "1606236593152" 13:10:43 DEBUG ] 13:10:43 DEBUG } 13:10:43 DEBUG ] 13:10:43 DEBUG } 13:10:43 DEBUG } 13:10:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528643" 13:10:43 INFO [http_cmd]: http status code OK 13:10:43 DEBUG --- status code --- 13:10:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:43 DEBUG --- http response --- 13:10:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:43 DEBUG ----- output ----- 13:10:43 DEBUG { 13:10:43 DEBUG "status": "success", 13:10:43 DEBUG "data": { 13:10:43 DEBUG "resultType": "vector", 13:10:43 DEBUG "result": [ 13:10:43 DEBUG { 13:10:43 DEBUG "metric": { 13:10:43 DEBUG "instance": "10.142.0.107:9100" 13:10:43 DEBUG }, 13:10:43 DEBUG "value": [ 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG [...] 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG "instance": "10.142.0.43:9100" 13:10:43 DEBUG }, 13:10:43 DEBUG "value": [ 13:10:43 DEBUG 1713528643, 13:10:43 DEBUG "28123.233333333334" 13:10:43 DEBUG ] 13:10:43 DEBUG } 13:10:43 DEBUG ] 13:10:43 DEBUG } 13:10:43 DEBUG } 13:10:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713528643" 13:10:43 INFO [http_cmd]: http status code OK 13:10:43 DEBUG --- status code --- 13:10:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 13:10:43 DEBUG --- http response --- 13:10:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 13:10:43 DEBUG ----- output ----- 13:10:43 DEBUG { 13:10:43 DEBUG "status": "success", 13:10:43 DEBUG "data": { 13:10:43 DEBUG "resultType": "vector", 13:10:43 DEBUG "result": [ 13:10:43 DEBUG { 13:10:43 DEBUG "metric": { 13:10:43 DEBUG "instance": "10.142.0.107:9100" 13:10:43 DEBUG }, 13:10:43 DEBUG "value": [ 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG [...] 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG 13:10:43 DEBUG "instance": "10.142.0.43:9100" 13:10:43 DEBUG }, 13:10:43 DEBUG "value": [ 13:10:43 DEBUG 1713528643, 13:10:43 DEBUG "35376.566666666666" 13:10:43 DEBUG ] 13:10:43 DEBUG } 13:10:43 DEBUG ] 13:10:43 DEBUG } 13:10:43 DEBUG } 13:10:45 INFO Finished: True 13:10:45 INFO Waiting for threads to register finish flag 13:11:43 INFO Done. Have a nice day! :) 127.0.0.1 - - [19/Apr/2024 13:11:43] "GET /monitoring/stop HTTP/1.1" 200 - 13:11:47 WARNING File /tmp/lodemon_data-20240419_064604/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6521', ...) 127.0.0.1 - - [19/Apr/2024 13:11:52] "GET /monitoring/process HTTP/1.1" 200 -